Human pose and skeleton reconstruction with deep neural networks from mmWave radar point clouds

The foreseen adoption of millimeter-wave (mmWave) communication systems in our everyday life has increased the research interest on using this technology for sensing applications. Previous work has shown that the reflections of the signals transmitted by a mmWave device can be used to localize objects or people in the environment and analyze their movement. Higher carrier frequencies, with their large bandwidth availability are key to obtain better spatial resolution, effectively allowing the usage of dedicated mmWave radar devices for the fine-grained analysis of human gaits. This is of particular interest in bio-mechanical motion tracking systems used in clinical and rehabilitation contexts, which are typically based on expensive and impractical marker-based devices. In this work we design and implement a deep-learning framework for mmWave radar-based motion tracking of human gait, which can reconstruct the position of a set of key points on the human body from the raw radar signal during motion. The proposed system extracts point clouds representing the moving subject, making use of signal processing and tracking techniques to remove noise from static objects and interference. Then, we adapt a state-of-the-art Neural Network (NN) for point clouds, Pointnet++, to our sparse and noisy radar data and propose a modification of the architecture to exploit the temporal correlation of radar point clouds. The results obtained on our own dataset show promising results despite the technical limitations of the motion tracking system used as a ground truth to train the proposed NN. Our system can reconstruct the position of 30 markers placed on the subject’s body with an average accuracy of 17 cm, paving the road for future work on mmWave markerless motion tracking based on a larger and more diverse set of measurements.