Micro-gestures (MGs) are subtle, involuntary body movements that provide valuable insights into human psychological states and concealed emotions. Despite their potential across a wide range of applications, the automatic analysis of MGs remains a challenging task due to their brief duration and the severe class imbalance present in real-world, unconstrained datasets. This thesis investigates and compares advanced neural network architectures for two complementary tasks: micro-gesture classification, which involves recognizing actions from trimmed video clips, and online micro-gesture recognition, which focuses on the continuous detection and segmentation of MGs in untrimmed video streams. Using 3D skeleton data from the Spontaneous Micro-gestures (SMG) dataset, this work compares several deep learning architectures across both tasks. For micro-gesture classification, ST-GCN and MS-G3D networks are reproduced as baselines, and this thesis extends the comparison by applying Channel-wise Topology Refinement Graph Convolutional Networks (CTR-GCN) to the micro-gesture domain. The evaluation focuses on how each architecture addresses the spatial weighting bias inherent to fixed-topology GCNs. For online recognition, an MLP is used as a baseline emission model within a hybrid ANN-HMM framework, compared against a BiLSTM network augmented with temporal and spatial attention. The results show that CTR-GCN and MS-G3D are comparably effective at overcoming the limitations present in the ST-GCN baseline, improving top-1 classification accuracy by a similar margin. With regard to online recognition, applying weighted cross-entropy consistently increased recall at the cost of accuracy, particularly for the BiLSTM, revealing a clear precision-recall trade-off; despite this, the BiLSTM consistently outperformed the MLP baseline, most notably on subjects for which the MLP failed to detect any gestures at all.
Micro-gestures (MGs) are subtle, involuntary body movements that provide valuable insights into human psychological states and concealed emotions. Despite their potential across a wide range of applications, the automatic analysis of MGs remains a challenging task due to their brief duration and the severe class imbalance present in real-world, unconstrained datasets. This thesis investigates and compares advanced neural network architectures for two complementary tasks: micro-gesture classification, which involves recognizing actions from trimmed video clips, and online micro-gesture recognition, which focuses on the continuous detection and segmentation of MGs in untrimmed video streams. Using 3D skeleton data from the Spontaneous Micro-gestures (SMG) dataset, this work compares several deep learning architectures across both tasks. For micro-gesture classification, ST-GCN and MS-G3D networks are reproduced as baselines, and this thesis extends the comparison by applying Channel-wise Topology Refinement Graph Convolutional Networks (CTR-GCN) to the micro-gesture domain. The evaluation focuses on how each architecture addresses the spatial weighting bias inherent to fixed-topology GCNs. For online recognition, an MLP is used as a baseline emission model within a hybrid ANN-HMM framework, compared against a BiLSTM network augmented with temporal and spatial attention. The results show that CTR-GCN and MS-G3D are comparably effective at overcoming the limitations present in the ST-GCN baseline, improving top-1 classification accuracy by a similar margin. With regard to online recognition, applying weighted cross-entropy consistently increased recall at the cost of accuracy, particularly for the BiLSTM, revealing a clear precision-recall trade-off; despite this, the BiLSTM consistently outperformed the MLP baseline, most notably on subjects for which the MLP failed to detect any gestures at all.
Comparison of Neural Network Architectures for the Development of Microgestures Classification and Online Recognition
ANDRADE DAMIAN, DIANA CRISTINA
2025/2026
Abstract
Micro-gestures (MGs) are subtle, involuntary body movements that provide valuable insights into human psychological states and concealed emotions. Despite their potential across a wide range of applications, the automatic analysis of MGs remains a challenging task due to their brief duration and the severe class imbalance present in real-world, unconstrained datasets. This thesis investigates and compares advanced neural network architectures for two complementary tasks: micro-gesture classification, which involves recognizing actions from trimmed video clips, and online micro-gesture recognition, which focuses on the continuous detection and segmentation of MGs in untrimmed video streams. Using 3D skeleton data from the Spontaneous Micro-gestures (SMG) dataset, this work compares several deep learning architectures across both tasks. For micro-gesture classification, ST-GCN and MS-G3D networks are reproduced as baselines, and this thesis extends the comparison by applying Channel-wise Topology Refinement Graph Convolutional Networks (CTR-GCN) to the micro-gesture domain. The evaluation focuses on how each architecture addresses the spatial weighting bias inherent to fixed-topology GCNs. For online recognition, an MLP is used as a baseline emission model within a hybrid ANN-HMM framework, compared against a BiLSTM network augmented with temporal and spatial attention. The results show that CTR-GCN and MS-G3D are comparably effective at overcoming the limitations present in the ST-GCN baseline, improving top-1 classification accuracy by a similar margin. With regard to online recognition, applying weighted cross-entropy consistently increased recall at the cost of accuracy, particularly for the BiLSTM, revealing a clear precision-recall trade-off; despite this, the BiLSTM consistently outperformed the MLP baseline, most notably on subjects for which the MLP failed to detect any gestures at all.| File | Dimensione | Formato | |
|---|---|---|---|
|
AndradeDamian_DianaCristina.pdf
accesso aperto
Dimensione
5.87 MB
Formato
Adobe PDF
|
5.87 MB | Adobe PDF | Visualizza/Apri |
The text of this website © Università degli studi di Padova. Full Text are published under a non-exclusive license. Metadata are under a CC0 License
https://hdl.handle.net/20.500.12608/115817