Spatio-temporal graph autoencoder for automated evaluation of human actions in 3D in immersive VR-based training for archaeologists

Motion capture and analysis is a field of application in constant development, and improvement can be seen in both the field of advanced models and new devices. Its integration with Virtual Reality (VR) can further expand the field of application. Here, a system is built to train all those movements that require precision and accuracy in body movements, using VR to immerse the user in the relevant environment and the Xsens MVN (suit) to track and analyze the movements. A model processes motion data by constructing graphs where nodes represent key movement features. These are input to an Autoencoder (AE), AEforGraph, composed of a Graph Convolutional Network (GCN) for spatial dependencies and a Long-Short Term Memory (LSTM) network for temporal modeling. The encoded representations undergo Semi-Supervised Clustering to classify movements based on their similarity to predefined centroids representing correct execution. The decoder reconstructs the movement to highlight deviations and provide real-time corrective feedback. Live tests confirm the system’s effectiveness in recognizing and analyzing movement patterns, making it a valuable tool for training applications.