LLM-Based VR Activity Recognition

This project aims to recognize human activities through analyzing sensor data with artificial intelligence (AI) technologies.

Collecting sensor data from VR headsets & controllers and develop machine learning models to classify human activities

This project will also involve leveraging large language models (LLMs) for sensor data interpretation.

Week 1: []

Week 2: []

Week 3: ​Slides

Studying papers related to activity recognition using motion sensor

• Motion sensor from VR: Real-Time Recognition of In-Place Body Actions and Head Gestures using Only a Head-Mounted Display [1]
• Motion sensor from wearable devices: FitCoach: Virtual fitness coach empowered by wearable mobile devices [2]

Studying the concept of Inertial Measurement Unit (IMU)

• IMU sensor includes both accelerometer and gyroscope
• Accelerometer and gyroscope record the linear acceleration and angular velocity of the device respectively, which can be used for classification

Studying programming using Java/Javascript in Android Studio

• Exploring APIs of the IMU sensor from VR headsets

Week 4: ​Slides

More in depth analysis of papers

• Motion sensor from VR: Real-Time Recognition of In-Place Body Actions and Head Gestures using Only a Head-Mounted Display [1]
• Motion sensor from wearable devices: FitCoach: Virtual fitness coach empowered by wearable mobile devices [2]

In depth analysis in the inner workings of an IMU sensor

Accelerometer

• Takes variance in capacitor charge to determine linear acceleration

• Measures linear acceleration for 3 axes: x, y, and z

Gyroscope

• Small silicon mass vibrating at high Hertz.

• Coriolis effect applies, deflecting the silicon mass perpendicularly

• Measures angular velocity around 3 axes: pitch, roll, and yaw

Week 5: Slides

---

---

Questions or contributions? Please contact us.

[1] Zhao, Jingbo, et al. "Real-time recognition of in-place body actions and head gestures using only a head-mounted display." 2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR). IEEE, 2023.

[2] Guo, Xiaonan, Jian Liu, and Yingying Chen. "FitCoach: Virtual fitness coach empowered by wearable mobile devices." IEEE INFOCOM 2017-IEEE Conference on Computer Communications. IEEE, 2017.