LLM-Based VR Activity Recognition

This project aims to recognize human activities through analyzing sensor data with artificial intelligence (AI) technologies.

Collecting sensor data from VR headsets & controllers and develop machine learning models to classify human activities

This project will also involve leveraging large language models (LLMs) for sensor data interpretation.

Week 1: []

Week 2: []

Week 3: ​https://docs.google.com/presentation/d/10RiVvwob7MiTjVPdoz9zFv8Z5Y2O55E6WgtBuDEMQMU/edit?slide=id.g3704141428e_2_180#slide=id.g3704141428e_2_180|Slides

Studying papers related to activity recognition using motion sensor

• Motion sensor from VR: Real-Time Recognition of In-Place Body Actions and Head Gestures using Only a Head-Mounted Display [1]
• Motion sensor from wearable devices: FitCoach: Virtual fitness coach empowered by wearable mobile devices [2]

Studying the concept of Inertial Measurement Unit (IMU)

• IMU sensor includes both accelerometer and gyroscope
• Accelerometer and gyroscope record the linear acceleration and angular velocity of the device respectively, which can be used for classification

Studying programming using Java/Javascript in Android Studio

• Exploring APIs of the IMU sensor from VR headsets

Week 4: ​Slides

More in depth analysis of papers

• Motion sensor from VR: Real-Time Recognition of In-Place Body Actions and Head Gestures using Only a Head-Mounted Display [1]
• Motion sensor from wearable devices: FitCoach: Virtual fitness coach empowered by wearable mobile devices [2]

In depth analysis in the inner workings of an IMU sensor

Accelerometer

• Takes variance in capacitor charge to determine linear acceleration

• Measures linear acceleration for 3 axes: x, y, and z

Gyroscope

• Small silicon mass vibrating at high Hertz.

• Coriolis effect applies, deflecting the silicon mass perpendicularly

• Measures angular velocity around 3 axes: pitch, roll, and yaw

Week 4: []

Week 5: []

Week 6: []

Week 7: []

Week 8: []

---

---

Questions or contributions? Please contact us.