LLM-Based VR Activity Recognition
This project aims to recognize human activities through analyzing sensor data with artificial intelligence (AI) technologies.
Collecting sensor data from VR headsets & controllers and develop machine learning models to classify human activities
This project will also involve leveraging large language models (LLMs) for sensor data interpretation.
Week 1: []
Studied paper "Real-Time Recognition of In-Place Body Actions and Head Gestures using Only a Head-Mounted Display"
Studied paper "FitCoach: Virtual fitness coach empowered by wearable mobile devices"
Week 2: []
Started setting up necessary programs on our devices
- Android Studio, Arctic Fox
- MATLAB
Worked on slides template to use for future presentations.
Week 3: Slides
Studying papers related to activity recognition using motion sensor
- Motion sensor from VR: Real-Time Recognition of In-Place Body Actions and Head Gestures using Only a Head-Mounted Display [1]
- Motion sensor from wearable devices: FitCoach: Virtual fitness coach empowered by wearable mobile devices [2]
Studying the concept of Inertial Measurement Unit (IMU)
- IMU sensor includes both accelerometer and gyroscope
- Accelerometer and gyroscope record the linear acceleration and angular velocity of the device respectively, which can be used for classification
Studying programming using Java/Javascript in Android Studio
- Exploring APIs of the IMU sensor from VR headsets
Week 4: Slides
More in depth analysis of papers
- Motion sensor from VR: Real-Time Recognition of In-Place Body Actions and Head Gestures using Only a Head-Mounted Display [1]
- Motion sensor from wearable devices: FitCoach: Virtual fitness coach empowered by wearable mobile devices [2]
In depth analysis in the inner workings of an IMU sensor
Accelerometer
- Takes variance in capacitor charge to determine linear acceleration
- Measures linear acceleration for 3 axes: x, y, and z
Gyroscope
- Small silicon mass vibrating at high Hertz.
- Coriolis effect applies, deflecting the silicon mass perpendicularly
- Measures angular velocity around 3 axes: pitch, roll, and yaw
Week 5: Slides
Questions or contributions? Please contact us.
[1] Zhao, Jingbo, et al. "Real-time recognition of in-place body actions and head gestures using only a head-mounted display." 2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR). IEEE, 2023.
[2] Guo, Xiaonan, Jian Liu, and Yingying Chen. "FitCoach: Virtual fitness coach empowered by wearable mobile devices." IEEE INFOCOM 2017-IEEE Conference on Computer Communications. IEEE, 2017.
