Advisor: Dr. Yingying (Jennifer) Chen
Mentors: Changming LiGR, Honglu LiGR, Tianfang ZhangGR
Team: Dirk Catpo RiscoGR, Brody VallierHS, Emily YaoHS
Final Poster }}} == Project Overview Augmented reality/virtual reality (AR/VR) is used for many applications and have been used for many purposes ranging from communicating and tourism, all the way to healthcare. Accessing the built-in motion sensors does not require user permissions, as most VR applications need to access this information in order to function. However, this introduces the possibility of privacy vulnerabilities: zero-permission motion sensors can be used in order to infer live speech, which is a problem when that speech may include sensitive information. == Project Goal The purpose of this project is to extract motion data from AR/VR devices inertial measurement unit (IMU), and then input this data to a large language model (LLM) to predict what the user is doing == Weekly Updates === Week 1 **[https://docs.google.com/presentation/d/1VZrjZfJtpEVUlbCmo1g7Ra4WfqlPO7QHdEdDePhxg54/edit#slide=id.g20dd9abe089_0_0, Week 1 Presentation]** **Progress** * Read research paper ![1] regarding an eavesdropping attack called Face-Mic **Next Week Goals** * We plan to meet with our mentors and get more information on the duties and expectations of our project === Week 2 **[https://docs.google.com/presentation/d/1ks7pjpWpulA2y2GsImNmkVBnvmLo06NcEL-JPvpfuwg/edit#slide=id.g20de48f9cd7_0_70, Week 2 Presentation]** **Progress** * Read research paper ![2] regarding LLMs comprehending the physical world * Build a connection between research paper and also privacy concerns of AR/VR devices **Next Week Goals** * Get familiar with AR/VR device: * Meta Quest * How to use device * Configure settings on host computer * Extract motion data from IMU * Connecting motion sensor application program interface (API) to access data * Data processing method === Week 3 **[https://docs.google.com/presentation/d/1HyIgimaEBOFLwYhSPUpR2JkvApu04FP9c8duMg6zeYc/edit#slide=id.g20de48f9cd7_0_70, Week 3 Presentation]** **Progress** * Set up host computer and android studio environment * Started extracting data from the inertial measurement unit (IMU) * Recorded and ran trials of varying head motions **Next Week Goals** * Run more tests to collect more data * Design more motions for data collection *Different head motions * Rotational * Linear * Combinations of head motions * Looking left and then right * Looking up and then down === Week 4 **[https://docs.google.com/presentation/d/1KRUhHDRpHhzC8x8RDTdJcpnMpVKybfsiIKIJ-DGMdBU/edit#slide=id.g20de48f9cd7_0_70, Week 4 Presentation]** **Progress** * Designed and collected more motion data * Looking up then back middle * Looking right then back middle * Moving head around in a shape * Moving head back and forward * Used MATLAB noise removing functions to clean graphs * smooth * lowpass * findpeaks * 3D visual of acceleration to show time and position **Next Week Goals** * Find a way to get hand motion data using Android Studio * Work on fixed prompts to get accurate LLM results using ChatGPT 4o and ChatGPT 4 === Week 5 **[https://docs.google.com/presentation/d/1Ub-aKYOBnlRYxKw9RfWrc-aVimDxZ3YlwLOPu0zaYlc/edit#slide=id.g2e889759ea4_0_30, Week 5 Presentation]** **Progress** * Enabling hand motion data collection using VR device * Utilize Android Studio and !VrApi to access VR controller interface and extract motion data * Conducted additional motion experiments to gather comprehensive data sets * Motion data with both head and hand activities * Implemented 3D plots to visualize and analyze hand motion data for accuracy **Next Week Goals** * Utilize motion research paper ![3] to model more motion activities * Start building a CNN model that can recognize activity based on motion data === Week 6 **[https://docs.google.com/presentation/d/1RKGL3x1_bact6RlaXQpRFMf6e8vnYp15zUTWYiKm1UY/edit#slide=id.g2e889759ea4_0_30, Week 6 Presentation]** **Progress** * Made a list of different motion data to capture and train a convolutional neural network (CNN) * Do research on previous work based on raw motion data and CNN’s * Specifics of the motion data * Samples * Users * Data * Design prompt for LLM **Next Week Goals** * Start getting more motion data * Start using LLM to analyze the collected data * Use the designed prompts * Design and try more prompt structures and compare LLM responses === Week 7 **[https://docs.google.com/presentation/d/1mz0NYl02uq7MzdPm-ePGp7_0iRGWYou3dY3CDbdslGI/edit#slide=id.g2e889759ea4_0_30, Week 7 Presentation]** **Progress** * Collected 250 samples from six different motions to enlarge datasets for the CNN task * Designed prompts with specific parts for LLMs to establish activity recognition tasks * Tested prompt with different LLMs * ChatGPT 4o * Gemini Advanced **Next Week Goals** * Improve the prompt design to get more accurate prediction results from LLMs * Begin developing a CNN using samples collected from the six motion patterns === Week 8 **[https://docs.google.com/presentation/d/110vsm3g8x1BW4J_OO6c98H_GSFWvOW5-4_shkLrZxmg/edit#slide=id.g2e889759ea4_0_30, Week 8 Presentation]** **Progress** * Placeholder **Next Week Goals** * Placeholder === Week 9 **[https://docs.google.com/presentation/d/1KE8qQEEzUupf4TrK-Y2B2nJP1qJqPZkHbaJRBsWVK1E/edit#slide=id.g2e889759ea4_0_30, Week 9 Presentation]** **Progress** * Placeholder **Next Week Goals** * Placeholder === Week 10 **[https://docs.google.com/presentation/d/1VZrjZfJtpEVUlbCmo1g7Ra4WfqlPO7QHdEdDePhxg54/edit#slide=id.g20dd9abe089_0_0, Week 10 Presentation]** **Progress** * Placeholder **Next Week Goals** * Placeholder == Links to Presentations {{{#!html
Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10 Final Presentation
}}} == References {{{#!html [5] MBrownlee, Jason. “1D Convolutional Neural Network Models for Human Activity Recognition.” MachineLearningMastery.Com, 27 Aug. 2020, machinelearningmastery.com/cnn-models-for-human-activity-recognition-time-series-classification/#comments. }}}