= Privacy Leakage Study and Protection for Virtual Reality Devices {{{#!html

Mentors: Changming Li^GR, Honglu Li^GR, Tianfang Zhang^GR

Team: Dirk Catpo Risco^GR, Brandon (Jinu) Son^UG, Brody^HS, Emily Yao^HS

Final Poster with Results }}} == Project Overview Augmented reality/virtual reality (AR/VR) is used for many applications and have been used for many purposes ranging from communicating and tourism, all the way to healthcare. Accessing the built-in motion sensors does not require user permissions, as most VR applications need to access this information in order to function. However, this introduces the possibility of privacy vulnerabilities: zero-permission motion sensors can be used in order to infer live speech, which is a problem when that speech may include sensitive information. == Project Goal The purpose of this project is to extract motion data from AR/VR devices inertial measurement unit (IMU), and then input this data to a large language model (LLM) to predict what the user is doing == Weekly Updates === Week 1 **Progress** * Read research paper ![1] regarding an eavesdropping attack called Face-Mic **Next Week Goals** * We plan to meet with our mentors and get more information on the duties and expectations of our project === Week 2 **Progress** * Read research paper ![2] regarding LLMs comprehending the physical world * Build a connection between research paper and also privacy concerns of AR/VR devices **Next Week Goals** * Get familiar with AR/VR device: * Meta Quest * How to use device * Configure settings on host computer * Extract motion data from IMU * Connecting motion sensor application program interface (API) to access data * Data processing method === Week 3 **Progress** * Set up host computer and android studio environment * Started extracting data from the inertial measurement unit (IMU) * Recorded and ran trials of varying head motions **Next Week Goals** * Run more tests to collect more data * Design more motions for data collection *Different head motions * Rotational * Linear * Combinations of head motions * Looking left and then right * Looking up and then down === Week 4 **Progress** * Placeholder **Next Week Goals** * Placeholder === Week 5 **Progress** * Placeholder **Next Week Goals** * Placeholder === Week 6 **Progress** * Placeholder **Next Week Goals** * Placeholder === Week 7 **Progress** * Placeholder **Next Week Goals** * Placeholder == Links to Presentations {{{#!html

Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Final Presentation

}}} == References {{{#!html

[1] Shi, C., Xu, X., Zhang, T., Walker, P., Wu, Y., Liu, J., Saxena, N., Chen, Y. and Yu, J., 2021, October. Face-Mic: inferring live speech and speaker identity via subtle facial dynamics captured by AR/VR motion sensors. In Proceedings of the 27th Annual International Conference on Mobile Computing and Networking (pp. 478-490).

[2] Xu, H., Han, L., Yang, Q., Li, M. and Srivastava, M., 2024, February. Penetrative ai: Making llms comprehend the physical world. In Proceedings of the 25th International Workshop on Mobile Computing Systems and Applications (pp. 1-7).

}}}