AR Mural

  WINLAB Summer Internship 2024

  Advisors: Ivan Seskar, Jennifer Shane

  Group Members: Jovana Lekic, Nima Fallah, Nishi Agrawal, Suhani Sengupta, Sachin Kandan

Project Description

This project aims to create a collaborative augmented reality application that will allow remote users to create a virtual mural collaboratively. This project will be made using the Unity game engine and deployed to AR platforms such as Microsoft Hololens.

In addition to creating the software that will manage the generation and rendering of the mural, another important component of this project is designing an infrastructure that will allow for very low latency collaboration. If time allows, one of the end goals of this project is to deploy it using an "edge cloud" architecture using the OpenAR Cloud project.

Students should get started with this project by familiarizing themselves with the Unity game engine and Hololens development tools.

Weekly Progress

Week 1: Presentation Link: ​https://docs.google.com/presentation/d/1e9WKNtZdiVpUDRzJE05nQMO2VyV6-vSEcMwTX7Ittqk/edit?usp=sharing

• Got acquainted with Unity and HoloLens, and linked the two
• Using the Mixed Reality tool (MRTK) in Unity
• Viewed last year's codebase

Week 2: Presentation Link: ​https://docs.google.com/presentation/d/1skRqUdaybDfopb9Gdp5hkInJ3mrngNzjlyoGl3765RU/edit?usp=sharing

• Worked on brush tool
• Moved codebase to Gitlab
• Found MRTK resources

Week 3: Presentation Link: ​https://docs.google.com/presentation/d/12aLB9c_lplN2I6Wq8cHUoKUdi36zhQ3g9WWYOEJ5wso/edit?usp=sharing

• Refined brush tool and added a select tool
• Added voice command for the brush tool
• Added a color palette

Week 4: Presentation Link: ​https://docs.google.com/presentation/d/1QMFfploRVZXWm-m6f_h3WRY83yJv5-BGWKhmnzGFggw/edit?usp=sharing

• Started on smoothness, loop, and line buttons
• Added size slider functionality
• Made shape brushes

Week 5: Presentation Link: ​https://docs.google.com/presentation/d/1aF3V1rku2ULFikMyMB7wBY5Ym2e4HuJrZPH0dsFyyCA/edit?usp=sharing

• Finalized sharpness and size features
• Buttons specific to a drawing pop up when in select mode
• Created text tool
• Refactored code

Week 6: Presentation Link: ​https://docs.google.com/presentation/d/1Gm3-1dNqIkTJQwAcrsrprOLf97ekPJRsANrg5AksCvs/edit?usp=sharing

• Made text tool based on user input
• Refactored/documented existing codebase
• Created a shape tool and added two shapes - rectangular prism and ellipsoid
• Added voice commands for all tools

Week 7: Presentation Link: ​https://docs.google.com/presentation/d/1kXYXT7qoGsePtHjU1wCP2iUc-QgQiC257jsCKCRiE0g/edit?usp=sharing

• Redesigned loop button
• Matched the line select buttons’ transparency to all other menus
• Install Holographic Remote App for Player Mode
• Created a custom server implementation on top of TCP protocols
• Worked on adding a tutorial feature and help feature integrated into the main scene

Week 8: Presentation Link: ​https://docs.google.com/presentation/d/1C8ZR-LPOed49XZkATWWTwg8Brs8bPAiEkxrv13r1Kv8/edit?usp=sharing

• Added functional muti-select move and delete options
• Fixed highlighting multiple objects
• Skeleton overlay works with webcam and prerecorded videos

Week 9: Presentation Link: ​https://docs.google.com/presentation/d/115jVq8q7LLpU7HA_pZv0NUuvJcpaw3NaLt2u2qJNiJA/edit?usp=sharing

• Added a color mixer
• MQTT Server works with add, delete, and move commands
• 3D tool
• Skeleton overlay application connected to the Hololens
• Tested 2-player and 3-player mode using not only the computer Unity Builds but also the Hololens