Augmented Reality Mural
Augmented Reality Mural
WINLAB Summer Internship 2023
Group Members: Alexander Clarke, Ana Obradovic, Arya Shetty, Nicholas Clarke, Pavan datta Vedire Reddy
Project Objective
The objective of this project is to create a collaborative augmented reality application which will allow remote users to collaboratively create a virtual mural. This project will be created using the Unity game engine and deployed to AR platforms such as the Microsoft Hololens.
In addition to creating the software that will manage the generation and rendering of the mural, another important component of this project is designing an infrastructure that will allow for very low latency collaboration. If time allows, one of the end goals of this project is to deploy it using an "edge cloud" architecture using the OpenAR Cloud project.
Students should get started with this project by familiarizing themselves with the Unity game engine and Hololens development tools.
Weekly Progress
Week 1: Presentation Link: https://docs.google.com/presentation/d/1FjPUL7XrPDCdE972nUbW7S74IX70BCDgDVR_eASt9H0/edit?usp=sharing
- Started to familiarize ourselves with the Unity Software and Hololens.
- We started with connecting the Hololens to the Unity app through remote connection.
- Once we were able to see the project scene through the Hololens we tested limits of the environment
Week 2: Presentation Link: https://docs.google.com/presentation/d/1LQVxsd-889PpTrCxg7lbWw3VhyYNj-niy3aS0FdTTnw/edit?usp=sharing
- Set up essential MRTK (mixed reality toolkit) packages in Unity
- Downloaded and deployed app on Hololens
- Moved and resized objects
- Used eye tracking feature for selection
- Used voice commands to switch color of objects
- Added template menu
- Wrote code for drawing application using mouse
- Worked on iterations of line drawings (2D & 3D)
Week 3: Presentation Link: https://docs.google.com/presentation/d/1sy6CcWGTzTh1RKmW2iR24s1odf7g0WaoQa8a8A7ZQSw/edit?usp=sharing
- Resize and move rendered lines
- Highlight selected line objects
- Selection button in menu
- Rendering smooth curves with control points using splines
- Ability to erase rendered lines
- Learned about the client-server architecture
- Learned about communication protocols —> Transmission Control Protocol (TCP) & User Datagram Protocol (UDP)
- Coded a simple TCP Chat Room with multiple clients
Week 4: Presentation Link: https://docs.google.com/presentation/d/16PWZwUThJP8fDfp_JMGwXbcaIh0d8_6qrnV-ixCmmJA/edit?usp=sharing
- Message Queuing telemetry transport —> Setup broker, Connected one unity client to broker, Implements Publisher-subscriber method
- Singleton pattern for save system
- Serialize game objects and scenes
- Save and load structures
- Merged features like smooth line drawing, erasing and editing
- Tool selection and drawing resizing
- Brush settings and color selection menu
Week 5: Presentation Link: https://docs.google.com/presentation/d/1w9NXW46-_z1c2cesLaWOiW27Q5gIE-tJYjH5PGD_khY/edit?usp=sharing
- More work with MQTT, setting up unity client
- More work on save system, and transferring game object files to JSON format
Week 6: Presentation Link: https://docs.google.com/presentation/d/1QC8dDQiOfP1Gx0GNxpk489A0xPRhpyjq8akN6ApKZss/edit?usp=sharing
- Used MQTT to send JSON files from different devices
- Implemented saving lines by position change
- Fixed UI and optimized brushes
- Set up Node-Red to manage MQTT topics
Week 7: Presentation Link: https://docs.google.com/presentation/d/1I73XR4envKSiZ2wyW71QvwwqoPtDomBs0DI1XrCHSLs/edit?usp=sharing
- Two player collaboration
- File Server
- Allow for multiple users to draw incrementally
- New menu flow and shape tools
Week 8: Presentation Link: https://docs.google.com/presentation/d/1zA52CP9krTh-wP0SNyQAoHZ3nT4O-MO8huG1WbuqBug/edit?usp=sharing
- Fixed bugs to reduce lag
- Allowed for multiple users to draw simultaneously
- Mesh, shape, and drawing functionality refined
- Applied save/load features to different textures
Week 9: Presentation Link: https://docs.google.com/presentation/d/1yg9vJt0DxlAo2eEjRVaes9K5XTRtt0YS3PNbwkqtp7c/edit?usp=sharing
- Fixed more bugs
- Deployed working single player app on hololens
- Put broker on a linux VM hosted by orbit nodes
- Created presentation and poster for final week
Week 10: Presentation Link: https://docs.google.com/presentation/d/1yg9vJt0DxlAo2eEjRVaes9K5XTRtt0YS3PNbwkqtp7c/edit?usp=sharing
- https://youtu.be/V-7BNvZp4og
- Created demo video
- Final app testing
- Commenting code for next year
Attachments (15)
-
erase1.mp4
(4.6 MB
) - added by 17 months ago.
Gif of erasing tool
- ezgif-4-54b8b1ac8f.gif (7.2 MB ) - added by 17 months ago.
-
ezgif-4-9bd62af260.gif
(4.5 MB
) - added by 16 months ago.
Gif of resizing tool
-
Screenshot 2023-06-21 095406.png
(60.0 KB
) - added by 16 months ago.
Picture of spline drawing
-
Screenshot 2023-06-14 124559.png
(294.3 KB
) - added by 16 months ago.
Picture of early work
- Node-red-icon.png (17.2 KB ) - added by 16 months ago.
- Screenshot 2023-08-01 102536.png (98.5 KB ) - added by 15 months ago.
-
2023 AR Mural Project Poster.pptx.jpg
(658.3 KB
) - added by 15 months ago.
Final Poster
- Screenshot 2023-08-02 143957.png (58.6 KB ) - added by 15 months ago.
- Screenshot 2023-08-03 104945.png (137.0 KB ) - added by 15 months ago.
- Screenshot 2023-06-21 095327.png (58.1 KB ) - added by 15 months ago.
- Screenshot 2023-08-08 121246.png (90.8 KB ) - added by 15 months ago.
-
Screenshot 2023-07-31 130717.png
(111.8 KB
) - added by 15 months ago.
early demo of mesh drawing tool
-
2023 AR Mural Project Poster.pptx.pdf
(998.7 KB
) - added by 15 months ago.
Final Poster 2
-
2023 AR Mural Project Poster.pptx.png
(1.7 MB
) - added by 15 months ago.
Final Poster 3