wiki:Other/Summer/2023/Mural

Augmented Reality Mural

    Augmented Reality Mural

    WINLAB Summer Internship 2023

    Group Members: Alexander Clarke, Ana Obradovic, Arya Shetty, Nicholas Clarke, Pavan datta Vedire Reddy

    Project Objective

    The objective of this project is to create a collaborative augmented reality application which will allow remote users to collaboratively create a virtual mural. This project will be created using the Unity game engine and deployed to AR platforms such as the Microsoft Hololens.

    In addition to creating the software that will manage the generation and rendering of the mural, another important component of this project is designing an infrastructure that will allow for very low latency collaboration. If time allows, one of the end goals of this project is to deploy it using an "edge cloud" architecture using the OpenAR Cloud project.

    Students should get started with this project by familiarizing themselves with the Unity game engine and Hololens development tools.

    Weekly Progress

    Week 1: Presentation Link: https://docs.google.com/presentation/d/1FjPUL7XrPDCdE972nUbW7S74IX70BCDgDVR_eASt9H0/edit?usp=sharing

    • Started to familiarize ourselves with the Unity Software and Hololens.
    • We started with connecting the Hololens to the Unity app through remote connection.
    • Once we were able to see the project scene through the Hololens we tested limits of the environment

    Picture of early work


    Week 2: Presentation Link: https://docs.google.com/presentation/d/1LQVxsd-889PpTrCxg7lbWw3VhyYNj-niy3aS0FdTTnw/edit?usp=sharing

    • Set up essential MRTK (mixed reality toolkit) packages in Unity
    • Downloaded and deployed app on Hololens
    • Moved and resized objects
    • Used eye tracking feature for selection
    • Used voice commands to switch color of objects
    • Added template menu
    • Wrote code for drawing application using mouse
    • Worked on iterations of line drawings (2D & 3D)


    Week 3: Presentation Link: https://docs.google.com/presentation/d/1sy6CcWGTzTh1RKmW2iR24s1odf7g0WaoQa8a8A7ZQSw/edit?usp=sharing

    • Resize and move rendered lines
    • Highlight selected line objects
    • Selection button in menu
    • Rendering smooth curves with control points using splines
    • Ability to erase rendered lines
    • Learned about the client-server architecture
    • Learned about communication protocols —> Transmission Control Protocol (TCP) & User Datagram Protocol (UDP)
    • Coded a simple TCP Chat Room with multiple clients

    Gif of resizing tool


    Week 4: Presentation Link: https://docs.google.com/presentation/d/16PWZwUThJP8fDfp_JMGwXbcaIh0d8_6qrnV-ixCmmJA/edit?usp=sharing

    • Message Queuing telemetry transport —> Setup broker, Connected one unity client to broker, Implements Publisher-subscriber method
    • Singleton pattern for save system
    • Serialize game objects and scenes
    • Save and load structures
    • Merged features like smooth line drawing, erasing and editing
    • Tool selection and drawing resizing
    • Brush settings and color selection menu

    Picture of spline drawing


    Week 5: Presentation Link: https://docs.google.com/presentation/d/1w9NXW46-_z1c2cesLaWOiW27Q5gIE-tJYjH5PGD_khY/edit?usp=sharing

    • More work with MQTT, setting up unity client
    • More work on save system, and transferring game object files to JSON format


    Week 6: Presentation Link: https://docs.google.com/presentation/d/1QC8dDQiOfP1Gx0GNxpk489A0xPRhpyjq8akN6ApKZss/edit?usp=sharing

    • Used MQTT to send JSON files from different devices
    • Implemented saving lines by position change
    • Fixed UI and optimized brushes
    • Set up Node-Red to manage MQTT topics


    Week 7: Presentation Link: https://docs.google.com/presentation/d/1I73XR4envKSiZ2wyW71QvwwqoPtDomBs0DI1XrCHSLs/edit?usp=sharing

    • Two player collaboration
    • File Server
    • Allow for multiple users to draw incrementally
    • New menu flow and shape tools


    Week 8: Presentation Link: https://docs.google.com/presentation/d/1zA52CP9krTh-wP0SNyQAoHZ3nT4O-MO8huG1WbuqBug/edit?usp=sharing

    • Fixed bugs to reduce lag
    • Allowed for multiple users to draw simultaneously
    • Mesh, shape, and drawing functionality refined
    • Applied save/load features to different textures

    early demo of mesh drawing tool


    Week 9: Presentation Link: https://docs.google.com/presentation/d/1yg9vJt0DxlAo2eEjRVaes9K5XTRtt0YS3PNbwkqtp7c/edit?usp=sharing

    • Fixed more bugs
    • Deployed working single player app on hololens
    • Put broker on a linux VM hosted by orbit nodes
    • Created presentation and poster for final week


    Week 10: Presentation Link: https://docs.google.com/presentation/d/1yg9vJt0DxlAo2eEjRVaes9K5XTRtt0YS3PNbwkqtp7c/edit?usp=sharing

    Final Poster 3

    Last modified 17 months ago Last modified on Aug 9, 2023, 2:29:52 PM

    Attachments (15)

    Note: See TracWiki for help on using the wiki.