Robotic IoT SmartSpace Testbed

WINLAB Summer Internship 2023

Group Members: Jeremy Hui, Katrina Celario

Project Objective

Based on the future research section, there seem to be two main goals: adding a web-based reservation system for remote access to the robot and automating the activity labeling process using the natural language descriptions of the data provided in video format. For the remote reservation/experimentation features, we need to create a user-friendly webpage that not only makes it easier for the user to execute commands, but also keeps them from changing things that they shouldn’t have access to. The remote access to the LoCoBot can be achieved through ROS/SSH as long as all machines involved are connected to the same network (in our case this could be a VPN).

The video auto-labeling can be done using neural networks (ex: CNN and LSTM) in an encoder-decoder architecture for both feature extraction and the language model. For activities that cannot be classified within a specific amount of certainty, the auto-labeling tool could save the time stamp and/or video clip and notify the user that it requires manual labeling. After being fully trained, the network would simply choose the label with the highest probability and could possibly mark that data as “uncertain”.

• Web-based remote reservation/experimentation

→ remote access to robot through ROS/SSH → create a user-friendly webpage that not only makes it easier for the user to execute commands, but also keeps them from accessing/changing things that they shouldn’t

• Automatic labeling using video/captioning tools

→ Neural network models for captioning involve two main elements:

→ Feature Extraction

→ Language Model