Self Driving Vehicular Project
Self Driving Vehicular Project
WINLAB Summer Internship 2023
Group Members: Adarsh Narayanan, Brandon Cheng, Christopher Sawicki, Joshua Menezes, Ruben Alias, Tommy Chu, Aleicia Zhu, Ranvith Adulla, Arya Chhabra, Suhani Sengupta
Project Objective
This project has several objectives involving the WINLAB miniature intersection.
RASCAL (Robotic Autonomous Scale Car for Adaptive Learning) | SCAMP (Self-guided Computer Assisted Mecanum Pathfinder) |
---|---|
Continue work on the miniature autonomous car hardware project developed last summer (see the project page here: https://www.orbit-lab.org/wiki/Other/Summer/2022/Hardware). The goal for this summer is to finish prototyping the hardware and develop drivers to control the car with ROS. If time allows, once the ROS integration is complete, we will be able to develop and test machine learning models to control the car. | Develop a remote-controllable car which can be used to mimic the path taken by a car in the real intersection. For this platform, mecanum wheels will be used to allow for real-time corrections to the path. The platform will be developed using Robot Operating System (ROS). Students will need to prototype hardware and develop software to extract the path from video of the real intersection, translate that into a path for the miniature intersection, and produce a set of commands for the car. |
*RASCAL pseudonyms: UpCar
, BARTHOLOMEW
*SCAMP pseudonyms: MIMIC
, CORNELIUS
Week 1
RASCAL | MIMIC |
---|---|
Worked on getting RASCAL driving with a controller Was able to parallel park | Started designing MIMIC hardware setup |
Week 2
RASCAL | MIMIC |
---|---|
Switched from PyCharm to Vim due to storage issues Started implementing ROS | Started wiring with IMU, Jetson Nano, Teensy, four brushless motors, and two motor drivers |
Week 3
RASCAL | MIMIC |
---|---|
Started setting up Rosbags to collect data to eventually train an ML model | Set up ROS and serial control |
Week 4
RASCAL | MIMIC |
---|---|
Created Flutter app to clean data from camera Set up scripts for converting .bag files to .csv | Set up controls with PS3 controller to prevent motor stall Created web server to track and simulate car movement |
Week 5
RASCAL | MIMIC |
---|---|
Worked on creating imitation learning model to drive autonomously | Created an electrical schematic Started working on PID for motor control |
Week 6
RASCAL | MIMIC |
---|---|
Started designing and testing for hardware issues with v2 (BARTHOLOMEWTWO) Set up IMU for better odometry and position tracking | Moves precise distances and angles for path following |
Week 7
RASCAL | MIMIC |
---|---|
Finished designing BARTHOLOMEWTWO and successfully ran the imitation learning model | Able to follow a path drawn on the web server |
Week 8
RASCAL | MIMIC |
---|---|
Tested pure pursuit and normalized ML inputs | Implement full omni-drive motion for path following |
Week 9
Week 10
Project Setup Instructions
Set up SSH key for Gitlab
- If you clone the repository with HTTPS, then you don't need to do this. However, then you will need to input your username and password every time you want to fetch or push from gitlab. To get around this, you can generate an SSH key pair and upload the public key to gitlab.
- navigate to your .ssh directory by doing "cd ~/.ssh"
- run the command "ssh-keygen".
- It will prompt you to name the file. For example, "gitlab".
- It will then prompt for a passphrase for the key. Leave it empty if you don't want to put in a password every time you use it, or put one for security if you prefer.
- Two files called "gitlab" and "gitlab.pub" will be created in the ~/.ssh directory now. We need to upload the public key to gitlab.
- Navigate to your .ssh directory and use the command "cat gitlab.pub" to print the contents of the file into the terminal. Copy all of the file, including the "sha-rsa" at the beginning and the username at the end.
- Login to gitlab (the winlab specific gitlab! https://gitlab.orbit-lab.org) and click on your profile picture on the top right.
- Go to SSH keys and add the public key in the text box
- Now go to ~/.ssh and add a file called "config" by doing "touch config". Edit this file (with nano or vim) to look like this:
Host gitlab.orbit-lab.org
PreferredAuthentications publickey IdentityFile ~/.ssh/gitlab
- make sure the file path goes to the private key that you made, if you named it something other than gitlab.
- You should now be able to access orbit gitlab repositories through ssh.
- NOTE: If you have previously cloned a repository using HTTPS instead of SSH, then you will need to update the remote URL. Follow this guide: https://docs.github.com/en/get-started/getting-started-with-git/managing-remote-repositories
Project instructions:
- Clone the desired repository. You can click "clone" on the repository page for the desired repository on Gitlab (https://gitlab.orbit-lab.org/self-driving-car-2023). For example, "git clone git@…:self-driving-car-2023/upcar.git"
- Make sure you set up ROS in your .bashrc file. You need to have "source /opt/ros/noetic/setup.bash" in it
- You need to add the setup file for the catkin workspace of the project in your .bashrc file. To do this, first take note of the absolute path to the project directory. You will need to add "source path_to_catkin_ws/catkin_ws/devel/setup.bash" (replace path_to_catkin_ws with the actual absolute path on your system.)
- One you are done changing the setup files, logout and login.
- Install the joy package by following this tutorial: http://wiki.ros.org/joy/Tutorials/ConfiguringALinuxJoystick
- The user needs to have input permission for ROS to read data from joystick inputs. To set this, do "sudo usermod -aG input username". -aG specifies adding an additional group (input) to a user (username)
- Changing groups requires the user to log out and in for effects to take place
- The user needs to have input permission for ROS to read data from joystick inputs. To set this, do "sudo usermod -aG input username". -aG specifies adding an additional group (input) to a user (username)
- Install the realsense package (Add Link!)
ROS setup instructions:
- Follow these instructions to set up ROS Noetic: http://wiki.ros.org/noetic/Installation/Ubuntu
- Note: any time you edit a startup file, like .bashrc, you need to logout and login (on VM you might need to power it off and on) for the effects to take place.
- ROS Noetic is used for Ubuntu 20.04 (on RASCAL's Intel UpBoard), ROS Melodic is used for Ubuntu 18.04 (on SCAMP's and new RASCAL's Jetson Nano)
Using ROS:
- To run a ROS node: use "rosrun package nodename". For example, "rosrun joy joy_node" to run the joystick package.
- roscore must be running first by typing "roscore" in the terminal.
- To run multiple processes at once, you can use Tmux (terminal multiplexer). More advanced uses can be found online, but the basic usage is as follows:
- use "sudo apt install tmux" to install it.
- use "tmux" to launch. hit ctrl+b and then % to split vertically, and ctrl+b and then " to split horizontally.
- ctrl+b and then arrow key lets you switch between panes
- Mouse can be enabled if you use a tmux configuration file. Follow this: https://unix.stackexchange.com/questions/516800/how-do-i-enable-tmux-mouse-support
- To run a ROS launch file, use "roslaunch package launchfile.launch". For example, "roslaunch rascal sim.launch".
- on the first launch, or any time you change build files (CMakeLists, package.xml) you will need to navigate to the catkin workspace directory and run "catkin_make".
- To create a new package: http://wiki.ros.org/ROS/Tutorials/CreatingPackage
- To create a new python file that can be run through rosrun: Create the file somewhere within the package's src directory. Make sure the shebang is in the beginning ("#!/usr/bin/env python"). Then add the file to the package CMakeLists.txt in catkin_install_python.
- To be able to import python modules from within the same package: make sure catkin_python_setup() is uncommented in the package's CMakeLists.txt. Then reference the directory structure visible in the rascal package. Make sure to include the "setup.py" file in the package's directory.
VM setup instructions:
- Download virtualbox https://www.virtualbox.org/wiki/Downloads
- Download ubuntu https://releases.ubuntu.com/focal/
- Follow a guide to install ubuntu on Virtualbox: https://www.wikihow.com/Install-Ubuntu-on-VirtualBox
- If you are on mac, MAKE SURE you start Virtualbox by typing "sudo virtualbox" in the commandline, instead of starting it normally. This is the only workaround I could find to get controller input to work in virtualbox!
- If you are using Ubuntu 22.04, then there is a bug when using it with VirtualBox where you can't open the terminal by default. You need to update the locale of the machine for some reason. Follow this link to get into terminal mode: https://superuser.com/questions/100693/how-to-switch-to-non-graphical-view-in-ubuntu
- Then follow this to update the locale: https://askubuntu.com/questions/1435918/terminal-not-opening-on-ubuntu-22-04-on-virtual-box-7-0-0
- Restart the VM (power it off and on in VirtualBox) and now you should be able to open the terminal in Graphical mode
- You will need to have Sudo access on your account in Ubuntu. Follow this StackOverflow: https://askubuntu.com/questions/124166/how-do-i-add-myself-into-the-sudoers-group
- Run "sudo apt update" to update packages before doing things.
- If you want to be able to ssh into the VM through your local machine, you need to set up port forwarding on the VM. Follow this guide: https://help.skytap.com/connect-to-a-linux-vm-with-ssh.html
To run the simulation on a VM:
- Switch to the simulation branch by using "git checkout simulation" (in the upcar repository)
- Joystick needs to be configured in virtualbox. If you are using mac, the VirtualBox must be run as sudo.
- Follow these instructions: https://askubuntu.com/questions/25596/how-to-set-up-usb-for-virtualbox
- The joystick needs to be set. navigate to /dev/input. if you do "ls -a" in here, you should see js0 and js1 which are added by virtualbox by default, and then js2 if your controller is configured correctly. Run "sudo jstest /dev/input/js2" to test if it is working.
- You need to open a port to be able to view the web display on the host machine of the VM.
- Open virtualbox and go to the settings for the VM.
- Go to network → Adapter 1 → make sure it is on NAT
- click advanced → port forwarding → add a new port forwarding rule
- Fill the following fields (you may need to scroll right): name: flask, Host port: 5001, Guest port: 5000
- note: you can use any unused port instead of 5001. 5000 is used for the guest port as it is the default for flask.
- you may need to restart the VM.
- run simlaunch.sh
- To view the website, go to 127.0.0.1:5001 on your web browser
Other Notes:
- For cars built on the Jetson Nano (ie SCAMP and new RASCAL), the use of ROS Melodic requires that any python files need to be marked as executable. To do this, run "sudo chmod +x path/to/file.py". This adds execute permission (+x) to the file.
- If you run out of space on ubuntu vm: https://technology.amis.nl/platform/virtualization-and-oracle-vm/ubuntu-vm-virtualbox-increase-size-disk-make-smaller-exports-distribution/
Instructions for MIMIC (aka SCAMP)
Clone repository into /opt
Teensy 4.0 Setup:
- copy the links for the downloadable tar files for arduino-cli and teensy_loader_cli
→ https://github.com/arduino/arduino-cli/releases
→ https://github.com/PaulStoffregen/teensy_loader_cli/releases
- Use wget to download the tar files and extract them with tar -xvf {filename}
- Follow the instructions on the teensy_loader_cli github (Run
sudo apt-get install libusb-dev
for Ubuntu, andmake
to compile to an executable)
- Move the two executable files to "/bin" to add it to path (makes it so that the executable can be run from any directory)
→ This allows the ./teensy-tool.sh file in mimic/teensy to compile and upload to the teensy
- Refer to https://arduino.github.io/arduino-cli/0.33/getting-started/#adding-3rd-party-cores to add Teensy support to arduino-cli
→ You may have to create a file called arduino-cli.yaml in ~/.arduino15 with information about additional urls, format is provided in the link above
→ https://www.pjrc.com/teensy/td_download.html
→ https://www.pjrc.com/teensy/package_teensy_index.json
- Follow the instructions to create the 00-teensy.rules file:
→ https://www.pjrc.com/teensy/00-teensy.rules
- Use arduino-cli board list command to find the port of the Teensy
→ Modify the sketch.yaml file in the ./teensy directory to makes sure default_port and default_fqbn matches what is listed from arduino-cli board list
- Upload teensy code by running ./teensy-tool -bu in mimic/teensy/ to build and upload the code
→ additionally, you can add the -m flag to open the monitor as well
ROS Melodic Setup:
Uses ROS Melodic for Ubuntu 18.04 on the Jetson Nano
- Refer to http://wiki.ros.org/melodic/Installation/Ubuntu for install
- Add the following lines to ~/.bashrc
- source/opt/ros/melodic/setup.bash
- source/opt/mimic/devel/setup.bash
- cd into mimic repo and run catkin_make to create ROS environment
- Run ROS nodes by running ./roslaunch.sh in mimic/
→ check ports for teensy for serial communication → check joystick connection
- Making new nodes/msg
→ List out .py files and .msg files as needed in the CMakeLists.txt file located in the package folder (ie test_pkg) → Run catkin_make to refresh environment messages/nodes
Integration with Web Display:
- Use ip a to find the ip of the Jetson Nano
- Open a web brower and go to {ip}:5000 to view
Troubleshooting
- "SerialException: Device reports readiness to read but returned no data"
ROS will not be able to communicate with serial if there is another service using the port, i.e. Serial Monitor
- Use lsof to check with programs are using the port
lsof /dev/ttyACM0
- Kill the program that is using the port
kill {PID}
- "socket.error: [Errno 48] Address already in use"
Flask server was not properly stopped before starting it again
- Close the connection
sudo fuser -k 5000/tcp