wiki:Internal/OpenFlow/HostSetup

Version 23 (modified by ssugrim, 15 years ago) ( diff )

—

return to OpenFlow index

Setting up the host for the NetFPGA

Here we provide the links and directions followed to set up a working NetFPGA host. The directions can be found on the following site: ​http://netfpga.org/netfpgawiki/index.php/Guide

Installing CentOS

CentOS is the recommended OS for use with NetFPGA. The most current version is 5.3, which is not listed as a version of CentOS that works with NetFPGA, but was used anyways.

CentOS 5.3

The latest version is CentOS 5.3.

For the installation, the net install iso was chosen because of its small size. This can be found on any mirror site.

CD install of CentOS 5.2

The .iso images can be found here: ​http://vault.centos.org/5.2/isos/i386/ You only need CD's 1 through 4. k3b on Ubuntu was used to burn the images since the images burned with Windows seemed to become corrupt. It seems that you need to manually configure the network interface for eth0.

Before installing the netFPGA base package, make sure to update the kernel, since the dependencies for NetFPGA software require a newer version of the kernel than the one for 5.3. To install the basic drivers, install teh yum repository and GPG key, followed by the netFPGA base package. 

rpm -Uhv http://netfpga.org/yum/el5/RPMS/noarch/netfpga-repo-1-1_CentOS5.noarch.rpm
yum install netfpga-base
/usr/local/NF2/lib/scripts/user_account_setup/user_account_setup.pl

The directions then go over installing CAD tools, but since we are just trying to make the FPGA into an !OPenFlow switch, we skip over that section of the guide. cd into the NetFPGA directory, then make, make install, and reboot. 

make
#a bunch of output showing status
make install
#more outputs
reboot

after the machine has rebooted, check that drivers are loaded. The PID should show up: 

[root@localhost ~]# /sbin/lsmod | grep nf2
nf2                    22540  0

Finally, verify the interfaces with /sbin/ifconfig -a at the terminal. This is the output for our setup: 

eth0      Link encap:Ethernet  HWaddr 00:1C:C0:B4:EC:D4  
          inet addr:192.168.202.102  Bcast:192.168.203.255  Mask:255.255.252.0
          inet6 addr: fe80::21c:c0ff:feb4:ecd4/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:825 errors:0 dropped:0 overruns:0 frame:0
          TX packets:212 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:76080 (74.2 KiB)  TX bytes:21489 (20.9 KiB)
          Interrupt:50 Base address:0xa000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:1162 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1162 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:2004680 (1.9 MiB)  TX bytes:2004680 (1.9 MiB)

nf2c0     Link encap:Ethernet  HWaddr 00:4E:46:32:43:00  
          BROADCAST MULTICAST  MTU:1986  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
          Interrupt:58 

nf2c1     Link encap:Ethernet  HWaddr 00:4E:46:32:43:01  
          BROADCAST MULTICAST  MTU:1986  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
          Interrupt:58 

nf2c2     Link encap:Ethernet  HWaddr 00:4E:46:32:43:02  
          BROADCAST MULTICAST  MTU:1986  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
          Interrupt:58 

nf2c3     Link encap:Ethernet  HWaddr 00:4E:46:32:43:03  
          BROADCAST MULTICAST  MTU:1986  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
          Interrupt:58 

sit0      Link encap:IPv6-in-IPv4  
          NOARP  MTU:1480  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

NetFPGA on Ubuntu (11/7)

Ultimately, the second NetFPGA host was built using Ubuntu 8.04 (kernel 2.6.24-16). To make things neat, user 'netfpga' was created for this process. Make sure the user is added to the sudoers list.

installing NetFPGA drivers

First off, you need an account at netfpga.org to download the .tar files needed to install the netfpga drivers. The files needed are netfpga_base_beta_x_x.tar.gz and netfpga_lib.tar.gz. With an account, they can be downloaded from here: ​http://netfpga.org/beta/distributions/

Untar and the files with tar -xvf <filename>. They will untar into one directory called NF2. Do not make and install yet - At this point, follow the directions on the ​Ubuntu compatibility page. They ask you to make a few changes on the files located in NF2. A few things to note regarding the directions:

  • Environmental variables can be added by the command 
    /usr/local/NF2/lib/scripts/user_account_setup/user_account_setup.pl
    as per the ​Guide, however you might need to edit /usr/local/NF2 in user_account_setup.pl to specify the exact path to your NF2 directory: 
    else
{
  print "Copying the NF2 directory to your user account\n";
  `cp -r /home/netfpga/test/NF2 ~`;
}

print "Adding the NetFPGA Enviornment Variables to your .bashrc\n";
`cat /home/netfpga/test/NF2/bashrc_addon >> ~/.bashrc\n`;

Then you can make and install the drivers. Note, the selftest can't be run in Ubuntu.

installing OpenFlow on NetFPGA

General directions are found here: ​http://netfpga.org/netfpgawiki/index.php/OpenFlow-NetFPGA-090

In order to install OpenFlow on the FPGA you need both the netFPGA OpenFlow switch driver and the actual OpenFlow software.

Don't bother with the ./install.sh stuff under the Usage section of the directions found at netfpga.org, since they are custom-fitted for CentOS. Instead, the directions found at ​openflowswitch.org should be referenced. The directions need to be modified slightly for Ubuntu:

  1. Install Dependencies

The following are needed to install everything properly (you should be able to apt-get install these):

  • git-core (recommended)
  • curl - dependency for Git that does not get installed properly due to a known bug in some versions of Ubuntu
  • autoconf
  • automake
  • libtool
  • gcc
  • pkg-config

The stuff needed to build kernel modules:

  • linux-kernel-devel
  • kernel-package
  • linux-libc-dev
  1. Install

The commands, as per the CentOS install page at openflow.org: 

git clone git://openflowswitch.org/openflow
cd openflow
./boot.sh

Reboot the machine, then build the kernel modules: 

./configure --with-l26=/lib/modules/`uname -r`/build --enable-hw-tables=nf2
make
sudo make install
  1. Check that all is well/ the kernel modules exist:

the command 

ls <path-to-your-openflow-directory>/datapath/linux-2.6 | grep ofdatapath

should bring up the following 2 things:

  • ofdatapath.ko
  • ofdatapath_netfpga.ko

Regression testing

Connecting the SATA feedback loop seems to prevent the test from loading correctly, throwing the following error: 

[root@localhost ~]# /root/NF2/lib/C/download/nf2_download ~/NF2/bitfiles/selftest.bit
Found net device: nf2c0
Bit file built from: nf2_top_par.ncd
Part: 2vp50ff1152
Date: 2008/ 7/21
Time: 17: 0:39
After resetting Programming interface, expected status to be 1 (FIFO empty).
However status & 0x102 is 0x102

Fatal Error, exiting...

In addition, the Ethernet cables used for the cross-connection should be crossover, not straight through. Once the SATA cable was removed, and the test run with the command 

~/NF2/projects/selftest/sw/selftest -n

the proper output was returned.

Installing the OpenFlow implementation (OpenFlow-NetFPGA-090)

After successfully installing the drivers, you can install the OpenFlow switch with the following command: 

[root@localhost ~]# yum install netfpga-openflow_switch

Setting up the NetFPGA host

The goal of this project is to set up two NetFPGA hosts as OpenFlow switches capable of tunneling. The hosts we are going to be using are the NetFPGA pre-built quad-core rack PCs (​found here). Although the PC's come with CentOS pre-installed with the proper drivers needed for the NetFPGAs, we want to use Ubuntu. This section will document the steps needed to build a host running Ubuntu 9.10 Server Edition.

Note: We saved the HDD w/ CentOS that came with the PC.

References

Following pages will be used to set the hosts up.

Host info

  • Distro used for PC: Ubuntu 9.10 Server Edition 32bit
  • IPaddr: 192.168.203.187 (this is dynamic so it might change)
  • user: netfpga

Setup Instructions

1. Installing Ubuntu

  1. Change boot priority of CD drive in BIOS - default will not let you boot into the install CD

  1. Follow most of the defaults, except for the following:
    • Disk partitioning - choose regular automatic partitioning
    • When prompted with the option to install additional packages, install OpenSSH-server
  1. Update packages
  1. Install dependencies for Ubuntu compatibility, as per the ​compatibility page. The packages are:

2. Installing NetFPGA Drivers

Steps described by the NetFPGA tunneling page:

  1. Install base package
  2. Install OpenFlow components

1. Installing base package

  1. Download the tarballs from ​http://netfpga.org/beta/distributions/

For this you need a FosWiki account (register ​here). Download the newest netfpga_base_beta package (currently 2.0.0). Untar:

tar -xvf netfpga_base_beta_2_0_0.tar.gz

Untarring the package will create the directory NF2.

  1. Edit files/ symlinks as per compatibility page

Create links: 

ln /usr/bin/lspci /sbin/lspci
ln /usr/bin/setpci /sbin/setpci
ln -s /usr/lib/libpcap.so.0.8 /usr/lib/libpcap.so.0.8.3

test_loopback_drop/run has been renamed run.pl in the most recent package. In this file you just need to change 'usleep 250' to 'sleep 0.00025'.

NF2/lib/C/kernel/Makefile is already in working condition - you don't need to modify it.

*Stopped at this point: The IRQF_SHARED flag is deprecated in kernels greater than 2.6.24. Instead, the SA_SHIRQ flag is used in include/linux/interrupt.h . in Ubuntu compatibility page

  1. Allocate more kernel memory

The NetFPGA requires 256M memory allocated. This can be done through /boot/grub/menu.lst on older kernels. For the newer versions of Ubuntu (e.g. 9.10), which uses Grub 2, you need to edit /boot/grub/grub.cfg from 

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash

to 

GRUB_CMDLINE_LINUX_DEFAULT="vmalloc=256M
  1. Edit and run user_account_setup.pl

The script sets environmental variables. It is located under /NF2/lib/scripts/user_account_setup/ . Running it as is will throw an error saying

cat: /usr/local/NF2/bashrc_addon: No such file or directory

The easiest thing to do is to edit the script to contain the exact path to your NF2 directory and bashrc_addon: 

else
{
  print "Copying the NF2 directory to your user account\n";
  `cp -r /home/netfpga/NF2 ~`;
}

print "Adding the NetFPGA Enviornment Variables to your .bashrc\n";
`cat /home/netfpga/NF2/bashrc_addon >> ~/.bashrc\n`;

Then it will run correctly (here NF2 is already in the home directory) 

$ ./user_account_setup.pl NF2 directory already exists copy has been canceled
Adding the NetFPGA Enviornment Variables to your .bashrc

trying to use the 9.10 compatible stuff, found in ​http://www.netfpga.org/foswiki/bin/view/NetFPGA/OneGig/RealTimeSwitch

Installing Ubuntu, take 3 take 4

After installing 9.10 server (AMD64) and 9.10 server 32 bit, we realized that there may be a "few" complications coming from the changes that were made in the more recent versions of Ubuntu (e.g. Grub 2, discarding "antiquated" libraries, ect), so things were rolled back to installing Ubuntu 9.04 server 32 bit. This should simplify *many* things. Hopefully.

1. Installation of the OS

  1. Go to the BIOS settings, make CD drive highest boot priority
  1. Follow the default choices given for intallation, save the following few:

  • no setup of LVM for disk partitioning
  • additional packages - install OpenSSH server
  • hostname: netfpga1
  • user: netfpga

  1. When rebooting, go back to the BIOS settings to make sure the HDD is included in the boot priority list. 'x' adds excluded devices to the boot priority list.
  1. Who's afraid of a little command line? If you are, y' shoulda stuck with desktop edition.

Host info

From past experience, the following info will come in handy when troubleshooting stuff…

  • uname -a shows: 2.6.28-11-server i686 GNU/Linux
  • IP address: 192.168.203.187 (DHCP - May change)
  • bootloader: grub

2. Installing drivers (NetFPGA)

The host is named "Netfpga1" because it will be a NetFPGA host. So this is sort of important.

  1. Get a Foswiki account so you can download the packages you need at their ​registration page.
  1. Update packages, then install the following dependencies as per the ​Comaptibility page:
    • build-essential
    • linux-headers-uname -r
    • ncurses-dev
    • libnet1-dev
    • libxml-simple-perl
    • libio-interface-perl
    • liblist-moreutils-perl
    • liberror-perl
    • libnet-perl-rawip
    • sun-java6-jre
    • sun-java6-jdk
    • libpcap0.8
    • libpcap0.8-dev
  1. Download the base package from ​http://netfpga.org/beta/distributions/. The page will prompt you for your FosWiki username and password. The newest is netfpga_base_beta_2_0_0. Untar: 
    tar xvf netfpga_base_beta_2_0_0.tar.gz

This will create a directory called NF2 in your home directory.

  1. create symlinks: 
    ln /usr/bin/lspci /sbin/lspci
ln /usr/bin/setpci /sbin/setpci
  1. Acquire libpcap.so.0.9.4 (find on bottom of this page), slap it into /usr/lib, and create symlinks, as such: 
    ln -s /usr/lib/libpcap.so.0.9.4 /usr/lib/libpcap.so.0.9 
ln -s /usr/lib/libpcap.so.0.9.4 /usr/lib/libpcap.so.0
ln -s /usr/lib/libpcap.so.0.9.4 /usr/lib/libpcap.so
  1. Edit files by following the compatibility page. Some of the stuff there is outdated, so keep an eye out for the following:
  • /test_loopback_drop/run is /test_loopback_drop/run.pl
  • edit "usleep 250" to "sleep 0.00025" in /test_loopback_drop/run.pl
  • Don't change anything on /NF2/lib/C/kernel/Makefile
  • Likewise, NF2/lib/C/download/nf2_download.c need not be edited.
  1. Edit /boot/grub/menu.lst to allocate more kernel memory: 
    title    Ubuntu 9.04, kernel 2.6.28-16-generic 
uuid    (BIG number here)
kernel  /vmlinuz-2.6.28-16-generic root=UUID=(BIG number here) ro quiet splash vmalloc=256M
  1. Edit /home/netfpga/NF2/projects/router_kit/sw/rkd.cc to include limits.h
  1. Replace asm/semaphores with linux/semaphores in:
    • /home/netfpga/NF2/lib/C/kernel/nf2main.c
    • /home/netfpga/NF2/lib/C/kernel/nf2_control.c
  1. make and make install in user's NF2 directory (you need to be root)

3. Regression testing

The selftest will not work on Ubuntu, so we skip right to the Regression tests.

  1. Reprogramming the CPCI

In cpci_reprogram.pl line 216, change the offset from 5 to 4: 

# Search for the NetFPGA device ID
                if ($words[4] eq $NF2_device_id)           <<---over here
                {
                        push (@device_list, $words[0]);
                }

Then run cpci_reprogram.pl (run from user home directory): 

/home/netfpga/NF2/lib/scripts/cpci_reprogram/cpci_reprogram.pl --all

Add the same command to /etc/rc.local so this is done automatically each time the machine is rebooted.

  1. Rename interfaces.

Since you need eth1 and 2 for the regression test, you need to change eth0 to eth2. Change "eth0" to "eth2" in /etc/udev/rules.d/70-persistent-net.rules so it looks similar to this: 

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:30:48:b1:9d:a9", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

# PCI device 0x8086:0x109a (e1000e)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:30:48:b1:9d:a8", ATTR{type}=="1", KERNEL=="eth*", NAME="eth2"

The changes will take effect after you reboot the machine.

  1. Much is hard-coded for CentOS. Change /sbin/arp to /usr/sbin/arp in NF2/projects/router_kit/regress/test_03_arp_table/setup
  1. running the regression test

The following command can be used to run the tests, which will take about 45 -60 min (according to the Guide). 

~/NF2/bin/nf2_regress_test.pl

Expect some things to fail; the tests even fail on the CentOS pre-installed.

4. Installing OpenFlow components

Prequisite packages

  • autoconf
  • libtool
  • pkg-config
  • linux-source (2.6.28)

Following the instructions of ​Labsetup we cloned the 0.8.9 branch of the their git repository, I'll include the steps because their pages don't stay up very long. 

 $ git clone git://openflowswitch.org/openflow.git
 $ cd openflow
 $ git checkout -b openflow_089_rev4 origin/release/0.8.9

Assuming all the prerequisites are installed, the next step is to compile the source. We're trying to make two kernel objects, and a bit file for the netfpga. 

 $ ./boot.sh
 $ ./configure --with-l26=/lib/modules/`uname -r`/ --enable-hw-tables=nf2  
 $ make

boot.sh will fail to run if the listed prerequisites are not installed (there may be more preqs based on what was installed prior, build-tools, kernel headers, etc…).

The original configure line read "./configure —with-l26=—with-l26=/lib/modules/uname -r/build —enable-hw-tables=nf2", while the extra —with-l26 is an obvious error, there are others. The configure script stuffs the argument of —wtih-l26 into a $path variable, then looks for the subdirectories $path/build and $path/source. Specifying —with-l26 as ../uname -r/build causes configure to fail, since there is no source directory in the build directory. Instead what they wanted was the directory 1 level up.

In our first attempt we just ran make and went on to the next step, however when we tried to insert the modules one of them failed. The problem was with the ofdatapath_netfpga.ko module. When we tried to insert it (with insmod) it would complain about a missing symbol. Dmesg would show an even every time you tried to insert the module that said 

WARNING: "nf2k_reg_read"
WARNING: "nf2k_reg$ ./boot.sh

Consulting a reference from the Stanford [$ ​https://mailman.stanford.edu/pipermail/openflow-discuss/2009-January/000051.html mailing list] on netfpga, we determined that the Module.symvers file needed to be edited. We needed to copy the entries from the ~/NF2/lib/c/kernel/Modules.symvers into the ~/openflow/datapath/linux-2.6/Modules.symvers. The NF2 directory comes from two tar files:

  • netfpga_base_beta_2_0_0_0.tar.gz
  • netfpga_openflow_switch.0_8_9-1.tar.gz - This file was chosen to match the version of the Netfpga openflow source we're compling.

Go to ​http://netfpga.org/beta/distributions/ in order to download these packages (you'd need your FosWiki account).

Making this change produces the bits files and modules that will be needed in the next section.

After a sucessfull complie we needed to load the openflow bitfile onto the netfpga 

 # cpci_reprogram.pl 
 # nf2_download ~/openflow/datapath/hwtable_nf2/openflow_switch.bit

The actual paths may vary but the basic process is the same. This is fairly straight forward assuming the netfpga tools were already built.

Once that was done we inserted the modules (required root privileges) 

# /sbin/insmod ./openflow/datapath/linux-2.6/ofdatapath.ko 
# /sbin/insmod ./openflow/datapath/linux-2.6/ofdatapath_netfpga.ko

At this point we're ready to try testing the netfpga as a open flow switch. I'm going to pause here and rearrange the networks that the specfic netfpgas belong to.

The host has been renamed to of1 and was given an official place in the WINLAB network with IP addr. 192.168.200.86

Next, you create the OpenFlow datapaths. What they refer to as ./utilities can be found under your openflow directory. We didn't assign a MAC address to the interface like they did in the tutorial. 

# ./utilities/dpctl adddp nl:0

Then, add the NetFPGA interfaces. Here we encounter another typo in the instructions - there is no nf2c4. 

# ./utilities/dpctl addif nl:0 nf2c0
# ./utilities/dpctl addif nl:0 nf2c1
# ./utilities/dpctl addif nl:0 nf2c2
# ./utilities/dpctl addif nl:0 nf2c3

Then you make the OpenFlow module listen to your controller. For this setup the controller is a PC running SNAC with the IP address 192.168.203.75. secchan can be found under your openflow directory. 

# ./secchan/ofprotocol nl:0 tcp:192.168.203.75:6633




return to OpenFlow index

Attachments (3)

Download all attachments as: .zip

Note: See TracWiki for help on using the wiki.