Table of Contents
Tutorial 2: Multiple Controllers
In the OpenFlow model, traffic can be separated along many different boundaries and flows can be controlled by multiple controllers. In this particular example we will run two separate controllers. The first one will handle the production traffic, and run a commodity controller, SNAC. The second controller will manage the "experimental" bit-torrent traffic, and run a NOX controller. The data interfaces (eth0) of 4 nodes are connected to the OpenFlow switch (in this case NEC IP8800). The OpenFlow switch is pre-configured to speak to a controller at console.sb9.orbit-lab.org:6633.
The logical setup should look like:
Controllers
Splitter: Flowvisor - IP port 6633
The flowvisor tool serves as an intermediary between controllers. It splits traffic based on predefined rules. For this example we're running flowvisor on the SB9 console. While it is completely possible to compile your own flowvisor from scratch and install it in your home directory, a precompiled installation is located in /opt. In order to properly steer the flowvisor tool, you must populate a flovisor-config.d directory with files that configure the flowvisor behavior. This directory must be a subdirectory of the current working directory, as flow visor will create statistics files and logs in the current working directory. For the purposes of this demo we'll be using these files:
bittorent.guest default.switch routing.guest
The default file primes the flowvisor and contains only the following lines:
# For switches that do not have a their own, specific config file Default: 1 # Start numbering "default" switches at 10000 Id: 10000
The routing.guest file directs production traffic to SNAC running on port 6634, it denies high numbered IP traffic from getting to port 6634:
Name: routing ID: 1 Host: tcp:localhost:6634 # By default, with now "FlowSpace" param, this slice has # permissions to affect all traffic FlowSpace: deny: tp_src: 10001 FlowSpace: deny: tp_dst: 10001 FlowSpace: deny: tp_src: 10002 FlowSpace: deny: tp_dst: 10002 FlowSpace: deny: tp_src: 10003 FlowSpace: deny: tp_dst: 10003 FlowSpace: deny: tp_src: 10004 FlowSpace: deny: tp_dst: 10004 FlowSpace: deny: tp_src: 10005 FlowSpace: deny: tp_dst: 10005 # limit 10000 == at most 10000 packet_in's per second == infinity FlowSpace: allow: limit: 10000
Finally the bittorent.guest directs high numbered IP port traffic to a controller on 6635:
Id: 1001 Host: tcp:localhost:6635 FlowSpace: allow: tp_src: 10001 limit: 10000 FlowSpace: allow: tp_dst: 10001 limit: 10000 FlowSpace: allow: tp_src: 10002 limit: 10000 FlowSpace: allow: tp_dst: 10002 limit: 10000 FlowSpace: allow: tp_src: 10003 limit: 10000 FlowSpace: allow: tp_dst: 10003 limit: 10000 FlowSpace: allow: tp_src: 10004 limit: 10000 FlowSpace: allow: tp_dst: 10004 limit: 10000 FlowSpace: allow: tp_src: 10005 limit: 10000 FlowSpace: allow: tp_dst: 10005 limit: 10000
This setup of flowvisor will split along IP port boundaries. If we want to direct more traffic to the experiment, we simply manipulate the flow rules. To start the actual controller in user:
user@console.sb9:~$ /opt/flowvisor-0.4/flowvisor ptcp:6633 Mar 14 22:57:08|00001|rconn|WARN|tcp:localhost:6635: connecting... Mar 14 22:57:08|00002|rconn|WARN|tcp:localhost:6635: connection failed (Connection refused) Mar 14 22:57:08|00003|rconn|WARN|tcp:localhost:6635: connection dropped (Connection refused)
You might notice some connection failures for port 6635, since the controller isn't running yet.
Production: SNAC - IP port 6634
On console.sb9 SNAC is preconfigured to listen to port 6634. For the purposes of this example, SNAC will handle the production traffic and as such is not configured to do anything beyond the scope of a simple learning switch. The web interface for SNAC is accessible externally at https://sb9.orbit-lab.org (default credentials).
Experimental: NOX - IP port 6635
For demonstration purposes we will have this controller simply pass traffic. We could manipulate this traffic by configuring NOX to behave differently, or by running a completely different controller that listens on port 6635. Since our model is that "production" traffic is handled by a controller at port 6634 and experimental traffic is handled by a controller at port 6635, it is expected that the states of the port 6635 controller be transient.
Using my compiled nox I ran
./nox_core -i ptcp:6635 switch packetdump
The experimental Traffic
FIXMEThis process will probably be Scripted.
Our experiment setup is as follows (refer to diagram above):
- Node1-1: Tracker 192.168.1.1:6969
- Node1-2: Seeder 192.168.1.2:10001-10005
- Node1-3: First downloader 192.168.1.3:10001-10005
- Node1-4: Second downloader (starts 1 Minute later to pull from multiple sources) 192.168.1.4:10001-10005
A torrent file and data file were created on node1-1 and the respective components moved to each node using the control interface.
root@node1-1:~# dd if=/dev/zero of=./data.mp3 bs=100MB count=1 root@node1-1:~# btmakemetafile data.mp3 http://192.168.1.1:6969/announce root@node1-1:~# scp data.mp3 root@node1-2: root@node1-1:~# scp data.mp3.torrent root@node1-2: root@node1-1:~# scp data.mp3.torrent root@node1-3: root@node1-1:~# scp data.mp3.torrent root@node1-4:
We then start the tracker on node1-1 and when the process is running we see:
root@node1-1:~# bttrack --port 6969 --dfile tracker.log 192.168.1.4 - - [14/Mar/2010:19:44:46] "GET /announce?info_hash=%7Es%D0%B1%03%E5%02%BA%B6c_%7D%D6%DB%E1%A9%F5%CD%BC%DE&peer_id=M3-4-2-- 7f9e2d5669d8&port=10001&key=8a1ab764&uploaded=0&downloaded=0&left=1000000000&compact=1&event=started HTTP/1.1" 200 33 "-" "Python-urllib/2.6" 192.168.1.4 - - [14/Mar/2010:19:44:47] "GET /announce?info_hash=%7Es%D0%B1%03%E5%02%BA%B6c_%7D%D6%DB%E1%A9%F5%CD%BC%DE&peer_id=M3-4-2-- 7f9e2d5669d8&port=10001&key=8a1ab764&uploaded=0&downloaded=0&left=1000000000&compact=1&event=stopped HTTP/1.1" 200 33 "-" "Python-urllib/2.6" ...
The seeders and downloaders on the respective nodes output:
root@node1-2:~# btdownloadheadless --minport 10001 --maxport 10005 data.mp3.torrent saving: data.mp3 (953.7 M) percent done: 0.0 time left: checking existing file download to: /root/data.mp3 saving: data.mp3 (953.7 M) percent done: 0.2 time left: checking existing file download to: /root/data.mp3 ...
root@node1-4:~# btdownloadheadless --minport 10001 --maxport 10005 data.mp3.torrent saving: data.mp3 (953.7 M) percent done: 0.0 time left: download to: /root/data.mp3 download rate: 0.00 K/s upload rate: 0.00 K/s download total: 0.0 M upload total: 0.0 M saving: data.mp3 (953.7 M) percent done: 9.5 time left: 5 min 55 sec download to: /root/data.mp3 download rate: 2497.60 K/s upload rate: 665.95 K/s download total: 91.5 M upload total: 19.8 M ...
root@node1-3:~# btdownloadheadless --minport 10001 --maxport 10005 data.mp3.torrent saving: data.mp3 (953.7 M) percent done: 1.9 time left: 6 min 57 sec download to: /root/data.mp3 download rate: 2556.24 K/s upload rate: 223.78 K/s download total: 18.8 M upload total: 2.5 M ...
Finally on the NOX controller we should see events like this:
NOX 0.8.0~full~beta (nox_core), compiled Mar 14 2010 16:21:17
Compiled with OpenFlow 0x97 (exp)
[(Giga-Byte Technology Co.,LTD.):84:9b:30>(Giga-Byte Technology Co.,LTD.):83:a0:5c:IP]([v:4hl:5l:60t:64]TCP cs:8a65[192.168.1.1>192.168.1.2]){44653>10001} seq:3830918239 ack:0 f:0x2
[(Giga-Byte Technology Co.,LTD.):83:a0:5c>(Giga-Byte Technology Co.,LTD.):84:9b:30:IP]([v:4hl:5l:40t:64]TCP cs:b76c[192.168.1.2>192.168.1.1]){10001>44653} seq:0 ack:3830918240 f:0x14
[(Giga-Byte Technology Co.,LTD.):84:9b:30>(Giga-Byte Technology Co.,LTD.):85:9b:64:IP]([v:4hl:5l:60t:64]TCP cs:8d0d[192.168.1.1>192.168.1.4]){45508>10001} seq:1710024440 ack:0 f:0x2
...
ASIDE: TESTS for connectivity
In case your experiment is failing, here are a few sanity checks.
The SNAC controller Web interface
Assuming your flowvisor is passing along high priority traffic to snac, the summary should look like the this:
Inter-node traffic
Once the data interfaces are brought up on each of the 4 nodes, they should be discovered by SNAC. Assuming we place them all in the same IP subnet, it should be possible to ping each node from every other node. A simple test can be done as follows:
for node1-i: ssh root@node1-i ifconfig eth0 up ifconfig eth0 192.168.1.i ping 192.168.1.(i-1) where i runs from 1 to 4.
This should be done with independent sessions, as each session will start to display traffic information. Once completed SNAC should report an increase in production traffic, and 4 discovered hosts with the specified IPs.
High Numbered port Test
Start a fresh NOX controller instance on sb9 and enable packet dumps
root@console:/root/nox/build/src# ./nox_core -i ptcp:6635 switch packetdump
Attempt to connect to a high numbered port on a different node (using the Ip of the data interface):
root@node1-1:~# telnet 192.168.1.2 10001 Trying 192.168.1.2... telnet: Unable to connect to remote host: Connection refused
If the traffic is split properly, your controller should register events similar to this:
[(Giga-Byte Technology Co.,LTD.):84:9b:30>(Giga-Byte Technology Co.,LTD.):83:a0:5c:IP]([v:4hl:5l:60t:64]TCP cs:8a65[192.168.1.1>192.168.1.2]){44653>10001} seq:3830918239 ack:0 f:0x2
[(Giga-Byte Technology Co.,LTD.):83:a0:5c>(Giga-Byte Technology Co.,LTD.):84:9b:30:IP]([v:4hl:5l:40t:64]TCP cs:b76c[192.168.1.2>192.168.1.1]){10001>44653} seq:0 ack:3830918240 f:0x14
