= Notes = This section covers discussions and some background work that went into this project. == VLANs == VLANs are a way to split up broadcast domains at L2. They can be statically or dynamically set, with dynamic VLANs sort of working in a similar way to DHCP. == DHCP Relay agents == Relay agents are virtual layer 3 devices residing on a switch with VLANs. In this case, they allow DHCP broadcasts to be relayed across 2 VLANs on the same switch. This is not necessary for us because the DHCP server lives beyond the ASA, and the ASA not only acts as a router between the switch and the server, but also serves to manipulate the VLAN tags (masquerades as hosts sending out DHCP requests). == Trunking == Trunking is done by making a "backbone" for all the different VLANs on a local switch to converge to. Frames from different VLANs are identified bythe means of VLAN tags, a four-byte addition to the Ethernet frame identifying which VLAN it came from. The tagging is also known as 802.1q. == Native VLANs == Normally the 4 byte addition to the Ethernet frame would cause the tagged frame to be rejected on a VLAN, and regular frames would be rejected if found in the trunk due to the lack of a tag. This is unless the "regular" frame is from a '' native VLAN '', whose frames are intended not to have tags for devices that don't understand tagging. A site explaining what a native vlan is: http://networkers-online.com/blog/2008/06/native-vlan-explained/ In our case this matters because DHCP coming from the trunk needs to assign IP addresses to both the hosts connected to the switch on different VLANs (VLAN 27) and to the switch itsself. To do this, The virtual interface to which the IP address is assigned needs to be on the trunked port(s). === some thoughts regarding th virtual switches (6/9) === * They are overlay devices that "run" on top of VLANs. VLANs need to be set up before virtual switches can be created. * They are identified by the VLAN number of the VLAN they overlay, so one virtual switch can't encompass more than 1 VLAN. * Conversely, they can overlay part of a VLAN. In that case, the whole VLAN takes on the virtual switch's behavior. The last point is somewhat annoying, because the default behavior of a controllerless virtual switch is to "wait till the flow table times out." That would mean "become nonfunctional" in our case, since the virtual switches start off controller-less, and hence, flow-less. The whole switch would become a virtual brick if the trunk is specified in `setvsi`. This was demonstrated by overlaying a virtual switch over a single port on VLAN 27. It stopped forwarding DHCP messages all together when it used to be the only VLAN with DHCP. Questions: * can a virtual switch be loaded with a default flow table that would allow it to function, at least until its table times out? === when openflow.conf gets messed up (6/22) === One method to rewrite the !OpenFlow configuration is to delete openflow.conf directly through the switch, then use the `setvsi` command to set the new virtual switches up. This should be unnecessary, but it happened to be effective when old configurations did not seem to go away despite editing the .conf file using a PC. {{{ > cd /mnt/ > rm openflow.conf remove 'openflow.conf'? y > setvsi 1 1,3,5,7,9,11,13,15,2.1 tcp 172.16.4.224 dpid 0x0123456789ab > setvsi 2 17,19,21,23,25,27,29,31,2.2 tcp 172.16.4.180 dpid 0x012345678abc > setvsi 3 33,35,37,39,41,43,45,47,2.3 tcp 172.16.4.64 dpid 0x01234567abcd > showswitch vlan ports secure channel ---- ----- -------------- 1 1, 3, 5, 7, 9, 11, 13, 15, 2.1 disconnected 2 17, 19, 21, 23, 25, 27, 29, 31, 2.2 disconnected 3 33, 35, 37, 39, 41, 43, 45, 47, 2.3 disconnected }}} However, it seems like it is not possible to set `double-wide-mode`, or `no-save` through the CLI. These have to be added by editing openflow.conf using the PC. === SSL setup, things to know for the next day. (7/13) === using this site as reference: http://www.debian-administration.org/article/Creating_and_Using_a_self_signed__SSL_Certificates_in_debian except renaming the output .pem files to what it would understand: *ca_cert.pem - from cacert.pem *sw_cert.pem - from cert.pem *sw_key.pem - from key.pem for the Common Name, I just used the IP address of the CA, 192.168.203.75. because in the example on the site ca_cert.pem is named cacert.pem, I had to change "cacert" to "ca_cert" in openssl.cnf for the very last part (signing the certificate): {{{ [ CA_default ] serial = $dir/serial database = $dir/index.txt new_certs_dir = $dir/newcerts certificate = $dir/ca_cert.pem #change over here private_key = $dir/private/cakey.pem default_days = 365 default_md = md5 preserve = no email_in_dn = no nameopt = default_ca certopt = default_ca policy = policy_match }}} I am not sure if changing the name of the key from ofpswitch.key.pem to sw_key.pem after making/signing the certification will affect anything. Some lessons for the day: [[BR]] * the SD card will show up as /media/disk on the PC * things to do: * control VLAN on switch * config control VLAN and statically assign PC to controller IP * VLAN not used for any legacy networking purposes for controller * packet sniff SSL handshake ==== VLAN 888 ==== arbitrary VLAN for OFP controller, since nothing probably uses that high a VLAN number. Currently only port 0/42 (formerly one of the trunk ports) {{{ sw-sb09(config)# vlan 888 !sw-sb09(config-vlan)# name "OpenFlow control VLAN" !sw-sb09(config-vlan)# interface gi 0/42 !sw-sb09(config-if)# sh interface gigabitethernet 0/42 switchport mode trunk switchport trunk allowed vlan 1,3,27-28 switchport trunk native vlan 1 ! !sw-sb09(config-if)# no sw mo tru !sw-sb09(config-if)# no switchport trunk allowed vlan 1,3,27-28 !sw-sb09(config-if)# no switchport trunk nat vlan 1 !sw-sb09(config-if)# sh interface gigabitethernet 0/42 switchport mode access ! !sw-sb09(config-if)# sw acc vlan 888 !sw-sb09(config-if)# interface vlan 888 !sw-sb09(config-if)# ip address 172.16.4.1 255.255.255.0 !sw-sb09(config-if)# save sw-sb09(config-if)# }}} this will probably not need a route specified for it since the controller is directly attached to the switch. === formal controller VLAN: VLAN100. (6/14) === VLAN 888 was removed since VLAN 100 is the formal controller VLAN, as decided by discussion. {{{ sw-sb09(config-if)# sh interface gi 0/42 interface gigabitethernet 0/42 switchport mode access switchport access vlan 888 ! sw-sb09(config-if)# interface gigabitethernet 0/42 sw-sb09(config-if)# no sw acc vlan 888 !sw-sb09(config-if)# switchport mod trunk !sw-sb09(config-if)# swi trunk all vlan 1,3,27,28 !sw-sb09(config-if)# swi tru nat vlan 1 !sw-sb09(config-if)# save sw-sb09(config-if)# no vlan 888 !sw-sb09(config)# vlan 100 !sw-sb09(config-vlan)# name "OpenFlow control VLAN" !sw-sb09(config-vlan)# save }}} === Testing throughput with Iperf (7/23) === config eth0 of nodes 1-1 and 1-2, then run iperf to test the difference in bandwidth between !OpenFlow and regular firmware. iperf reference: http://openmaniak.com/iperf.php node1-1: 192.168.1.1/24 [[BR]] eth0: negotiated 1000baseT-HD flow-control [[BR]] eth1: negotiated 1000baseT-HD flow-control [[BR]] node1-2: 192.168.1.2/24 [[BR]] eth0: negotiated 1000baseT-HD flow-control [[BR]] eth1: negotiated 1000baseT-HD flow-control [[BR]] Server: Node1-2 Client: Node1-1 [[BR]] !OpenFlow mode || ||Transfer || Server ||Client || ||1|| 278 MBytes || 230 Mbits/sec ||230 Mbits/sec|| ||2|| 291 MBytes || 243 Mbits/sec ||244 Mbits/sec|| ||3|| 291 MBytes || 244 Mbits/sec ||244 Mbits/sec|| ||4|| 290 MBytes || 243 Mbits/sec ||244 Mbits/sec|| ||5|| 284 MBytes || 231 Mbits/sec ||231 Mbits/sec|| Normal mode ||1|| 278 MBytes || 229 Mbits/sec ||230 Mbits/sec|| ||2|| 291 MBytes || 228 Mbits/sec ||228 Mbits/sec|| ||3|| 291 MBytes || 228 Mbits/sec ||229 Mbits/sec|| ||4|| 290 MBytes || 231 Mbits/sec ||243 Mbits/sec|| ||5|| 284 MBytes || 229 Mbits/sec ||230 Mbits/sec|| Server: Node1-1 Client: Node1-2 [[BR]] !OpenFlow mode ||1|| 278 MBytes || 229 Mbits/sec ||230 Mbits/sec|| ||2|| 291 MBytes || 229 Mbits/sec ||229 Mbits/sec|| ||3|| 291 MBytes || 232 Mbits/sec ||244 Mbits/sec|| ||4|| 290 MBytes || 230 Mbits/sec ||230 Mbits/sec|| ||5|| 284 MBytes || 230 Mbits/sec ||230 Mbits/sec|| Normal mode ||1|| 278 MBytes || 243 Mbits/sec ||244 Mbits/sec|| ||2|| 291 MBytes || 243 Mbits/sec ||242 Mbits/sec|| ||3|| 291 MBytes || 243 Mbits/sec ||236 Mbits/sec|| ||4|| 290 MBytes || 243 Mbits/sec ||244 Mbits/sec|| ||5|| 284 MBytes || 242 Mbits/sec ||243 Mbits/sec|| Overall, there doesn't seem to be too much difference in performance. === more realtime throughput testing (7/30) === Using a tool called BWM-NG (Bandwidth Monitor NG): http://www.gropp.org/?id=projects&sub=bwm-ng the tool can output logs as a csv, which can be parsed with some more practice-Ruby code. Hopefully will get Scruffy or something working so the data can be graphed. BWM-NG was installed on SB9 with command `apt-get install bwm-ng`. `bwm-ng --help` gives you all the options you have. For csv output of tests for all eth1 interfaces at 1sec intervals: `bwm-ng -I eth1,eth1.27,eth1.28,eth1.100 -o csv -t 1000 -F ` Three tests were done: * nothing connected, no controller active * controller active * activate 2 nodes, ping one from the other with 65500 bit payload * controller active, virtual switch connected to LAN (WINLAB network) === socket programming in Ruby (8/1) === Some resources found: * IBM tutorial: https://www6.software.ibm.com/developerworks/education/l-rubysocks/l-rubysocks-a4.pdf * TCP client/server (in C): http://devmentor.org/articles/network/Socket%20Programming.pdf having issues with trying it, getting this error: {{{ A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied. - getpeername(2) (Errno::ENOTCONN) }}} supposedly meaning I'm not getting the connection to happen right idea for !OpenFlow: use code from reference model to simulate a handshake, display on a webpage or something else, to get a sense of how to integrate things === Anatomy of !OpenFlow Protocol (8/3) === !OpenFlow Switch has a [http://www.openflowswitch.org/documents/openflow-spec-v0.9.0.pdf paper] you can read for information on how the protocol works. The initial handshake is the important piece of info that makes a switch communicate with what it considers a controller. ==== The Handshake ==== The OFP handshake contains the following steps: 1. OFPT_HELLO message - sent by both parties to negotiate what version of !OpenFlow to use 1. SSL connection (optional) 1. OFPT_FEATURES_REQUEST - sent by controller 1. OFPT_FEATURES_REPLY - returned by switch 1. ECHO reply/request or some flow actions The steps need to be recreated, but there are several things that need to be done: * when creating a socket in Ruby you need a port number, but it is a passive connection (ptcp) - the port can be anything * pick apart either OF reference system or NOX for handshake code === picking at OFP (8/4) === !OpenFlow data structure is defined under /include/openflow/openflow.h . Whether it is NOX or the !Openflow Regression Suite it is all in lots of C. Trying to make Ruby work with C seemed a bit too messy for our own good, so we sent an email to the NOX developer's mailing list in hopes of some elucidation of how to create a Ruby interface to !OpenFlow, instead of Python (which is what NOX does, as described [http://noxrepo.org/wp/?page_id=2 here]) === Some giant lapse in time, with a pinch of SNMP (8/21) === So far: * have some Ruby code that interrogates and configures SNMP agents and retrieves OFP info. * mild understanding of net-snmp * tutorial to get started with SB9 * Seething suspicion that Bob's firmware is preventing reassignment of vlans to port (see below) To-do: * find a way to snmp-set vlan assignments to ports * maybe write OFP remote config code * test code on other devices Some links: * libsnmp-Ruby classes : http://snmplib.rubyforge.org/doc/index.html * MIBs : http://www.mibdepot.com/cgi-bin/getmib4.cgi?abc=0&n=Q-BRIDGE-MIB&r=baynetworks&f=rfc2674-q.mib&t=tree&v=v2&i=0&obj=VlanId * dot1q MIBs : http://en.community.dell.com/forums/p/19257012/19425860.aspx === The battle continues (8/22) === The NEC MIB manual revealed that while traditionally read-write, dot1q family of OIDs are made non-writable for the IP8800. * NEC's big refrence guide of MIBs (in Japanese) : http://www.nec.co.jp/ip88n/s36_sw/html/mibref/index.html * 2.13.5 - qBridgeMIB (VLANs) So instead, using Telnet to change values in a gorier way * Ruby Telnet : http://www.ruby-doc.org/stdlib/libdoc/net/telnet/rdoc/index.html The following code bit creates a ruby telnet object, which essentially logs in and reconfigures port vlan assignments (I am particularily impressed/irked by the way I can now recall commands without even sitting in front of the CLI) {{{ def set_port_vlan(port, vlan) #sets port VLAN. For NEC switches whose port-vlans can't be set using dot1qPvid SNMP value. sw = Net::Telnet::new("Host" => "172.16.100.10","Timeout"=> 10,"Prompt" => /[$%#>] \z/n) sw.login("operator") sw.cmd("enable") sw.cmd("configure") { |c| print c } sw.cmd("interface gigabitethernet"+" "+"#{port}") { |c| print c } puts "changing port vlan value" sw.cmd("switchport access vlan"+" "+"#{vlan}") { |c| print c } #sleep(1) sw.cmd("save") sw.cmd("exit") {|c| print c} sw.cmd("exit") {|c| print c} sw.cmd("exit") {|c| print c} end }}} === Different switch, different syntax === SNMP responses differ from switch to switch - what a NEC switch calls !GigabitEther 0/12 might be called !GigabitEthernet1/12 by a Cisco switch: {{{ > snmpwalk -v 2c -c public 172.28.13.9 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: Cisco IOS Software, Catalyst 4500 L3 Switch Software (cat4500-IPBASE-M), Version 12.2(25)SG1, RELEASE SOFTWARE (fc2) Technical Support: http://www.cisco.com/techsupport ... > snmpwalk -v 2c -c public 172.28.13.9 ifDescr.10 IF-MIB::ifDescr.10 = STRING: GigabitEthernet2/7 }}} To Do: * figure out any syntactical nuances of Netgear switches === Tutorial, more modifications necessary. (8/27) === The !OpenFlow tutorials are up and running, but there exists a few issues, all in regards to how the nodes can be used as controllers: 1. The virtual switch needs a specific controller IP it can listen to (no dynamic discovery protocol for OFP as of now) 1. The OF VLAN must be made available to the node being used as controller. 1. BUT - trunking is not practical with the nodes (config script?) So when am experimenter uses a node as a OF controller, we need the following things to happen: 1. OF VLAN becomes accessible to that node 1. The virtual switch is informed of the node's IP address, so it can listen for it (the node has to have an IP address before it can be controller) I guess either script or manual configs are ok, but manual configs will need to give the user access to the CLI of the switch - this can be risky to-do: * brainstorm on different methods * look for possible dynamic VLAN methods on switch manual. * shell scripting? === 9/2 - another switch, FPGA needed. === got another switch. name pending. On meanwhile, making it so I can flash it with new firmware. enabling telnet from VLAN 100 on SW-SB09. preliminary configs on a new NEC switch: * VLAN 1, 100 (Network, OFP) * disable STP, PVST+ still need to do: * group acls on switch * NetFPGA === 9/15 === * group acl on switches : ok * update perl-Net libraries on centOS === ??? (9/20) === for some reason, things were changed in the SB9 setup. notably: * network interface configs on SB9 had VLANs commented out * Bob lost his virtual switch instance I don't recall doing any of the two. So, it boils down to two possibilities: I am getting dumber, or someone else came in to change it. to-do: * group discussion * write proposal == 10/3 == To-Do: * Reinstall NetFPGA !OpenFlow software * Move NetFPGA to new chassis == NetFPGA lives! 10/17 == The box that was set up by Nithin finally works - it passed the OpenFlow switch regression test. The other box has a compromised NIC - the network does not work if the machine is started up while connected to the LAN, however it functions if it is connected after start-up. The on-board NIC is incompatible with drivers that come with CentOS 5.2; as a standby solution, another NIC has been slapped on for the net-install, after which I will try to update the drivers. == L2TP SUCCESS!!!! 10/29 == The setup process was actually not so pain full. We only needed 2 packages ''vtun'' and ''bridge-utils''. We copied the config files almost verbatim from (http://omf.mytestbed.net/wiki/omf/L2HowTo). The only exception being we changed the client address to 10.40.0.251. Log messages on both machines seem to indicate success. We'll have to "test" it with some switching sauce. To bring up the br0 interface we need to use ifup (not ifconfig dev up). Note this will fail if ''bridge-utils'' is not installed since the interface is virtual and made by brctl. Once br0 is up, restarting vtun caused many messages to appear in the syslog of landing1 (where our packets go to die). == CentOS shinanigans, Ubuntu NetFPGA compatibility. == CentOS proved to ba a difficult beast to deal with when it came to getting it to work with anything. Notably speaking: * CentOS 5.2 realtek NIC driver incompatibility (will give you a great time trying to do the net-install) * NetFPGA drivers do not compile without kernel update (what it uses expects the use of a different kernel version due to dependencies) * Even after the FPGA is working, kernel modules for !OpenFlow switches failing to build properly Supposedly, NetFPGAs are capable of working with machines running Ubuntu. hopefully this works. * Ubuntu-NetFPGA Compatibility info: http://www.netfpga.org/netfpgawiki/index.php/Ubuntu_Compatibility#NetFPGA_Beta_Changes_.28for_Ubuntu.29 ---So, the other box is now running Ubuntu 8.04 desktop edition. They seem to have changed a few things around, i.e. run is run.pl, ect. You also need an account to download the NetFPGA base package. [[BR]] [[BR]] [[BR]] [[BR]] [wiki:Internal/OpenFlow/ return to OpenFlow index]