= Notes = This section covers discussions and some background work that went into this project. == VLANs == VLANs are a way to split up broadcast domains at L2. They can be statically or dynamically set, with dynamic VLANs sort of working in a similar way to DHCP. == DHCP Relay agents == Relay agents are virtual layer 3 devices residing on a switch with VLANs. In this case, they allow DHCP broadcasts to be relayed across 2 VLANs on the same switch. This is not necessary for us because the DHCP server lives beyond the ASA, and the ASA not only acts as a router between the switch and the server, but also serves to manipulate the VLAN tags (masquerades as hosts sending out DHCP requests). == Trunking == Trunking is done by making a "backbone" for all the different VLANs on a local switch to converge to. Frames from different VLANs are identified bythe means of VLAN tags, a four-byte addition to the Ethernet frame identifying which VLAN it came from. The tagging is also known as 802.1q. == Native VLANs == Normally the 4 byte addition to the Ethernet frame would cause the tagged frame to be rejected on a VLAN, and regular frames would be rejected if found in the trunk due to the lack of a tag. This is unless the "regular" frame is from a '' native VLAN '', whose frames are intended not to have tags for devices that don't understand tagging. A site explaining what a native vlan is: http://networkers-online.com/blog/2008/06/native-vlan-explained/ In our case this matters because DHCP coming from the trunk needs to assign IP addresses to both the hosts connected to the switch on different VLANs (VLAN 27) and to the switch itsself. To do this, The virtual interface to which the IP address is assigned needs to be on the trunked port(s). === some thoughts regarding the virtual switches (6/9) === * They are overlay devices that "run" on top of VLANs. VLANs need to be set up before virtual switches can be created. * They are identified by the VLAN number of the VLAN they overlay, so one virtual switch can't encompass more than 1 VLAN. * Conversely, they can overlay part of a VLAN. In that case, the whole VLAN takes on the virtual switch's behavior. The last point is somewhat annoying, because the default behavior of a controllerless virtual switch is to "wait till the flow table times out." That would mean "become nonfunctional" in our case, since the virtual switches start off controller-less, and hence, flow-less. The whole switch would become a virtual brick if the trunk is specified in `setvsi`. This was demonstrated by overlaying a virtual switch over a single port on VLAN 27. It stopped forwarding DHCP messages all together when it used to be the only VLAN with DHCP. Questions: * can a virtual switch be loaded with a default flow table that would allow it to function, at least until its table times out? === when openflow.conf gets messed up (6/22) === One method to rewrite the !OpenFlow configuration is to delete openflow.conf directly through the switch, then use the `setvsi` command to set the new virtual switches up. This should be unnecessary, but it happened to be effective when old configurations did not seem to go away despite editing the .conf file using a PC. {{{ > cd /mnt/ > rm openflow.conf remove 'openflow.conf'? y > setvsi 1 1,3,5,7,9,11,13,15,2.1 tcp 172.16.4.224 dpid 0x0123456789ab > setvsi 2 17,19,21,23,25,27,29,31,2.2 tcp 172.16.4.180 dpid 0x012345678abc > setvsi 3 33,35,37,39,41,43,45,47,2.3 tcp 172.16.4.64 dpid 0x01234567abcd > showswitch vlan ports secure channel ---- ----- -------------- 1 1, 3, 5, 7, 9, 11, 13, 15, 2.1 disconnected 2 17, 19, 21, 23, 25, 27, 29, 31, 2.2 disconnected 3 33, 35, 37, 39, 41, 43, 45, 47, 2.3 disconnected }}} However, it seems like it is not possible to set `double-wide-mode`, or `no-save` through the CLI. These have to be added by editing openflow.conf using the PC. === SSL setup, things to know for the next day. (7/13) === using this site as reference: http://www.debian-administration.org/article/Creating_and_Using_a_self_signed__SSL_Certificates_in_debian except renaming the output .pem files to what it would understand: *ca_cert.pem - from cacert.pem *sw_cert.pem - from cert.pem *sw_key.pem - from key.pem for the Common Name, I just used the IP address of the CA, 192.168.203.75. because in the example on the site ca_cert.pem is named cacert.pem, I had to change "cacert" to "ca_cert" in openssl.cnf for the very last part (signing the certificate): {{{ [ CA_default ] serial = $dir/serial database = $dir/index.txt new_certs_dir = $dir/newcerts certificate = $dir/ca_cert.pem #change over here private_key = $dir/private/cakey.pem default_days = 365 default_md = md5 preserve = no email_in_dn = no nameopt = default_ca certopt = default_ca policy = policy_match }}} I am not sure if changing the name of the key from ofpswitch.key.pem to sw_key.pem after making/signing the certification will affect anything. Some lessons for the day: [[BR]] * the SD card will show up as /media/disk on the PC * things to do: * control VLAN on switch * config control VLAN and statically assign PC to controller IP * VLAN not used for any legacy networking purposes for controller * packet sniff SSL handshake ==== VLAN 888 ==== arbitrary VLAN for OFP controller, since nothing probably uses that high a VLAN number. Currently only port 0/42 (formerly one of the trunk ports) {{{ sw-sb09(config)# vlan 888 !sw-sb09(config-vlan)# name "OpenFlow control VLAN" !sw-sb09(config-vlan)# interface gi 0/42 !sw-sb09(config-if)# sh interface gigabitethernet 0/42 switchport mode trunk switchport trunk allowed vlan 1,3,27-28 switchport trunk native vlan 1 ! !sw-sb09(config-if)# no sw mo tru !sw-sb09(config-if)# no switchport trunk allowed vlan 1,3,27-28 !sw-sb09(config-if)# no switchport trunk nat vlan 1 !sw-sb09(config-if)# sh interface gigabitethernet 0/42 switchport mode access ! !sw-sb09(config-if)# sw acc vlan 888 !sw-sb09(config-if)# interface vlan 888 !sw-sb09(config-if)# ip address 172.16.4.1 255.255.255.0 !sw-sb09(config-if)# save sw-sb09(config-if)# }}} this will probably not need a route specified for it since the controller is directly attached to the switch. === formal controller VLAN: VLAN100. (6/14) === VLAN 888 was removed since VLAN 100 is the formal controller VLAN, as decided by discussion. {{{ sw-sb09(config-if)# sh interface gi 0/42 interface gigabitethernet 0/42 switchport mode access switchport access vlan 888 ! sw-sb09(config-if)# interface gigabitethernet 0/42 sw-sb09(config-if)# no sw acc vlan 888 !sw-sb09(config-if)# switchport mod trunk !sw-sb09(config-if)# swi trunk all vlan 1,3,27,28 !sw-sb09(config-if)# swi tru nat vlan 1 !sw-sb09(config-if)# save sw-sb09(config-if)# no vlan 888 !sw-sb09(config)# vlan 100 !sw-sb09(config-vlan)# name "OpenFlow control VLAN" !sw-sb09(config-vlan)# save }}} === Testing throughput with Iperf (7/23) === config eth0 of nodes 1-1 and 1-2, then run iperf to test the difference in bandwidth between !OpenFlow and regular firmware. iperf reference: http://openmaniak.com/iperf.php node1-1: 192.168.1.1/24 [[BR]] eth0: negotiated 1000baseT-HD flow-control [[BR]] eth1: negotiated 1000baseT-HD flow-control [[BR]] node1-2: 192.168.1.2/24 [[BR]] eth0: negotiated 1000baseT-HD flow-control [[BR]] eth1: negotiated 1000baseT-HD flow-control [[BR]] Server: Node1-2 Client: Node1-1 [[BR]] !OpenFlow mode || ||Transfer || Server ||Client || ||1|| 278 MBytes || 230 Mbits/sec ||230 Mbits/sec|| ||2|| 291 MBytes || 243 Mbits/sec ||244 Mbits/sec|| ||3|| 291 MBytes || 244 Mbits/sec ||244 Mbits/sec|| ||4|| 290 MBytes || 243 Mbits/sec ||244 Mbits/sec|| ||5|| 284 MBytes || 231 Mbits/sec ||231 Mbits/sec|| Normal mode ||1|| 278 MBytes || 229 Mbits/sec ||230 Mbits/sec|| ||2|| 291 MBytes || 228 Mbits/sec ||228 Mbits/sec|| ||3|| 291 MBytes || 228 Mbits/sec ||229 Mbits/sec|| ||4|| 290 MBytes || 231 Mbits/sec ||243 Mbits/sec|| ||5|| 284 MBytes || 229 Mbits/sec ||230 Mbits/sec|| Server: Node1-1 Client: Node1-2 [[BR]] !OpenFlow mode ||1|| 278 MBytes || 229 Mbits/sec ||230 Mbits/sec|| ||2|| 291 MBytes || 229 Mbits/sec ||229 Mbits/sec|| ||3|| 291 MBytes || 232 Mbits/sec ||244 Mbits/sec|| ||4|| 290 MBytes || 230 Mbits/sec ||230 Mbits/sec|| ||5|| 284 MBytes || 230 Mbits/sec ||230 Mbits/sec|| Normal mode ||1|| 278 MBytes || 243 Mbits/sec ||244 Mbits/sec|| ||2|| 291 MBytes || 243 Mbits/sec ||242 Mbits/sec|| ||3|| 291 MBytes || 243 Mbits/sec ||236 Mbits/sec|| ||4|| 290 MBytes || 243 Mbits/sec ||244 Mbits/sec|| ||5|| 284 MBytes || 242 Mbits/sec ||243 Mbits/sec|| Overall, there doesn't seem to be too much difference in performance. === more realtime throughput testing (7/30) === Using a tool called BWM-NG (Bandwidth Monitor NG): http://www.gropp.org/?id=projects&sub=bwm-ng the tool can output logs as a csv, which can be parsed with some more practice-Ruby code. Hopefully will get Scruffy or something working so the data can be graphed. BWM-NG was installed on SB9 with command `apt-get install bwm-ng`. `bwm-ng --help` gives you all the options you have. For csv output of tests for all eth1 interfaces at 1sec intervals: `bwm-ng -I eth1,eth1.27,eth1.28,eth1.100 -o csv -t 1000 -F ` Three tests were done: * nothing connected, no controller active * controller active * activate 2 nodes, ping one from the other with 65500 bit payload * controller active, virtual switch connected to LAN (WINLAB network) === socket programming in Ruby (8/1) === Some resources found: * IBM tutorial: https://www6.software.ibm.com/developerworks/education/l-rubysocks/l-rubysocks-a4.pdf * TCP client/server (in C): http://devmentor.org/articles/network/Socket%20Programming.pdf having issues with trying it, getting this error: {{{ A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied. - getpeername(2) (Errno::ENOTCONN) }}} supposedly meaning I'm not getting the connection to happen right idea for !OpenFlow: use code from reference model to simulate a handshake, display on a webpage or something else, to get a sense of how to integrate things === Anatomy of !OpenFlow Protocol (8/3) === !OpenFlow Switch has a [http://www.openflowswitch.org/documents/openflow-spec-v0.9.0.pdf paper] you can read for information on how the protocol works. The initial handshake is the important piece of info that makes a switch communicate with what it considers a controller. ==== The Handshake ==== The OFP handshake contains the following steps: 1. OFPT_HELLO message - sent by both parties to negotiate what version of !OpenFlow to use 1. SSL connection (optional) 1. OFPT_FEATURES_REQUEST - sent by controller 1. OFPT_FEATURES_REPLY - returned by switch 1. ECHO reply/request or some flow actions The steps need to be recreated, but there are several things that need to be done: * when creating a socket in Ruby you need a port number, but it is a passive connection (ptcp) - the port can be anything * pick apart either OF reference system or NOX for handshake code === picking at OFP (8/4) === !OpenFlow data structure is defined under /include/openflow/openflow.h . Whether it is NOX or the !Openflow Regression Suite it is all in lots of C. Trying to make Ruby work with C seemed a bit too messy for our own good, so we sent an email to the NOX developer's mailing list in hopes of some elucidation of how to create a Ruby interface to !OpenFlow, instead of Python (which is what NOX does, as described [http://noxrepo.org/wp/?page_id=2 here]) === Some giant lapse in time, with a pinch of SNMP (8/21) === So far: * have some Ruby code that interrogates and configures SNMP agents and retrieves OFP info. * mild understanding of net-snmp * tutorial to get started with SB9 * Seething suspicion that Bob's firmware is preventing reassignment of vlans to port (see below) To-do: * find a way to snmp-set vlan assignments to ports * maybe write OFP remote config code * test code on other devices Some links: * libsnmp-Ruby classes : http://snmplib.rubyforge.org/doc/index.html * MIBs : http://www.mibdepot.com/cgi-bin/getmib4.cgi?abc=0&n=Q-BRIDGE-MIB&r=baynetworks&f=rfc2674-q.mib&t=tree&v=v2&i=0&obj=VlanId * dot1q MIBs : http://en.community.dell.com/forums/p/19257012/19425860.aspx === The battle continues (8/22) === The NEC MIB manual revealed that while traditionally read-write, dot1q family of OIDs are made non-writable for the IP8800. * NEC's big refrence guide of MIBs (in Japanese) : http://www.nec.co.jp/ip88n/s36_sw/html/mibref/index.html * 2.13.5 - qBridgeMIB (VLANs) So instead, using Telnet to change values in a gorier way * Ruby Telnet : http://www.ruby-doc.org/stdlib/libdoc/net/telnet/rdoc/index.html The following code bit creates a ruby telnet object, which essentially logs in and reconfigures port vlan assignments (I am particularily impressed/irked by the way I can now recall commands without even sitting in front of the CLI) {{{ def set_port_vlan(port, vlan) #sets port VLAN. For NEC switches whose port-vlans can't be set using dot1qPvid SNMP value. sw = Net::Telnet::new("Host" => "172.16.100.10","Timeout"=> 10,"Prompt" => /[$%#>] \z/n) sw.login("operator") sw.cmd("enable") sw.cmd("configure") { |c| print c } sw.cmd("interface gigabitethernet"+" "+"#{port}") { |c| print c } puts "changing port vlan value" sw.cmd("switchport access vlan"+" "+"#{vlan}") { |c| print c } #sleep(1) sw.cmd("save") sw.cmd("exit") {|c| print c} sw.cmd("exit") {|c| print c} sw.cmd("exit") {|c| print c} end }}} === Different switch, different syntax === SNMP responses differ from switch to switch - what a NEC switch calls !GigabitEther 0/12 might be called !GigabitEthernet1/12 by a Cisco switch: {{{ > snmpwalk -v 2c -c public 172.28.13.9 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: Cisco IOS Software, Catalyst 4500 L3 Switch Software (cat4500-IPBASE-M), Version 12.2(25)SG1, RELEASE SOFTWARE (fc2) Technical Support: http://www.cisco.com/techsupport ... > snmpwalk -v 2c -c public 172.28.13.9 ifDescr.10 IF-MIB::ifDescr.10 = STRING: GigabitEthernet2/7 }}} To Do: * figure out any syntactical nuances of Netgear switches === Tutorial, more modifications necessary. (8/27) === The !OpenFlow tutorials are up and running, but there exists a few issues, all in regards to how the nodes can be used as controllers: 1. The virtual switch needs a specific controller IP it can listen to (no dynamic discovery protocol for OFP as of now) 1. The OF VLAN must be made available to the node being used as controller. 1. BUT - trunking is not practical with the nodes (config script?) So when am experimenter uses a node as a OF controller, we need the following things to happen: 1. OF VLAN becomes accessible to that node 1. The virtual switch is informed of the node's IP address, so it can listen for it (the node has to have an IP address before it can be controller) I guess either script or manual configs are ok, but manual configs will need to give the user access to the CLI of the switch - this can be risky to-do: * brainstorm on different methods * look for possible dynamic VLAN methods on switch manual. * shell scripting? === 9/2 - another switch, FPGA needed. === got another switch. name pending. On meanwhile, making it so I can flash it with new firmware. enabling telnet from VLAN 100 on SW-SB09. preliminary configs on a new NEC switch: * VLAN 1, 100 (Network, OFP) * disable STP, PVST+ still need to do: * group acls on switch * NetFPGA === 9/15 === * group acl on switches : ok * update perl-Net libraries on centOS === ??? (9/20) === for some reason, things were changed in the SB9 setup. notably: * network interface configs on SB9 had VLANs commented out * Bob lost his virtual switch instance I don't recall doing any of the two. So, it boils down to two possibilities: I am getting dumber, or someone else came in to change it. to-do: * group discussion * write proposal == 10/3 == To-Do: * Reinstall NetFPGA !OpenFlow software * Move NetFPGA to new chassis == NetFPGA lives! 10/17 == The box that was set up by Nithin finally works - it passed the OpenFlow switch regression test. The other box has a compromised NIC - the network does not work if the machine is started up while connected to the LAN, however it functions if it is connected after start-up. The on-board NIC is incompatible with drivers that come with CentOS 5.2; as a standby solution, another NIC has been slapped on for the net-install, after which I will try to update the drivers. == L2TP SUCCESS!!!! 10/29 == The setup process was actually not so pain full. We only needed 2 packages ''vtun'' and ''bridge-utils''. We copied the config files almost verbatim from (http://omf.mytestbed.net/wiki/omf/L2HowTo). The only exception being we changed the client address to 10.40.0.251. Log messages on both machines seem to indicate success. We'll have to "test" it with some switching sauce. To bring up the br0 interface we need to use ifup (not ifconfig dev up). Note this will fail if ''bridge-utils'' is not installed since the interface is virtual and made by brctl. Once br0 is up, restarting vtun caused many messages to appear in the syslog of landing1 (where our packets go to die). == CentOS shinanigans, Ubuntu NetFPGA compatibility. == CentOS proved to ba a difficult beast to deal with when it came to getting it to work with anything. Notably speaking: * CentOS 5.2 realtek NIC driver incompatibility (will give you a great time trying to do the net-install) * NetFPGA drivers do not compile without kernel update (what it uses expects the use of a different kernel version due to dependencies) * Even after the FPGA is working, kernel modules for !OpenFlow switches failing to build properly Supposedly, NetFPGAs are capable of working with machines running Ubuntu. hopefully this works. * Ubuntu-NetFPGA Compatibility info: http://www.netfpga.org/netfpgawiki/index.php/Ubuntu_Compatibility#NetFPGA_Beta_Changes_.28for_Ubuntu.29 ---So, the other box is now running Ubuntu 8.04 desktop edition. They seem to have changed a few things around, i.e. run is run.pl, ect. You also need an account to download the NetFPGA base package. == Running multiple !OpenFlow devices using one controller. 10/30 == Two IP8800's have been connected to eachother through a trunk connection allowing VLAN 100 and 28. Two instances of the controller were launched on SB9, along with virtual switch instances on the IP8800s. Here's the configs: === SB9 === controller instances started with assive connections: * vtcp:6633 * vtcp:6634 === SW-SB09 === on interface VLAN 100 (IP addr. 172.16.100.10/24): * Virtual switch on vlan 28 on ports 25-36 * OFP interface on 0/37, listening to port 6633 === CS switch === on interface VLAN 100 (IP addr. 172.16.100.30/24): * virtual switch on vlan 888 on ports 0/1-0/4 * vlan 28 on ports 25-36 * OFP interface on port 0/48, listening on port 6634 * CentOS NetFPGA host connected to port 0/1 VLAN 888 was used to test if hosts can connect through the virtual switch on the new OF switch. VLAN 888 was downlinked to the winlab network to handle production data - except, configuration changes to te CS switch were not saved, allowing ORBIT network traffic and backbone to be visible to the WINLAB side through the switch with a trunk conection to ORBIT. While not crippling, this did allow for DHCP messages from ORBIT to travel terrifyingly convoluted paths to still cause complications on the WINLAB network. moral of the story: double check configuration changes. == 12/31-Top switch configs. == VLAN 50 is the only real exposed VLAN - 4 ports (DMZ, 2 planet lab, firewall out) 1001-1005 - Cisco associated (ignore) 9,10- not used ports can be divided up into 3 groups 1. Infrastructure - Not really visible to user - Aruba, Internal, Instrumental ect - VLANs 1-6 2. Experimental - Pertains to Grid, outdoor, Sandboxes - VLANs 7,8,11-38 3. Public - Visible from outside - VLAN 50 We want to combine the internal, firewall feed, and top switch port configs onto one NEC IP8800. The ports that comprise this setup are: 1. 4 VLAN 50 ports (ports 1-4) 1. 4 firewall ports (only 3 actually needed to go into ASA) 1. 10 Internal (VLAN 2) Ports 1. 16 Top switch ports (Assorted, originally 20 including 4 VLAN 50 ports) refrences (these are in Japanese): * port config parameter specs: http://www.nec.co.jp/ip88n/s36_sw/html/cfref/cfref-chap1-4.html * interface vlan context specs: http://www.nec.co.jp/ip88n/s36_sw/html/cfref/cfref-chap11-03.html * vlan context specs: http://www.nec.co.jp/ip88n/s36_sw/html/cfref/cfref-chap11-22.html === Creating VLANs === Step 1 of switch configs always start off with creating vlans. you can create a single VLAN using the command [[BR]] `interface vlan ` However, VLAN creation cannot be done in groups using the command [[BR]] `interface range vlan ` Like in group configs of ports. This will return an error {{{ (config)# interface range vlan 7-38 interface : Not found VLAN-ID <7>. }}} Hence, when you need to make ranges of VLANs you must specify the range of VLANs to create using the command [[BR]] `vlan ` making the commands to create the vlans necessary for this switch into these lines on the CLI: {{{ (config)# vlan 1-8 !(config-vlan)# vlan 11-48 !(config-vlan)# vlan 50 !(config-vlan)# save (config-vlan)# }}} ---- Then, you may ask, why have the "interface vlan" context?. To be able to understand this, you need to know that the "interface vlan" context lets you treat a VLAN as an abstraction layer that allows access to both Layer 2 and 3 information for a specific group of hosts. This means using the "interface vlan" context allows you to actually specify IP layer characteristics for groups of hosts based on which vlans they belong in. This is, as you may guess, quite useful. The `interface range vlan` context can be used to configure groups of already existing vlans. So given that you have a group of vlans, you can configure them all at once like this: {{{ (config)# #this switch already has vlans 1-6 on it (config)# interface range vlan 1-5 (config-if-range)# sh interface vlan 1 ! interface vlan 2 ! interface vlan 3 ! interface vlan 4 ! interface vlan 5 ! }}} In this case, you can see that you enter the proper (config-if-range)# mode. doing a `show` in this mode lets you see which vlans you are configuring together. ---- === Naming VLANs === Once all VLANs are created, you'd want to name them. this is done through the `vlan` context. {{{ (config-vlan)# vlan 2 (config-vlan)# name "Internal" }}} rinse and repeat this process. A table of the vlans coming up soon. === Associating VLANs with ports === Now that you have properly named VLANs, you can associate them with ports. <> first, the 4 publically visible VLAN 50 ports: {{{ (config)# int range gi 0/1-4 (config-if-range)# switchport access vlan 50 }}} Then, the firewall ports feeding the VLANs into the ASA; These are specialized trunk ports: * geth0/7 - VLAN 50 * geth0/8 - VLANs 1-6 * geth0/9 - VLANs 7,8 * geth0/10 - VLANs 11-38 The commands to make this happen are the following. First make four ports into trunk ports using the 'interface range' context: {{{ (config)# int ran gi 0/7-10 (config-if-range)# sw mo tru !(config-if-range)# }}} Then, specify allowed VLANs for each port. This is according to the list above. {{{ !(config-if-range)# ##geth0/7 - VLAN 50## !(config-if-range)# interface gigabitethernet 0/7 !(config-if)# swi trunk allowed vlan 50 !(config-if)# ##geth0/8 - VLANs 1-6## !(config-if)# interface gi 0/8 !(config-if)# swi trunk allow vlan 1-6 !(config-if)# ##geth0/9 - VLANs 7,8## !(config-if)# interface gi 0/9 !(config-if)# swi tru all vl 7,8 !(config-if)# ##geth0/10 - VLANs 11-37## !(config-if)# interface gi 0/10 !(config-if)# swi tru all vl 11-38 !(config-if)# save (config-if)# exit }}} the configuration results: {{{ (config)# int ran gi 0/7-10 (config-if-range)# sh interface gigabitethernet 0/7 switchport mode trunk switchport trunk allowed vlan 50 ! interface gigabitethernet 0/8 switchport mode trunk switchport trunk allowed vlan 1-6 ! interface gigabitethernet 0/9 switchport mode trunk switchport trunk allowed vlan 7-8 ! interface gigabitethernet 0/10 switchport mode trunk switchport trunk allowed vlan 11-38 ! }}} Next, the 10 Internal ports; These are regular switchports all associated to VLAN 2. {{{ (config)# int ran gi 0/13-22 (config-if-range)# sw mo dot1q-tunnel !(config-if-range)# sw acc vlan 2 }}} Finally, the sw_top configurations; This is perhaps the most complicated part. the ports will be configured from 0/48 to leave ample space between the Internal ports and these ports, which are all specialized trunk ports (we won't want people to accidentally be able to plug into them!). organizing them: Infrastructure VLANs ||port ||VLANs ||Description || ||0/29 ||1,2 ||Internal (1) || ||0/30 ||1,2 ||Internal (2) || ||0/31 ||1,3 ||CM || ||0/32 ||1,4,5 ||Aruba, Instrumental || ||0/33 ||1,6 ||DMZ || ||0/34 ||1-6 ||Main Subnets || Testbed VLANs ||0/37 ||1,3,29,30 ||CM, Outdoor Data and Ctrl || ||0/38 ||1,7,31:2:37 ||Grid and VGrid Ctrl || ||0/39 ||1,8 ||Grid Data || ||0/40 ||7,8 ||Grid Data and Ctrl || ||0/41 ||1,6-8 ||DMZ, Grid Data and Ctrl || ||0/42 ||11-48 ||?? || ||0/43 ||1,11-38 ||Testbed Data and Ctrl || ||0/44 ||1,3,11-38 ||CM, Testbed Data and Ctrl || Master Trunk ||0/47 ||1 ||Access Net || ||0/48 ||1-38 ||Master Trunk || ---- The error messages are quite...verbose. {{{ !(config-if)# sw tru all vlan 10-48 interface : Can't set switchport trunk allowed vlan which is not configured to use vlan vlan 10. }}} This simply means you are trying to add a nonexistent vlan to the list of allowed VLANs for a trunk. Make sure the VLAN exists before trying to add it! ---- === killing Spanning Tree === Spanning Tree Protocol should be disabled. Both PVST and PVST+ count as spanning tree. {{{ !(config)# no spanning-tree vlan 3,7,8,11-38 !(config)# save }}} === Hacked in notes for Installing the Open flow capabilities on the NetFpga === ''' FIXME: Please put me in the correct place ''' '''Prequisite packages''' * autoconf * libtool * pkg-config * linux-source (2.6.28) Following the instructions of [http://www.openflowswitch.org/foswiki/bin/view/OpenFlow/Deployment/HOWTO/LabSetup Labsetup] we cloned the 0.8.9 branch of the their git repository, I'll include the steps because their pages don't stay up very long. {{{ $ git clone git://openflowswitch.org/openflow.git $ cd openflow $ git checkout -b openflow_089_rev4 origin/release/0.8.9 }}} Assuming all the prerequisites are installed, the next step is to compile the source. We're trying to make two kernel objects, and a bit file for the netfpga. {{{ $ ./boot.sh $ ./configure --with-l26=/lib/modules/`uname -r`/ --enable-hw-tables=nf2 $ make }}} boot.sh will fail to run if the listed prerequisites are not installed (there may be more preqs based on what was installed prior, build-tools, kernel headers, etc...). The original configure line read "./configure --with-l26=--with-l26=/lib/modules/`uname -r`/build --enable-hw-tables=nf2", while the extra --with-l26 is an obvious error, there are others. The configure script stuffs the argument of --wtih-l26 into a $path variable, then looks for the subdirectories $path/build and $path/source. Specifying --with-l26 as ../`uname -r`/build causes configure to fail, since there is no source directory in the build directory. Instead what they wanted was the directory 1 level up. In our first attempt we just ran make and went on to the next step, however when we tried to insert the modules one of them failed. The problem was with the ofdatapath_netfpga.ko module. When we tried to insert it (with insmod) it would complain about a missing symbol. Dmesg would show an even every time you tried to insert the module that said {{{ WARNING: "nf2k_reg_read" WARNING: "nf2k_reg$ ./boot.sh }}} Consulting a reference from the Stanford [$ https://mailman.stanford.edu/pipermail/openflow-discuss/2009-January/000051.html mailing list] on netfpga, we determined that the Module.symvers file needed to be edited. We needed to copy the entries from the ~/NF2/lib/c/kernel/Modules.symvers into the ~/openflow/datapath/linux-2.6/Modules.symvers. The NF2 directory comes from two tar files: * netfpga_base_beta_2_0_0_0.tar.gz * netfpga_openflow_switch.0_8_9-1.tar.gz - This file was chosen to match the version of the Netfpga openflow source we're compling. Making this change produces the bits files and modules that will be needed in the next section. After a sucessfull complie we needed to load the openflow bitfile onto the netfpga {{{ # cpci_reprogram.pl # nf2_download ~/openflow/datapath/hwtable_nf2/openflow_switch.bit }}} The actual paths may vary but the basic process is the same. This is fairly straight forward assuming the netfpga tools were already built. Once that was done we inserted the modules (required root privileges) {{{ # /sbin/insmod ./openflow/datapath/linux-2.6/ofdatapath.ko # /sbin/insmod ./openflow/datapath/linux-2.6/ofdatapath_netfpga.ko }}} At this point we're ready to try testing the netfpga as a open flow switch. I'm going to pause here and rearrange the networks that the specfic netfpgas belong to. === 2/14/10 === James encountered a problem where the kernel nf2 module was throwing errors about buffers: {{{ [ 943.700185] nf2: no available transmit/receive buffers [ 943.700230] nf2: no available transmit/receive buffers [ 943.700275] nf2: no available transmit/receive buffers [ 943.700321] nf2: no available transmit/receive buffers [ 943.700366] nf2: no available transmit/receive buffers [ 943.700411] nf2: no available transmit/receive buffers }}} According to [http://www.openflowswitch.org/bugs/openflow/ticket/4 Openflow guys], its due to the defective card. James swapped out a new card and is attempting to see if it makes a diffrence. Currently the new card is functioning correctly (2/14/10 5:03 pm). The previous card died (started throwing buffer errors) at around 2:00pm on 2/14/10 after about 12 hours of continous operation. This problem is not entirely new, as it was seen before with the original card during the regressions tests on Ubuntu. It may be that case that this is an os driver problem as this problem dosen't seem to show up in centos. === 4/2/10 - Attempts to make a NOX app === Nox does come with a very incomplete [http://noxrepo.org/manual/app.html handbook]. Because it is incomplete, the Nox folks recommend getting "very comfortable with the source." Here are some attempts to create an app for a hot standby demo. The following were carried out in the user's home directory with a copy of the pre-installed Nox on SB9. To copy the Nox directory, one should create a tar archive, copy that to their directory, and extract with modified ownership (details in the man pages for tar) since it was owned by root. 1. made directory for new app in ~/nox-3-20-10/src/nox/coreapps/ called demosw 1. copied meta.xml from switch app: cd demosw; cp ../switch/meta.xml ./ 1. edit meta.xml to match name of app: {{{ demosw <<--here demosw <<--here too }}} '''the app'''[[BR]] Now comes the part of understanding the app. The nox app is based on the component class, described in component.hh (found in ./src/nox/, assuming you're in your nox directory), so you must always include this file. the basic structure of an app is as followed: {{{ #include "component.hh" class Hub : public Component { public: Hub(const Context* c, const xercesc::DOMNode*) : Component(c) { } void configure(const Configuration*) { } void install() { } }; REGISTER_COMPONENT(container::Simple_component_factory, Hub); }}} this is just a stripped down version of the hub code. Most nox apps can be found in ./src/nox/coreapps/. To debug, also include vlog.hh to enable log, warning and other messages. vlog.hh and other important functionalities, called "events" are in ./src/include. Events must be registered through the Disposition handler, which returns either STOP or CONTINUE, as defined in event.hh. [[BR]] [[BR]] [[BR]] [[BR]] [wiki:Internal/OpenFlow/ return to OpenFlow index]