Building Network Topologies

This page aims to be a compilation of various notes regarding things to consider when, as well as techniques for, setting up topologies for network experiments. We consider the case where nodes share a broadcast domain at L2, and can hear all traffic.

The first section describes methods for simulating point-to-point links between two nodes sharing a single switch. The second section describes possible ways to describe a desired topology to an automated system so that it can build the topology for a user.

Contents

1. Simulating Point-to-Point Links
   
2. Topology Description Methods

Prerequisites

We assume that you have a setup similar to ​SB9, of several nodes connected to shared switch(es). For NetFPGA-based topology setups, you need nodes with NetFPGA drivers and bitfiles. For software-based switching, we use OpenvSwitch or Linux bridging. For the OpenFlow methods, you also need an OpenFlow controller or some other tool like dpctl that allows you to push flows to your software defined switch. You should have access to the switch that the nodes are sharing as well, since you need to configure VLANs and the likes. The following links describe setup and use of theses components (internal links):

• ​OpenVswitch - A software-defined virtual switch with OpenFlow support, no special hardware required.
• ​NetFPGA - FPGA-based network device with OpenFlow support
• ​Quanta LB9A - The shared medium switch. In this page this switch will be used in XorPlus (normal) mode.
• ​collection - A short list of common OpenFlow controllers.

We also assume that you have basic knowledge about L2 constructs, and how they apply to switched networks.

The nodes that were used for the topologies in this page are ​NetFPGA cubes running Ubuntu10.10 (kernel: 2.6.35-30-generic). The shared switch is a ​Quanta LB9A running ​Pica8's !XorPlus.

I. Simulating point-to-point Links

This section introduces topology setup using the simplest case of building a single link between two nodes.

Contents

1.2 Some Considerations
1.3 Non-OpenFlow Methods

1.3.1 Kernel IP routing (Layer 3)
1.3.2 Linux Bridge (Layer 2)
1.3.3 Packet filters (pending) (Layers 2-4)

1.4 OpenFlow Methods (Layers 1-4)

1.4.1 Manually adding flows and an intro to `dpctl`.
1.4.2 OpenvSwitch
1.4.3 NetFPGA OpenFlow switch
1.4.4 Prototyping - With Mininet (pending)

1.5 Summary

1.1 Overview

In general, a topology describes the restrictions on traffic flow between one or more nodes (with the simplest case being a single node by itself). We build a topology by first isolating the nodes from each other, and then introducing relays that can move traffic between these nodes in a controlled manner. The way the traffic is relayed produces the topology. Our base topology, in which all nodes can reach each-other through a switch, is logically a fully connected graph:

(A)   A - B         
       \ /          
        C           

A usual method to isolate the nodes is to configure the switch to place each node on a separate VLAN. Picking node C as a traffic relay (A router-on-a-stick in the case of VLANs) produces the following topology:

(B)
     A-C-B

We call A and B end nodes and C a network node straddling A and B's VLANs. From this logic it follows that the VLAN is the link (despite it actually being a logical setup on the shared switch, rather than a wire). Given that the partitions have identifiers, such as IP block addresses, nodes on the same link must share the identifier, and relays must know the identifiers of all partitions that it connects; for the case above, C has knowledge of, and interfaces, both A and B's VLANs, so we have the following links:

 A-C  (VLAN A)
 C-B  (VLAN B)

The next section explores several methods to produce similar effects.

1.2 Some Considerations

The techniques used to partition the broadcast domain will heavily depend on two things:

1. the type of experiment
2. the available interfaces on the nodes

In terms of 1., we aim to keep topology setup and the experiment as two individual problems as opposed to one monolithic one. For example, we may not want to use IP routing if we don't plan on using TCP/IP, or are planning to modify layer 3 ¹. 2. is important in that, depending on how it's done, the number of links you can have on the node will be restricted to the number of interfaces you have. When you only have one interface, you may want to use virtual interfaces to increase the number of links to/from your node. In turn, you may also need to modify the partitioning scheme of the shared switch.

A standard way to deploy virtual interfaces is with VLANs and trunking. VLANs may be combined with other configuration schemes, are relatively simple to configure, and a good portion of networked devices understand them. Many of the examples here that require virtual interfaces will make use of this standard technique. So, to make things easier, we will quickly describe how to add virtual interfaces and VLAN awareness to a node before moving on to Section 1.1 .

1. Install and load VLAN module:

    apt-get install vlan
    modprobe 8021q
   

   The module can be made to load at boot time by appending '8021q' to the list /etc/modules.
2. Add VLAN interfaces using vconfig:

    vconfig add eth0 111
    vconfig add eth0 222
   

   This creates two virtual LAN interfaces, eth0.111 and eth0.222 on eth0. An interface configured in this manner is said to be VLAN aware.

Note, virtual interfaces are workarounds to being restricted to one physical interface. Any setup with nodes with multiple interfaces (e.g. using NetFPGAs) will not require the above configs, lest you want more interfaces than you have. For nodes with multiple physical interfaces, the steps describing 'eth0.xxx' can be replaced by the names of each unique interface. Keep in mind, however, that if the interface is connected to a switchport configured as a trunk, it must also be made VLAN aware even if it does not hold multiple virtual interfaces.

1.3 non-OpenFlow Methods

The methods in this section should work on any *nix machine, so they can serve as "sanity checks" for the system you are using as the network node.

1.3.1 Kernel IP routing

Kernel IP routing has the least requirements, in that no extra packages are required if you have multiple Ethernet ports on your node. As its name indicates, it works strictly at layer 3. Partitioning occurs across IP blocks; you would need one block per link.

Network node setup

1. This setup assumes a 1-to-1 mapping of VLANs to subnets. Choose IP blocks, one for each VLAN/interface. For example, if you have two clients connected across your node, you need two IP blocks, one for each VLAN:
   • VLAN 111: 192.168.1.0/24, gateway 192.168.1.13
   • VLAN 222: 192.168.2.0/24, gateway 192.168.2.23

The gateway IPs chosen above will be the IP addresses assigned to the VLAN interfaces you have set up earlier on your network node.

2. Bring up VLAN interfaces with the IP addresses/blocks you have chosen:

    ifconfig eth0 0.0.0.0 up
    ifconfig eth0.111 inet 192.168.1.23 broadcast 192.168.1.255 netmask 0xffffff00 up
    ifconfig eth0.222 inet 192.168.2.23 broadcast 192.168.2.255 netmask 0xffffff00 up
   

   This configuration can be made permanent by modifying /etc/network/interfaces:

   auto eth0.111
   iface eth0.111 inet static
       address 192.168.1.13
       netmask 255.255.255.0
       vlan-raw-device eth0
   
   auto eth0.222
   iface eth0.222 inet static
       address 192.168.2.23
       netmask 255.255.255.0
       vlan-raw-device eth0
   

3. Enable routing on network node

    route add -net 192.168.1.0 netmask 255.255.255.0 gw 192.168.1.13
    route add -net 192.168.2.0 netmask 255.255.255.0 gw 192.168.2.23
    echo 1 >  /proc/sys/net/ipv4/ip_forward
   

   The last line in the above block is equivalent to running the command:

   sysctl -w net.ipv4.ip_forward=1
   

   The ip_forward flag resets itself after reboot. To make it permanent, add

   sysctl net.ipv4.ip_forward=1
   

   to /etc/sysctl.conf.

End node setup

Unless you have set up DHCP, you must manually assign an IP address and default gateway to each node. The former should be consistent with the subnet associated with the VLAN to which the end host belongs. For example, the following host is connected to a switch port associated with VLAN 222, so it is assigned an address from the 192.168.2.0/24 block:

 ifconfig eth0 inet 192.168.2.4

Then you must add reachability information to the node's routing table e.g. the IP addresses that it must send data to in order to have it reach remote subnets. Since there is only one other subnet in this example, a single entry specifying the destination subnet (192.168.1.0/24 - VLAN 111) and the gateway IP in/out of the current node's subnet is added:

 route add -net 192.168.1.0 netmask 255.255.255.0 gw 192.168.2.23

Do this for each remote subnet that the node should be able to communicate with. Once all of the nodes are configured, you should be able to ping end-to-end.

1.3.2 Linux Bridge

In terms of implementation, this is probably the simplest method.

A bridge will ignore VLAN tags, so if you have two VLAN interfaces e.g. eth0.111 and 222 sitting on a trunk, the packets will come in tagged. An intermediate abstraction will strip the tag from the packet (at br0), and the packet will get tagged as appropriate on the outbound. Unlike kernel IP forwrding, bridging works purely at Layer 2, hence you do not need to worry about IP addressing.

The first three steps refer to the network node.

1. Configure and bring VLANS up as before, sans IP addresses

2. Install bridge-utils:

    apt-get install bridge-utils
   

3. Create bridge interface, add ports:

    brctl addbr br0
    brctl addif br0 eth0.111
    brctl addif br0 eth0.222
   

4. Make sure all interfaces (br0, eth0.*) are up, as they may not come up automatically.
5. Set all hosts on the bridged VLANs to the same IP block.

The Linux Foundation keeps a page that may be useful for various troubleshooting: ​http://www.linuxfoundation.org/collaborate/workgroups/networking/bridge

1.4 OpenFlow Methods

This section assumes that you have all of the OpenFlow components (e.g. OVS, NetFPGA drivers) set up and working, and that you have several choices of controller. The controller used primarily in this section is the Big Switch Networks (BSN) controller, which has a decent CLI and RESTful API for pushing flows.

1.4.1 Manually adding flows, and an intro to dpctl.

OpenFlow flow entries can be manually pushed to a switch; This section focuses on building links by specifying pairs of flow entries for inputs and outputs. Some controllers come with APIs and CLI commands that let you manually build and push flow entries, but that requires you to install a full controller for this purpose. A bare-bones tool, called dpctl, can be used to do this. There are several merits to using dpctl:

• few requirements to build
• "just works" with any switch listening for a controller using ptcp
• quick OF switch setup sans controller.

You can grab dpctl as a part of the OpenFlow reference package from the Stanford git repo:

apt-get install git-core autoconf, libtool, pkg-config 
git clone git://gitosis.stanford.edu/openflow.git

and doing ./boot.sh, ./configure, make, and make install. man dpctl gives a pretty good explanation of syntax.

A few things to keep in mind about statically pushed flows are:

1. A list of flow entries behaves like an ACL. Incoming packets are matched by both entry priority value and header content. Rules with higher priority values are used before lower ones. When multiple rules have the same explicit priority number, the first match applies.

2. A rule based on higher layer matches must have the lower layer rules defined. For example, to match on IP, you must also specify the Ethertype (0x800 = IP). Not doing so causes the switch to skip checking of the IP source, destination, protocol, and ToS fields.

3. Not all switches support FLOOD. FLOOD and ALL both cause the switch to output to all ports, but differently: The former instructs the switch to use its regular (non-OpenFlow) network stack, while the latter is purely OpenFlow based and must be supported by all OF switches. As you can guess, purely OpenFlow switches do not support FLOOD.

1.4.2 OpenvSwitch

OpenvSwitch (OVS) is a user-space software defined switch with OpenFlow support, complete with its own implementation of a controller. It can, and is assumed to be, built as a kernel module throughout this page. Details on setup are found ​here.

the following only needs to be done once, in the initial configurations.

1. Add ports:

    ovs-vsctl add-br br0
    ovs-vsctl add-port br0 eth0.111
    ovs-vsctl add-port br0 eth0.222
   

By the time the 'add-port' commands are used, you should not be able to ping across the two VLANS, even with correct route table entries and packet forwarding enabled in the kernel. Here, br0 is a virtual interface similar to tap0 in the bridge. There should be one virtual interface per virtual switch to be instantiated. By default, ports added to the switch are trunked. Using the option tag=VLAN ID makes the interfaces behave as access ports for the VLAN ID specified:

 ovs-vsctl add-port br0 eth0.111 tag=111
 ovs-vsctl add-port br0 eth0.222 tag=222

However, this is unrelated to what needs to happen here so we will not explore its uses any further (for now). On the meanwhile, if not up already, bring up br0 and any ports you have added to it with ifconfig.

2. If it has not been done already, initialize the OpenFlow controller. The procedures for this step differ according to the controller in use, and are discussed in the pages for each respective controller.
   

A sanity check for this step is to test your virtual switch with the OVS built-in controller, ovs-controller, which may be initialized on the same node running OVS:

ovs-controller -v ptcp:6633

When ovs-controller is used, the controller IP is, unsurprisingly, 127.0.0.1.

3. Point ovs-vswitchd to the OpenFlow controller.

   ovs-vsctl set-controller br0 tcp:172.16.0.14:6633
   

In this example, the OVS process is pointed to a BSN controller (kvm-big) on 172.16.0.14, listening on port 6633². Alternatively, tcp above can be substituted with ptcp for a passive connection that simply listens for an incoming connection. This is useful if you need to push some commands to the switch with dpctl.

With a properly initialized and configured database, when launched ovs-vswitchd will spit out a bunch of messages as it attempts to connect to the controller. Its output should look something similar to this:

root@node1-4:/opt/openvswitch-1.2.2# vswitchd/ovs-vswitchd unix:/usr/local/var/run/openvswitch/db.sock --pidfile --detach
Nov 07 17:37:02|00001|reconnect|INFO|unix:/usr/local/var/run/openvswitch/db.sock: connecting...
Nov 07 17:37:02|00002|reconnect|INFO|unix:/usr/local/var/run/openvswitch/db.sock: connected
Nov 07 17:37:02|00003|bridge|INFO|created port br0 on bridge br0
Nov 07 17:37:02|00004|bridge|INFO|created port eth0.101 on bridge br0
Nov 07 17:37:02|00005|bridge|INFO|created port eth0.102 on bridge br0
Nov 07 17:37:02|00006|ofproto|INFO|using datapath ID 0000002320b91d13
Nov 07 17:37:02|00007|ofproto|INFO|datapath ID changed to 000002972599b1ca
Nov 07 17:37:02|00008|rconn|INFO|br0<->tcp:172.16.0.14:6633: connecting...

The OpenvSwitch OpenFlow switch should be functional as soon as it finds and connects to the controller. As you can see above, a DPID is chosen at random; if a random DPID does not suit your needs, a DPID may be specified manually using ovs-vsctl:

ovs-vsctl set bridge <mybr> other-config:datapath-id=<datapathid>

Where <datapathid> is a 16-digit hex value. For our network node, this becomes:

ovs-vsctl set bridge br0 other-config:datapath-id=0000009900113300

1.4.3 NetFPGA OpenFlow switch

This method is probably the most involved and difficult to get right, although in theory would be the best since you would get the programmatic flexibility of the OVS switch and the speed of a hardware-implemented device.

Assuming that you already have NetFPGA drivers installed, no special configurations are needed for the NetFPGA, save the OpenFlow-switch bitfile. The typical flow rules also do not apply to the NetFPGA e.g. default flow modules on NOX that work with OVS will break under the NetFPGA. Therefore extra flows must be added to compensate.

The easiest manner to control the NetFPGA is via the BSN controller. The following two flow entries needed to be added to the controller in order for the client-nodes to be able to ping each other across the NetFPGA.

  flow-entry vlan111-ip
    active True
    src-mac 00:15:17:d6:da:4a
    vlan-id 111
    actions set-vlan-id=222,output=all
  flow-entry vlan222-ip
    active True
    src-mac 00:15:17:d6:ce:20
    vlan-id 222
    actions set-vlan-id=111,output=all	  

This set of flows basically implements VLAN stitching based on source MAC address. Unlike in the Linux bridge, one cannot see the VLAN-stripped packets on the virtual interface (tap0 on the NFPGA, br0 on bridge); they will already have the proper tag, since the processing is probably occurring in the FPGA and not in the kernel.

1.5 Morals of the story

For quick setup of a network topology using nodes sharing a medium, point-to-point links should be defined at as low a layer as possible. The next best thing (that is even better because of its flexibility) to actually going in and connecting up the topology using cables is to carve up the shared switch into VLANs. This lets you restrict the broadcast domain however you want, without hard-wiring everything, even when you are restricted to just one physical interface per node.

As for OpenFlow switching, OVS nodes (or a Mininet prototype topology) controlled by a controller supporting flow pushing (e.g. BSN or Floodllight) is the most flexible, least-hassle choice for this task.

---

^{1. this, of course, depends on what you are trying to demonstrate. 2. This specific example requires a bit of network reconfiguration and comes with substantial risk of disconnecting your node from the network if done carelessly.}