Changes between Initial Version and Version 1 of Internal/Infrastructure/SetupTestbed/BOOTB

Timestamp:

Nov 7, 2008, 5:21:33 PM (16 years ago)

Author:

korakis

Comment:

—

Internal/Infrastructure/SetupTestbed/BOOTB

@@ +1 @@
+
+[[TOC(heading=Table of Contents, depth=3)]]
+= Building our own Testbed =
+
+In this section we give a detailed description of our efforts on setting up a mid-size ORBIT-like testbed. In the right of this page you can see a list of all the related actions we had to take from the initial stage until the stage of having a full functional testbed that can be remotely accessed for uploading particular experiments, running the experiments and collecting the results.
+
+
+
+== Hardware Setup ==
+
+=== Orbit Node ===
+Each ORBIT Radio Node is a PC with a 1 GHz VIA C3 processor, 512 MB of RAM, 40 GB of local disk, two Ethernet ports, two 802.11 a/b/g cards and a Chassis Manager to control the node, see Figure 3.  The description about the ethernet ports is following.
+
+'''Control port''' - The ethernet port between 4 USB ports, it is a Rtl-8169 Gigabit ethernet port, which is used to load and control the ORBIT node and collect measurements. 
+
+'''Data port''' - The ethernet port above two USB ports, it is a VT6102 Rhine-II 100/10baseT Ethernet port, which is used for data communication, 
+
+'''CM port''' - The 10BaseT Ethernet port on Chassis Manager Card, which is used to communicate with gridservice (not gridservice2) [[BR]]
+[[BR]]
+[[Image(node-50.png)]] [[Image(orbit_node.JPG)]]
+
+                           
+=== Testbed ===
+The test bed consists of nodes and several servers. Technically, all servers can be put in one machine with at lease two ethernet ports, but it's not recommendatory because of potential security consideration. A typical test bed include three servers discribed below[[BR]]
+[[Image(testbed.PNG)]]
+                                
+
+'''Services''' - It is used to host various services including DHCP, DNS, NTP, TFTP, PXE, Frisbee, NFS, mysql, OML and Apache. We have different aliases for the management host to segregate the services that it hosts. This machine or port shall be connected with '''Control port''' of nodes. [[BR]]
+
+'''Console''' - It is used to run experiments with nodehandler4. Console is also connected with '''Control port''' of nodes. It may share one Ethernet port with '''Services'''. A better way is setuping a console in one machine exclusively and let it accessible by experimenters with ssh or XDMCP.[[BR]]
+
+'''CMC''' - It is the control and monitoring manager for all CM elements of ORBIT nodes. It is connected with '''CM port''' of nodes and can NOT share Ethernet port with '''Services''' and '''Console'''.
+
+In our situation, Service and Console share one ethernet port with address 10.10.0.10/16 and CMC is on another ethernet port with address 10.1.200.1/16.
+
+You may connect 10.10.0.10 and 10.1.200.1 with an internal route. In my situation. Since both machines connect to outside with their eth1, I explicitly set route on both machine as following.[[BR]]
+On Console/Services
+{{{
+console:~# route add -host 10.1.200.1 gw 128.238.34.248 dev eth1
+console:~# route
+Kernel IP routing table
+Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
+cmc.grid.poly.e cmc.local       255.255.255.255 UGH   0      0        0 eth1
+localnet        *               255.255.255.0   U     0      0        0 eth1
+10.10.0.0       *               255.255.0.0     U     0      0        0 eth0
+default         128.238.34.1    0.0.0.0         UG    0      0        0 eth1
+}}} 
+
+On CMC
+{{{
+cmc:~# route add -host 10.10.0.10 gw 128.238.34.247 dev eth1
+cmc:~# route
+Kernel IP routing table
+Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
+console.grid.po console.local   255.255.255.255 UGH   0      0        0 eth1
+128.238.34.0    *               255.255.255.0   U     0      0        0 eth1
+10.1.0.0        *               255.255.0.0     U     0      0        0 eth0
+default         128.238.34.1    0.0.0.0         UG    0      0        0 eth1
+cmc:~# ping console
+PING console.grid.poly.edu (10.10.0.10) 56(84) bytes of data.
+64 bytes from console.grid.poly.edu (10.10.0.10): icmp_seq=1 ttl=64 time=0.128 ms
+64 bytes from services.grid.poly.edu (10.10.0.10): icmp_seq=2 ttl=64 time=0.139 ms
+}}}
+
+== Software and Services ==
+=== Linux Installation ===
+You need install Linux on two or more machines. If you want to use only one machine, it must equipped with two or more ethernet interfaces. You can choose whatever Linux distribution you prefer. However, Debian is strongly recommended because some testbed related software are distributed in both  source code and Deb package.
+The installation guide can be found [http://www.debian.org/releases/stable/installmanual here]. For more information about Debian, please visit http://www.debian.org
+   
+=== Configure Apt ===
+If the Linux distribution you choosed is Debian/Ubuntu, please add the following lines in file '''/etc/apt/sources.list''', so let apt-get can find Debian packages provided by [http://www.orbit-lab.org Orbit lab].
+
+{{{
+deb http://apt.orbit-lab.org/orbit testing main
+deb http://apt.orbit-lab.org/orbit unstable main
+deb http://apt.orbit-lab.org/orbit stable main
+}}}
+
+After each time sources.list is changed, please run command "apt-get update" to resynchronize the package index files from their sources.
+For more about sources.list and apt-get, please refer with command "man sources.list" and "man apt-get".
+
+=== Configure host name ===
+If Console/Service and CMC are on different machines, please edit file '''/etc/hostname''' respectively. The host name of Console/Service is "'''console'''", and that of CMC is "'''cmc'''".
+If the whole three server are on one machine, please set it's host name to "'''console'''". 
+
+=== Configure network interface ===
+Since we place Console/Service and CMC on different machine. Interface configuration file should like following[[BR]]
+For Console/Service
+{{{
+# This file describes the network interfaces available on your system
+# and how to activate them. For more information, see interfaces(5).
+
+# The loopback network interface
+auto lo eth0 eth1
+iface lo inet loopback
+
+# The primary network interface access to outside internet
+allow-hotplug eth1
+iface eth1 inet static
+address 128.238.34.247
+netmask 255.255.255.0
+network 128.238.34.0
+broadcast 128.238.34.255
+gateway 128.238.34.1
+#iface eth1 inet dhcp
+## dns-* options are implemented by the resolvconf package, if installed
+dns-nameservers 10.10.0.10 128.238.1.68
+dns-search grid.poly.edu
+dns-domain grid.poly.edu
+
+# The internal network interface for console and services(nodehandler4,
+# dhcp, dns, gridservice2, OML, etc)
+allow-hotplug eth0
+iface eth0 inet static
+address 10.10.0.10
+netmask 255.255.0.0
+network 10.10.0.0
+broadcast 10.10.255.255
+## dns-* options are implemented by the resolvconf package, if installed
+dns-nameservers 10.10.0.10 128.238.1.68
+dns-search grid.poly.edu
+dns-domain grid.poly.edu
+}}} 
+
+For CMC
+{{{
+# This file describes the network interfaces available on your system
+# and how to activate them. For more information, see interfaces(5).
+
+# The loopback network interface
+auto lo eth0 eth1
+iface lo inet loopback
+
+# The primary network interface access to outside internet
+allow-hotplug eth1
+iface eth1 inet static
+address 128.238.34.248
+netmask 255.255.255.0
+network 128.238.34.0
+broadcast 128.238.34.255
+gateway 128.238.34.1
+# dns-* options are implemented by the resolvconf package, if installed
+dns-nameservers 10.10.0.10 128.238.1.68
+dns-search grid.poly.edu
+dns-domain grid.poly.edu
+#iface eth1 inet dhcp
+
+# The internal network interface for CMC(gridservice).
+allow-hotplug eth0
+iface eth0 inet static
+address 10.1.200.1
+netmask 255.255.0.0
+network 10.1.0.0
+broadcast 10.1.255.255
+## dns-* options are implemented by the resolvconf package, if installed
+dns-nameservers 10.10.0.10 128.238.1.68
+dns-search grid.poly.edu
+dns-domain grid.poly.edu
+}}}
+
+In each server, interface eth0 is for test bed, and eth1 is for access outside. If all servers is on one machine, the configuration of eth0 on CMC should be moved to /etc/network/interfaces on Console/Services. 
+
+=== Name resolve ===
+Because eth1 might connect with other DHCP/DNS servers from outside, and it used to change file /etc/resolv.conf when servers boot up, we need resolvconf to fix the setting in in resolv.conf.
+
+First, install "resolvconf" with command "''apt-get install resolvconf''". Then, run command "dpkg-reconfigure resolvconf" and agree to symlink /etc/resolv.conf to /etc/resolvconf/run/resolv.conf. At last, edit file /etc/resolvconf/interface-order like following
+
+{{{
+# interface-order(5)
+eth*
+}}}
+
+Some comment for file /etc/network/interfaces:
+
+"'''dns-nameservers 10.10.0.10 128.238.1.68'''" indicates resolvconf to add nameservers 10.10.0.10 and 128.238.1.68 in resolv.conf. 10.10.0.10 is the address of DNS used for testbed, and 128.238.1.68 is the address of DNS for outside, which you can change to others in your situation. Please make sure 10.10.0.10 is the first address appear in the list.
+
+"'''dns-search grid.poly.edu'''" and "'''dns-domain grid.poly.edu'''" indicate resolvconf to add "search" and "domain" entries in resolv.conf. "'''grid'''" is considered as the name of test bed, which will be used in later configuration.
+
+The final /etc/resolv.conf might looks like below after rebooting the machine.
+{{{
+# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
+#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
+nameserver 10.10.0.10
+nameserver 128.238.1.68
+search grid.poly.edu
+}}}
+
+For more information about resolvconf, please refer /usr/share/doc/resolvconf/README.gz and "man resolvconf".
+
+=== DHCP ===
+'''Purpose''': This software runs a DHCP server that assigns IP addresses to clients on demand.
+
+'''Installation and Configuration: ''':
+ * Run command on '''Console/Services''', and select eth0 as interface.
+{{{
+apt-get install dhcp3-server
+}}}
+ * Make sure that /etc/default/dhcp3-server has eth0 as default interface which DHCP will server for. Just like following
+{{{
+# On what interfaces should the DHCP server (dhcpd) serve DHCP requests?
+#       Separate multiple interfaces with spaces, e.g. "eth0 eth1".
+INTERFACES="eth0"
+}}}
+
+ * Edit /etc/dhcp3/dhcpd.conf. The configuration looks like below.
+{{{
+# The ddns-updates-style parameter controls whether or not the server will
+# attempt to do a DNS update when a lease is confirmed. We default to the
+# behavior of the version 2 packages ('none', since DHCP v2 didn't
+# have support for DDNS.)
+ddns-update-style interim;
+use-host-decl-names on;
+
+allow booting;
+allow bootp;
+
+# option definitions common to all supported networks...
+option domain-name "grid.poly.edu";
+
+default-lease-time 259200;
+max-lease-time 259200;
+
+# If this DHCP server is the official DHCP server for the local
+# network, the authoritative directive should be uncommented.
+#authoritative;
+
+# Use this to send dhcp log messages to a different log file (you also
+# have to hack syslog.conf to complete the redirection).
+log-facility local7;
+
+subnet 10.10.0.0 netmask 255.255.0.0 {
+   range 10.10.1.1 10.10.255.254;
+   option domain-name "grid.poly.edu";
+   ddns-updates off;
+   ddns-domainname "grid.poly.edu";
+   ddns-rev-domainname "in-addr.arpa";
+   option domain-name-servers 10.10.0.10;
+   next-server 10.10.0.10;
+   
+   host services {hardware ethernet 00:1B:2F:BE:EF:94; fixed-address services.grid.poly.edu;}
+   host console {hardware ethernet 00:1B:2F:BE:DF:6E; fixed-address console.grid.poly.edu;}
+
+  filename "/tftpboot/pxelinux.bin";
+
+#node 10.10.x.y
+
+    group {
+       host node1-1 {hardware ethernet 00:0F:EA:8C:AE:39; fixed-address node1-1.grid.poly.edu;}
+       host node1-2 {hardware ethernet 00:03:2D:08:19:fe; fixed-address node1-2.grid.poly.edu;}
+       host node1-3 {hardware ethernet 00:03:2D:07:67:CE; fixed-address node1-3.grid.poly.edu;}
+   }
+}
+}}}
+ '''Some comments on the dhcpd.conf'''[[BR]]
+
+ First line is '''use-host-decl-names''' on which means DNS has actual IP mappings, DHCP just gives out hostnames. This means only IP to name updates need to be done only at DNS.[[br]]
+ '''next-server'''  is used to specify the host address of the server from which the initial boot file is to be loaded. In our case, it's the address tftp server.[[BR]]
+ The  '''filename'''  statement is used to specify the name of the initial boot file which is to be loaded by a client. [[BR]]
+ '''domain-name-servers''' specifies a list of Domain Name System name servers available to the client.[[BR]]
+ The node name in form of node'''x'''-'''y''' determines it's address must be 10.10.'''x'''.'''y'''. For example, the address of node2-3 is 10.10.2.3. The actual address mapping is done by DNS.
+
+'''To Run: ''': /etc/init.d/dhcp3-server start  -- errors go to /var/log/daemon.log
+
+ === DNS ===
+'''Purpose''': Services.poly.edu” hosts the primary DNS service for the zone grid.poly.edu. The DNS server is the standard BIND9 software packaged.
+
+
+'''Installation and Configuration: ''':
+ * ''apt-get install bind9''
+ * You need edit Named.conf, named.conf.options and named.conf.local under /etc/bind as below
+   * named.conf
+{{{
+include "/etc/bind/named.conf.options";
+include "/etc/bind/named.conf.local";
+
+Controls {
+      inet 127.0.0.1 port 953
+      allow { 127.0.0.1; };
+};
+
+// prime the server with knowledge of the root servers
+zone "." {
+        type hint;
+        file "/etc/bind/db.root";
+};
+
+// be authoritative for the localhost forward and reverse zones, and for
+// broadcast zones as per RFC 1912
+
+zone "localhost" {
+        type master;
+        file "/etc/bind/db.local";
+};
+
+zone "127.in-addr.arpa" {
+        type master;
+        file "/etc/bind/db.127";
+};
+
+zone "0.in-addr.arpa" {
+        type master;
+        file "/etc/bind/db.0";
+};
+
+zone "255.in-addr.arpa" {
+        type master;
+        file "/etc/bind/db.255";
+};
+}}}
+   * named.conf.options
+{{{
+options {
+        directory "/etc/bind";
+        auth-nxdomain no;    # conform to RFC1035
+        listen-on-v6 { any; };
+};
+}}}
+   * named.conf.local
+{{{
+// Consider adding the 1918 zones here, if they are not used in your
+// organization
+//include "/etc/bind/zones.rfc1918";
+zone "grid.poly.edu" IN {
+       type master;
+       file "/etc/bind/orbit.zone";
+};
+zone "in-addr.arpa" IN {
+      type master;
+      file "/etc/bind/zone.orbit";
+};
+}}}
+ * You need create orbit.zone and zone.orbit under /etc/bind as below
+   * orbit.zone: Forward lookup
+{{{
+$TTL 3600
+@ IN  SOA services.grid.poly.edu.  root.services.grid.poly.edu. (
+                          2008072501 ; serial
+                          3600       ; refresh (1 hour)
+                          600        ; retry (10 min)
+                          10000      ; expire (2 hours)
+                          3600    );
+                                    ;
+@        IN    NS        services.grid.poly.edu.
+$ORIGIN grid.poly.edu.
+$TTL 129600
+windows      IN   A       10.10.1.8
+rxwarp       IN   A       10.10.1.9
+node1-1      IN   A       10.10.1.1
+node1-2      IN   A       10.10.1.2
+node1-3      IN   A       10.10.1.3
+
+console      IN   A      10.10.0.10
+cmc          IN   A      10.1.200.1
+
+services     IN   CNAME  console.grid.poly.edu.
+dhcp         IN   CNAME  console.grid.poly.edu.
+frisbee      IN   CNAME  console.grid.poly.edu.
+pxe          IN   CNAME  console.grid.poly.edu.
+oml          IN   CNAME  console.grid.poly.edu.
+repository   IN   CNAME  console.grid.poly.edu.
+repository1  IN   CNAME  console.grid.poly.edu.
+repository2  IN   CNAME  console.grid.poly.edu.
+ntp          IN   CNAME  console.grid.poly.edu.
+loghost      IN   CNAME  console.grid.poly.edu.
+idb1         IN   CNAME  console.grid.poly.edu.
+idb2         IN   CNAME  console.grid.poly.edu.
+}}}
+   * zone.orbit: Reverse lookup
+{{{
+$TTL 3600
+@           IN SOA services.grid.poly.edu. root.services.grid.poly.edu. (
+                                           2008021817    ; serial
+                                           28800    ; refresh (8hours)
+                                           900      ; retry (15 mins)
+                                           604800   ; expire (1 week)
+                                           86400    ; minimum (1 day)
+                                    );
+@              IN    NS              services.grid.poly.edu.
+$ORIGIN in-addr.arpa.
+$ORIGIN 10.in-addr.arpa.
+$ORIGIN 0.10.10.in-addr.arpa.
+$TTL 129600
+10     IN  PTR    services.grid.poly.edu.
+10     IN  PTR    console.grid.poly.edu.
+
+$ORIGIN 1.10.10.in-addr.arpa.
+9     IN  PTR   rxwarp.grid.poly.edu.
+8     IN  PTR   windows.grid.poly.edu.
+1     IN  PTR   node1-1.grid.poly.edu.
+2     IN  PTR   node1-2.grid.poly.edu.
+3     IN  PTR   node1-3.grid.poly.edu.
+
+$ORIGIN 200.1.10.in-addr.arpa.
+1     IN  PTR   cmc.grid.poly.edu.
+}}}
+ * Make sure there are dots at the end of the domains. The owner and group of orbit.zone and zone.orbit might be like following.
+{{{
+console:/etc/bind# ls -l orbit.zone zone.orbit
+-rw-r--r-- 1 root bind 1217 2008-08-07 17:00 orbit.zone
+-rw-r--r-- 1 root bind  928 2008-08-07 17:01 zone.orbit
+}}}
+
+ '''To Run: ''':
+/etc/init.d/bind9 start  -- errors go to /var/log/daemon.log
+
+ * You may run command "'''host'''" on Console/Services or CMC  as below to verify if bind works well.
+{{{
+console:~# host cmc
+cmc.grid.poly.edu has address 10.1.200.1
+console:~# host console
+console.grid.poly.edu has address 10.10.0.10
+console:~# host services
+services.grid.poly.edu is an alias for console.grid.poly.edu.
+console:~# host node1-1
+node1-1.grid.poly.edu has address 10.10.1.1
+console:~# host pxe
+pxe.grid.poly.edu is an alias for console.grid.poly.edu.
+console.grid.poly.edu has address 10.10.0.10
+console:~# host frisbee
+frisbee.grid.poly.edu is an alias for console.grid.poly.edu.
+console.grid.poly.edu has address 10.10.0.10
+}}}
+
+=== Apache Web Server ===
+'''Purpose:'''
+Apache server is required for maintaining the ORBIT local repository for debian packages and also to view the results of the experiment
+
+'''Installation:'''
+ * ''apt-get install apache2'' [[BR]]
+Note that no additional configuration is needed for apache. Also, make sure that /var/www/cgi-bin points to /usr/lib/cgi-bin (or create a soft link if one does not exist using cd /var/www/; ln –s /usr/lib/cgi-bin cgi-bin). [[BR]]
+
+'''To run:'''/etc/init.d/apache2 start -- errors go to /var/log/daemon.log
+
+We also need install libgd, which is used to view the results of the experiment. The command below can install it
+{{{
+apt-get install libgd-gd2-perl
+}}}
+
+=== NTP ===
+
+'''Purpose:'''
+ All the machines synchronize their time using the time server as the reference.
+
+'''Installation and Configuration:'''
+ * ''apt-get install ntp''
+ * You may add ntp server "pool.ntp.org" into /etc/ntpd.conf if there is server setting in it. 
+
+'''To run:'''
+ * /etc/init.d/ntpd start -- errors go to /var/log/daemon.log
+
+=== TFTP Server ===
+'''Purpose:'''
+TFTP is needed to install PXE images whenever you need to install an image onto the node (using Frisbee). It is also used to load a memory based image that can be used to fetch the current image of the node into the repository
+
+'''Installation and Configuration:'''
+ * ''apt-get install atftpd''
+
+There are two options here: either to run atftpd as a standalone daemon or run it under inetd. For heavy duty tftp services, you can choose to run is as a standalone daemon.
+For our installation, we choose the standalone daemon.
+
+ * Edit file /etc/inetd.conf and point the tftp directory to /tftpboot. The configuration may look like following
+{{{
+#:BOOT: TFTP service is provided primarily for booting.  Most sites
+#       run this only on machines acting as "boot servers."
+tftp            dgram   udp     wait    root /usr/sbin/atftpd   /usr/sbin/in.tftpd /tftpboot
+}}}
+ * The PXE image can be download from [http://witestlab.poly.edu/attachment/wiki/BOOTB/tftpboot.tar.bz2?format=raw here]. You need to extract it with command[[BR]]tar -xjvf tftpboot.tar.bz2[[BR]]
+The final content of directory /tftpboot looks like as below.
+{{{
+console:~# ls /tftpboot -R
+/tftpboot:
+initramfs-orbit-pxe-2.0.3.gz  linux-orbit-pxe-2.6.25.1  pxelinux.bin  pxelinux.cfg
+
+/tftpboot/pxelinux.cfg:
+default  orbit-2.0.3-omf
+}}}
+
+In case of problems, make sure that lo interface is up.
+
+=== NFS Service ===
+
+'''Purpose:'''
+This service is used to remotely mount directories on the nodes while fetching their image using imagezip utility. Also, Frisbee service makes use of this directory to install images onto nodes.
+
+'''Installation and Configuration:'''
+
+ * ''apt-get install nfs-kernel-server''
+
+ * create a path like "/export/orbit/image/tmp"
+
+ * Add a line in /etc/exports file as follows. "/export/orbit/image/tmp" is the default path nodes use to save frisbee images.
+
+{{{
+/export/orbit/image/tmp     10.10.0.0/16(rw,sync,no_root_squash)
+}}}
+
+'''To run:'''
+ * /etc/init.d/nfs-kernel-server start -- errors go to /var/log/daemon.log
+
+=== Mysql Server ===
+
+'''Purpose:'''
+This service is used to store the results of the experiments conducted on ORBIT
+
+''' Installation and Configuration:'''
+ * ''apt-get install mysql-server-4.1''
+ * Edit /etc/mysql/my.cnf and replace bind-address from 127.0.0.1 to 10.10.0.10
+type mysql  and at the prompt, enter the following
+
+ * Replace the password by an appropriate one. Basically, this creates a new account called orbit with the password specified and allows access to mysql databases from localhost and any other machine on the network.
+
+'''To run:'''
+ * /etc/init.d/mysql start -- errors go to /var/log/mysql.log
+
+=== ORBIT Server ===
+Orbit specific services include nodehandler, nodeagent, frisbee, gridservices, gridservices2 and OML(Orbit Measurement Library). If you have added configuration in sources.list as described [http://witestlab.poly.edu/wiki/BOOTB#ConfigureApt here], you can follow the commands below to install them. All these services are installed on Console, except gridservices on CMC.
+
+For Console
+ * ''apt-get update''[[BR]]
+
+ * ''apt-get install otg''[[BR]]
+
+ * ''apt-get install nodehandler4''[[BR]]
+   Since some files in nodehandler4 debian package are obsolete, you need to update it with a [http://witestlab.poly.edu/attachment/wiki/BOOTB/nodehandler.tar.gz?format=raw tar ball]. Extract it and replace fold '''/opt/nodehandler4-4.2.0/'''.[[BR]]
+   The configuration of nodehandler4 is based on [http://www.yaml.org YAML] script file /etc/nodehandler4/nodehandler.yaml. You can create a symbol link named "'''nodehandler.yaml'''" to the actual configuration file. The example and comments below could be helpful to understand it.
+{{{
+  1 # NOTE: use only 'spaces' to indent !
+  2 # ('tab' indents are not supported by the ruby yaml parser used to read this file)
+  3 #
+  4 # This is the Config file for the NodeHandler4 on the WINLAB platform
+  5 #
+  6 ---
+  7 nodehandler:
+  8   name_resolv: |
+  9     name = nil
+ 10     if NodeHandler.JUST_PRINT
+ 11       name = 'debug'
+ 12     else
+ 13       # take first subdomain as grid name (sb0.orbit-lab.org)
+ 14       IO.popen('hostname -d') {|f| name = f.gets.split('.')[0] }
+ 15     end
+ 16
+ 17   testbed:
+ 18
+ 19     # Config Parameter for the "default" Testbed
+ 20     #
+ 21     # In the WINLAB setting the default testbed is the "grid" testbed, using gridservice2
+ 22     default:
+ 23
+ 24       repository:
+ 25         path: [".", "../repository", "/opt/nodehandler4-4.2.0/repository"]
+ 26
+ 27       pxe:
+ 28         # This is the URL where NH can find the PXE GridService
+ 29         url: 'http://pxe:5022/pxe'
+ 30
+ 31       cmc:
+ 32         # This is the URL where NH can find the CMC GridService
+ 33         url: "http://cmc:5012/cmc"  # Contact the CMC of GS 1 - Not ported yet for GS 2
+ 34
+ 35       oml:
+ 36         # NodeAgents will use the numerical IP address in 'local_host' to connect
+ 37         # to the machine running the NodeHandler, in order to retrieve the OML defs
+ 38         # (in XML, and generated by NH). These OML defs are used by the NAs' applications
+ 39         # Thus, 'local_host' = Control IP address (reachable by NAs) of the NH's machine
+ 40         local_host: '10.10.0.10'
+ 41         # The parameters below are the contact details for the OML GridService
+ 42         url: "http://oml:5022/oml"
+ 43         port: 5022
+ 44         host: "oml"
+ 45
+ 46       frisbee:
+ 47         # The parameters below are the contact details for the Frisbee GridService
+ 48         default_disk: '/dev/sda'
+ 49         url: 'http://frisbee:5022/frisbee'
+ 50
+ 51       inventory:
+ 52         # This is the URL where NH can find the inventory GridService
+ 53         url: 'http://cmc:5022/inventory'
+ 54
+ 55       # Command used to launch the communication module
+ 56       # The type of comm module to launch depends on the cmd line params
+ 57       # '-c PORT' runs a TCP comm. module that will connect to the node Agent's TCP server on PORT
+ 58       # '-l PORT' runs a TCP comm. Server module that will listen for node Agent's connection on PORT
+ 59       # default: runs a Multicast comm. module
+ 60       #
+ 61       # The following line runs the commServer in TCP Client Mode
+ 62       #commServer: /opt/nodehandler4-4.2.0/sbin/commServer --logfile /tmp/commServer-%ID%.log -d 4 --iface eth1 -c 9026
+ 63       # The following line runs the commServer in Multicast Mode
+ 64       commServer: /opt/nodehandler4-4.2.0/sbin/commServer --logfile /tmp/commServer-%ID%.log -d 4 --iface eth0
+ 65
+ 66       #
+ 67       # Return the IP address of the control interface of
+ 68       # the node a coordinates x:y
+ 69       #
+ 70       # @param x X coordinate of node
+ 71       # @param y Y coordinate of node
+ 72       #
+ 73       controlIp: |
+ 74         |x, y|
+ 75           # This is the Node Agents control IP address used in the WINLAB grid testbed
+ 76           "10.10.#{x}.#{y}"
+ 77
+ 78       #
+ 79       # Return the x:y coordinates of a node signing on with
+ 80       # 'idString'. This string is supposed to be of the type
+ 81       # '/ip/CONTROL_IP'.
+ 82       #
+ 83       # @param idString String of type '/ip/CONTROL_IP'
+ 84       # @return Array of [x, y]
+ 85       #
+ 86       nodeId2coord: |
+ 87         |idString|
+ 88           match = /.*\.(\d+)\.(\d+)$/.match(idString)
+ 89           if (match != nil && match.size == 3)
+ 90             x = match[1].to_i
+ 91             y = match[2].to_i
+ 92             if x > 100
+ 93               # sandbox
+ 94               x = y / 100
+ 95               y = y % 100
+ 96             end
+ 97             return [x, y]
+ 98           end
+ 99           raise "Can't parse #{idString}"
+100
+101       # Return the control IP address (as string) or DNS name for a node
+102       # at a given coordinate.
+103       #
+104       # Throws an ConfigException if no IP address can be found.
+105       # In this testbed, nodes are identified using only 1-dimensional coordinate: X
+106       # (At WinLab, nodes are identified using 2D coordinates)
+107       #
+108       coord2ip: |
+109         |x, y|
+110           # This is the base name used in the WINLAB testbeds
+111           name = "node#{x}-#{y}"
+112           begin
+113             Socket.gethostbyname(name)[3].unpack('C4').join('.')
+114           rescue SocketError
+115             raise("Unknown host '#{name}'")
+116           end
+117
+118       load: |
+119         | uri, evalRuby |
+120           path = [ uri.split(':').join('_') + '.rb']
+121           postfix = '/' + uri.split(':').join('/') + '.rb'
+122           REPOSITORY_DEFAULT().each { |dir|
+123             path << dir + postfix
+124           }
+125           #puts "PATH: #{path.join(':')}"
+126           file = path.inject(nil) { |found, p|
+127             if found == nil && File.readable?(p)
+128               found = p
+129             end
+130             found
+131           }
+132           if file == nil
+133             raise IOError, "Can't find any of '#{path.join(', ')}'"
+134           end
+135           str = File.new(file).read()
+136           if evalRuby
+137             #eval(str, nil, path, 1)
+138             require file
+139           end
+140           [str, 'text/ruby']
+141
+142     # Config Parameter for the "grid" Testbed
+143     #
+144     # To use this testbed, call nodeHandler with the option "-d grid"
+145     # Any parameter settings within this section will override the settings
+146     # done in the "default" section. The nodeHandler fisrt load the "default"
+147     # settings, then it uses the "domain" specific settings to override the
+148     # relevant parameters
+149     #
+150     # For more details: see comments in "default" domain section
+151     grid:
+152       X_MAX: 2
+153       Y_MAX: 20
+154       oml:
+155         local_host: '10.10.0.10'
+156         url: "http://oml:5022/oml"
+157         port: 5022
+158         host: "oml"
+159       controlIp: |
+160         |x, y|
+161           "10.10.#{x}.#{y}"
+162
+163     # Config Parameter for the "debug" Testbed
+164     #
+165     # To use this testbed, call nodeHandler with the option PRINT_ONLY or "-d debug"
+166     # Any parameter settings within this section will override the settings
+167     # done in the "default" section. The nodeHandler fisrt load the "default"
+168     # settings, then it uses the "domain" specific settings to override the
+169     # relevant parameters
+170     debug:
+171       repository:
+172         path: ['../repository']
+173       commServer: ../c/commServer/commServer -d 4 --iface eth0
+174       coord2ip: |
+175         |x, y|
+176           "10.99.#{x}.#{y}"
+}}}
+
+ Comments:[[BR]]
+ Line 29: PXE is one of services from gridservices2, and 5022 is its default port. Please make sure http://pxe could be resolved from '''Console'''.[[BR]]
+ Line 33: CMC is the only service provided by gridservices, and 5012 is its default port. In my situation. The address is http://cmc is 10.1.200.1.[[BR]]
+ Line 48: '/dev/sda' is the paramter frisbee need to image hard drivers. If the hard drivers in nodes are equipped with ATA interface, Please change it to "/dev/hda"[[BR]]
+ Line 49: Similar with PXE, it's also a service by gridsercies2.[[BR]]
+ Line 64: This is command used to launch the communication module of Nodehandler4. The value for parameter "--iface" should be the interface with IP address 10.10.0.10.[[BR]]
+ Line 151: "grid" is the name of testbed which must match with the first word of domain name of '''Console'''. For example, the output of command "hostname -d" is "grid.poly.edu", so "grid" is the name of testbed.[[BR]]
+ Line 152,153: The maximum value of two dimensions of testbed. If the '''Control port''' of each node has IP address 10.10.'''x'''.'''y''', the node's hostname should be like node'''x'''-'''y'''.grid.poly.edu, which is controlled by DNS. '''x''' and '''y''' are intergers less than X_MAX and Y_MAX respectively.[[BR]]
+
+ * frisbee package is an dependency of gridservices2, which can be downloaded from [http://witestlab.poly.edu/attachment/wiki/BOOTB/frisbee_1.0.3-1_i386.deb?format=raw here] and install it with command[[BR]]
+   ''dpkg -i  frisbee_1.0.3-1_i386.deb''[[BR]]
+   You can also get the latest source code of frisbee from [http://www.emulab.net/downloads/emulab-080630.tar.gz here] if you prefer to build it in your own system. You need get at least 2 executable files, '''frisbeed''' and '''frisbee'''. If you want make frisbee images, you must build '''imagezip''' from  source code, which is not provided by frisbee debian package. For more information about frisbee, please vist http://www.emulab.net/software.php3 or read "README" attached with the source code.[[BR]][[BR]]
+ * ''apt-get install gridservices2 oml-collection-server''[[BR]]
+ The configuration of gridservices2 is under path /etc/gridservices2, Please go through README.txt under it first. Gridservices2 must enable at least 2 services, frisbee and PXE. The configuration of frisbee, '''frisbee.yaml''', looks like below
+{{{
+  1 # NOTE: use only 'spaces' to indent !
+  2 # ('tab' indents are not supported by the ruby yaml parser used to read this file)
+  3 #
+  4 # This is the Config file for the Frisbee GridService on the NICTA platform
+  5 #
+  6 ---
+  7 frisbee:
+  8
+  9   # Max. number of active daemons allowed
+ 10   maxDaemons: 10
+ 11
+ 12   testbed:
+ 13     default:
+ 14       # Directory images are stored
+ 15       imageDir: /home/node
+ 16       defaultImage: baseline0.4
+ 17
+ 18       # max bandwidth for frisbee server
+ 19       bandwidth: 50000000
+ 20
+ 21       # Multicast address to use for servicing images
+ 22       mcAddress: 224.0.0.2
+ 23       # Using ports starting at ...
+ 24       startPort: 7000
+ 25
+ 26       # Time out frisbee server if nobody requested it within TIMEOUT sec
+ 27       timeout: 3600
+ 28
+ 29       # Directory to find frisbee daemon
+ 30       frisbeeBin: /usr/sbin/frisbeed
+ 31
+ 32       # Local interface to bind to for frisbee traffic
+ 33       multicastIF: 10.10.0.10
+ 34
+ 35     indoor:
+ 36       imageDir: /home/node
+ 37       defaultImage: baseline0.4
+ 38       bandwidth: 50000000
+
+
+ 39       mcAddress: 224.0.0.2
+ 40       startPort: 7000
+ 41       timeout: 3600
+
+
+ 42       frisbeeBin: /usr/sbin/frisbeed
+ 43       multicastIF: 10.10.0.10
+}}} 
+
+ Comments:[[BR]]
+ Line 15: The place where frisbee image files are places.[[BR]]
+ Line 16: The file name of image without "ndz" suffix. For example, if file name is '''baseline0.4.ndz''', it should be set to '''baseline0.4'''. The file will be the default image when no explicit image is given in omf command.[[BR]]
+ Line 19: The maximum bandwidth in bps which frisbee can use to image hard drivers.[[BR]]
+ Line 30: The place where gridservices2 can find frisbeed.[[BR]]
+
+The configuration of PXE, '''pxe.yaml''', looks like below.
+{{{
+  1 pxe:
+  2   # Name of PXE config file
+  3   defImage: orbit-2.0.3-omf
+  4
+  5   # Directory pxe config files are stored
+  6   cfgDir: /tftpboot/pxelinux.cfg
+  7
+  8   # Maximum age of PXE symbolic link [sec]
+  9   linkLifetime: 900
+ 10   # linkLifetime: 5 # for testing only
+ 11
+ 12   # toIP: mapping from x@y to IP address
+ 13   # listAll: return array of x/y coodinates of all nodes in '[x,y]' form.
+ 14   #
+ 15   testbed:
+ 16     default:
+ 17       toIP: |
+ 18         |x,y|
+ 19           assertRange(x, 1..1, "unknown node #{x}@#{y}")
+ 20           assertRange(y, 1..20, "unknown node #{x}@#{y}")
+ 21           "10.10.#{x}.#{y}"
+ 22       listAll: |
+ 23         defGrid(1,4)
+}}}
+
+If all configuration are done, run following command to start gridservices2
+ * ''/etc/init.d/gridservices2 start''
+
+For CMC,
+ * ''apt-get install libmysqlclient15''
+ * ''apt-get install gridservices''[[BR]]
+ Some files of gridservices needs upgrade. Get a [http://witestlab.poly.edu/attachment/wiki/BOOTB/gridservices.tar.gz?format=raw tar ball] and extract it to replace fold /ect/gridservices.[[BR]]
+ Gridservices ONLY provides CMC service. It's configuration is defined in file /ect/gridservices/cmc.yaml, which looks like below.[[BR]]
+
+
+
+{{{
+  1 primaryIF: "128.238.34"
+  2 communicators:
+  3   default: &comm_default
+  4     ip: 10.1.200.1
+  5     port: 9030
+  6
+  7 testbeds:
+  8   grid:
+  9     x_max: 2
+ 10     y_max: 20
+ 11     ip_block: lambda {|x,y| "10.1.#{x}.#{y}"}
+ 12     inactive_list: [ ]
+ 13     3vStatus: 0.016
+ 14     5vStatus: 0.032
+ 15     12vStatus: 0.064
+}}}
+
+ Comments:[[BR]]
+ Line 1: This line defines the network address of the interface which connects to outside network. In my situation, CMC and Console are connected within network 128.238.34.*.[[BR]]
+ Line 4: This line define the IP address of the interface which connects to the nodes.[[BR]]
+ Line 9,10: The maximum value of two dimensions of testbed. The '''CM port''' of each node has IP address 10.1.'''x'''.'''y''', which is within the same network with 10.1.200.1.[[BR]]
+
+If all configuration are done, run following command to start gridservices
+ * ''/etc/init.d/gridservices start''
+
+If gridservices starts successfully, you can input the address below in web browser. You will find a couple of commands provided by CMC on web page and you can control CMC with the web interface.
+ * ''http://cmc:5012/cmc''
+
+
+== About the CM ==
+The ORBIT Chassis Manager (CM) is a simple, reliable, platform-independent subsystem for managing and autonomously monitoring the status of each node in the Witestbed. Basically, it is a small PCI card port on the node. Administrators can turn on/off and reboot nodes remotely and monitor the status of nodes throught CM. 
+User can access CM through serial console or telnet. For example. If a node's name is node2-3. the IP address of its '''control port''' is 10.10.2.3 and that of its '''CM port''' is 10.1.2.3. We can telnet to CM with command,
+
+ * ''telnet 10.1.2.3''
+
+We can also telnet to the node with either of two commands below
+
+ * ''telnet 10.1.2.3 3025''
+ * ''telnet 10.10.2.3''
+
+If you want to access CM through serial console, please set the baud rate to '''57600kps'''.
+
+== Updating the node BIOS ==
+
+The new BIOS fixes some bugs and provides better support to network boot and power management. The detail description could be found [wiki:Internal/Infrastructure/SetupTestbed/UpdateTheNodeBios here].