Changes between Version 1 and Version 2 of Software/fImages/kPxeImage

Timestamp:

Jun 20, 2014, 6:59:20 PM (10 years ago)

Author:

ssugrim

Comment:

—

Software/fImages/kPxeImage

@@ -1 +1 @@
 == PXE Image ==
 
-=== Build Instructions for PXE images ===
-
+=== Build Instructions for PXE Images ===
+
+''' Build Setup '''
+
+The current build package for making pxe images lives on:
+{{{
+console.sb5.orbit-lab.org:/root/pxe/omf/pxe/
+}}}
+
+In there is a make master make file with all the parameters to build the kernel / initramfs image pair. It will also build a pxelinux.bin from sources. This version of pxelinux.bin doesn't need to match with kernel / initramfs pair. 
+
+in the Makefiel the values of the following variables control what kernel directory the make process will consult to build a kernel:
+{{{
+KERNEL_VERSION = 3
+KERNEL_PATCHLEVEL = 0
+KERNEL_SUBLEVEL = 4
+}}}
+
+In the /root/pxe/omf/pxe/src directory should be a directory named linux-VERSIONNUMBERS. In this directory is the config file for the kernel being built. 
+
+Additionally ins the same source dir is a busbox-VERSION directory with a .config file that controls what features are built into busybox.
+
+''' Building in NTP clock updates '''
+
+ 1. Even though the version of the kernel source tree is
+    specified in the make file, the build process does not 
+    use the kernel config file that was downloaded from the
+    mirror. Instead it uses the file named
+    {{{
+    linux-omf-pxe-#.#.#.config
+    }}}
+    in the directory:
+    {{{
+    /root/pxe/omf/pxe/config
+    }}} 
+ 2. the same is true for the busy box, I think it uses the file:
+    {{{
+    busybox-winlab.config
+    }}}
+ 3. Finally the startup script rcS.actual will be copied to the init-ram-fs and run as part
+    of the pxe preOS sequence. We recently modified that file to have it poll NTP and 
+    correct the system clock. We added the following lines:
+    {{{
+    /usr/sbin/ntpd -d -n -q -p consolec
+    /sbin/hwclock --directisa -w
+    }}}
+
+''' Bumping the Kernel Version '''
+
+Made a few modification for the most current version based on kernel 3.11.1
+
+ 1. Copied a running ubuntu config from an ubuntu 13.10 install to serve as sane default. It does throw one question about vesa support which I said yes to, but mostly complies. Happily.
+ 2. After some testing we decided to remove R8169 (realtek) support and only enumerate the intel interfaces (this will cause some other boards to pxe/boot off eth0).
+ 3. Made specially modified pxelinux.cfg file that boots off of eth0 for the newest (Gen4 asus) nodes
+
+''' 
+
+=== Usage Notes ===
+
+''' Makefile '''
+
+PXE has a 'changelog' in the top directory. The following extracts the version number from the first line in the changelog - enjoy:
+
+{{{
+        24      VER = $(shell grep -o -E '[0-9]+\.[0-9.-]+' -m 1 changelog)  
+}}}
+
+The following line copies a directory from one place to the other, but without ".svn" directories and emacs temporary files (*~)
+
+{{{
+        119             tar -C static --exclude=.svn --exclude='*~' -cf - . | (cd $(RDISK); tar -xf -) 
+}}}
+
+The image target should now exist on its own (the incuded software is a sub target). I just added a few 'sudo' commands to allow it to compile from a user account.
+
+The main addition is the apt target:
+{{{
+        141     apt: dirs kernel image  
+        142             @echo Building APT V$(VER)  
+        143             rm -rf $(BUILD_APT_DIR)  
+        144             install -d $(BUILD_APT_DIR)  
+        145             cp -rp debian $(BUILD_APT_DIR)  
+        146             rm -rf $(BUILD_APT_DIR)/debian/.svn       
+        147             mv $(BUILD_APT_DIR)/debian/Makefile $(BUILD_APT_DIR)  
+        148             cp changelog $(BUILD_APT_DIR)/debian  
+        149             cp ../copyright $(BUILD_APT_DIR)/debian  
+        150             if [ -r README.debian ] ; then \  
+        151                     cp README.debian $(BUILD_APT_DIR)/debian; \  
+        152             fi  
+        153             env O_VER=$(VER) O_KERNEL_V=$(KERNEL_VERSION) $(MAKE) -C $(BUILD_APT_DIR) deb  
+        154     #       $(MAKE) DESTDIR=$(BUILD_APT_DIR)/debian/$(APT_NAME) install  
+        155      
+        156     apt-install: apt  
+        157             scp $(BUILD_APT_DIR)/../*.deb $(REPOSITORY):$(REPOSITORY_ROOT)/binary  
+        158             ssh $(REPOSITORY) sudo $(REPOSITORY_ROOT)/rebuild.sh  
+}}}
+
+The BUILD_APT_DIR variable was defined above, but it's in build. 143 and 144 create new apt directory. I added a new top level directory 'debian' to hold all the debian related files. This directory is copied into the apt build area (145, 146 - Should really be using something similar to 119). The 'debian' directory contains the Makefile which should really be a level up in the build area. Debian apt's also need a change log which we copy into the build area (148) as well. Same goes for the copyright file which we take from the nodehandler proper (149), and the README.debian, if it exists (150-152). 153 is a bit tricky. Basically we need to know the version of this apt, as well as the kernel version for the 'install' target in 'debian/Makefile'. However, we are calling the 'deb' target, which calls some dpkg util, which ultimately uses the 'debian/rules' file to call the 'install' target of 'debian/Makefile' - still following? In short, we put the two versions into environment variables "env O_VER=... make ...' before calling the first make which the 'install' target can retrieve and use (remember a non-initialized Makefile variable takes it's value from an environment variable of the same name if it exists).
+
+''' debian/Makefile '''
+
+{{{
+install:
+        echo "INSTALL $(O_VER) $(O_KERNEL_V)"
+        install -d $(TFTP_DIR)
+        install -m666 ../initrd-orbit-pxe-$(O_VER).img $(TFTP_DIR)
+        install -d $(CFG_DIR)
+        sed -e s/@KERNEL_VER@/$(O_KERNEL_V)/ ../../config/default.orbit | sed -e s/@VER@/$(O_VER)/ > $(CFG_DIR)/default.orbit-$(O_VER)
+        install -m755 ../../config/makeLink $(CFG_DIR)
+}}}
+
+The O_VER and O_KERNEL_V come form the environment variables defined in 153 above. This apt installs the image we built into /tftpboot, as well as 'default.orbit-XXX' and 'makeLink' in /tftpboot/pxelinux.cfg. The 'default.orbit-XXX' file is created from the  template stored in the config directory.
+
+{{{
+DEFAULT boel
+
+LABEL boel
+KERNEL linux-@KERNEL_VER@
+APPEND vga=extended root=/dev/ram0 rw load_ramdisk=1 prompt_ramdisk=1 ramdisk_size=32768 init=/vmlinuz initrd=initrd-orbit-pxe-@VER@.img console=ttyS0,9600
+DISPLAY message.txt
+PROMPT 1
+TIMEOUT 1
+}}}
+
+
+''' Default Pxe Issues '''
+
+As part of the normal (non-pxe) boot process the nodes must pass though the pxe phase of the bios and "fail". The pxe client on all of the nodes should contact the tftp server and request a file named with the node ip adress in hex. It will then proceed to remove the trailing character until it runs out of characters. Once that process fails, the node will look for a file named default (no extention).
+
+The current default used the localboot pxe directive (part of the standard) to instruct the nodes to fail over to disk. The file (now named default.localboot) has the following contents:
+{{{
+default harddisk
+
+label harddisk
+  localboot 0
+}}}
+
+A recent test with some of the newest mother boards however failed. The pxeclient did not honor the locaboot 0 option (or rather they locked up when given this option). A work around documented [http://www.gossamer-threads.com/lists/syslinux/users/7127 here] (stored as a pdf for future use) recommended trying the syslinux chainloader. This required a new default file with the contents:
+{{{
+default harddisk
+
+label harddisk
+KERNEL chain.c32 
+APPEND hd0 0
+}}}
+
+=== Service Requirements ===
+
+
+To bootstrap the testbed nodes, in fact any server in the system, we use [http://en.wikipedia.org/wiki/PXE PXE] to have the nodes load an image which contains a [wiki:NodeHandler/NodeAgent nodeagent] and a [http://www.cs.utah.edu/flux/papers/frisbee-usenix03-base.html frisbee] client. 
+
+PXE requires the following support:
+
+ 1. A [http://en.wikipedia.org/wiki/Dynamic_Host_Configuration_Protocol DHCP] server pointing to an TFTP server
+ 1. A [http://en.wikipedia.org/wiki/TFTP TFTP] server proving the following:
+   1. A network bootstrap program (NBP)
+   1. A file containing instructions for the NBP
+   1. A boot image
+ 1. A PXE [wiki:NodeHandler/GridService grid service]
+
+''Ideally the later referenced "orbit-pxe-server" apt package should include dependencies to all required packages and should setup everything. For instance, it should depend on gridservices and should then automatically instantiate the PXE grid service.''
+
+''' DHCP Server '''
+
+To support PXE we need a DHCP server. This server does not necessarily need to be instantiated at the same machine as the TFTP server, but it's configuration needs to point to it.
+
+Installing a DHCP server on Debian is as simple as:
+
+{{{
+% apt-get install dhcp
+}}}
+
+
+To support PXE, we need to add to /etc/dhcpd.conf something like:
+
+{{{
+option domain-name "example.com";
+
+default-lease-time 600;
+max-lease-time 7200;
+
+allow booting;
+
+# The next paragraph needs to be modified to fit your case
+subnet 192.168.1.0 netmask 255.255.255.0 {
+  range 192.168.1.200 192.168.1.253;
+  option broadcast-address 192.168.1.255;
+# the gateway address which can be different
+# (access to the internet for instance)
+  option routers 192.168.1.1;
+# indicate the dns you want to use
+  option domain-name-servers 192.168.1.3;
+}
+
+group {
+# The address of the TFTP server that hosts the PXE image.
+ next-server 192.168.1.3;
+ host tftpclient {
+# tftp client hardware address
+  hardware ethernet  00:10:DC:27:6C:15;
+
+# The PXE boot loader that instructs the client to load a script corresponding to its IP address.
+# The script defines the kernel and memory based image to download and boot.  Think of it as a 
+# network aware grub/lilo type boot loader.
+  filename "/tftpboot/pxelinux.bin";
+ }
+}
+
+}}}
+
+After you have edited the dhcpd configuration file, restart it with "/etc/init.d/dhcpd restart".
+
+''' Installing a TFTP server '''
+
+Debian makes that easy.  We prefer atftpd since its the only tftp server that has verbose output.  It also gives the ability to control the number of threads which handle transfers.  With 400+ simultaneous clients, a good tftp server is required.
+{{{
+% apt-get install atftd
+}}}
+
+''' Installing NBP -- pxelinux '''
+
+'pxelinux.bin' is a file found in the syslinux package. [http://syslinux.zytor.com/ Syslinux]  The nitty gritty of how it works, what to do to make a machine netboot, and all of the other specifics can be found at [http://syslinux.zytor.com/pxe.php Syslinux].
+
+
+''' Installing an PXE image '''
+
+The PXE client gets the address of the TFTP server through DHCP. Taking it's IP address it attempts to download a specific file containing further instructions from the TFPT server. The name of the file is created from the PXE client's IP address. For Orbit, the instructions are identical for every node and are stored in "/tftpboot/pxelinux.cfg/default.orbit-''version''". This file instructs the node to fetch the image "/tftpboot/initrd-orbit-pxe-''version''.img" from the TFTP server and boot into it.
+
+Both files will be installed through:
+
+{{{
+% apt-get install orbit-pxe-server
+}}}
+
+''' Installing the PXE Grid Service '''
+
+Every service on Orbit which can be controlled by the experimenter requires a [wiki:NodeHandler/GridService grid service] component. The PXE grid service must be installed on the TFTP server.
+
+{{{
+% apt-get install gridservices
+% cd /etc/gridservices/enabled
+% ln -s ../available/pxe .
+% /etc/init.d/gridservices restart
+}}}