Orbit > Tutorial > Methodology
Testbed Experiments
This section begins with a point concerning experimental methodology then describes each of the steps that might be needed to perform an ORBIT Radio Grid Testbed experiment. The architecture of the ORBIT Radio Grid Testbed was designed to support each phase of the lifecycle of an experiment, see ''ORBIT Testbed Software Architecture: Supporting Experiments as a Service''.
Experimental Methodology
During each phase of the design, development, execution and interpretation of an ORBIT Radio Grid Testbed experiment it is important to keep a written laboratory notebook or at least regularly record blog entries. The Instructions for Keeping Research Records in the front of most laboratory notebooks apply to ORBIT Radio Grid Testbed experiments. Some instructions may need to be adapted, for example, to record the names and version numbers of procedures, applications and system software or to record commands issued using a GUI interface.
While working on ORBIT projects, each student is expected to record regular blog entries. Lack of blog entries implies the student is no longer working on the associated project. To add a blog entry follow the "create content" link on the left menu of local.orbit-lab.org.
Laboratory notebook or blog entries should have sufficient information to help the experimenter or a colleague keep track of his or her work, for example, to be able to say a week or a month later how you did something and why you did it that way. It should not include "editorial" or subjective comments that may limit later claims.
Experimental Design
Much of the work in conducting an experiment is in its design and development. Running it and interpreting the results should be relatively easy when the design is based on a good model of the underlying phenomena, sufficient measurement points are defined, and the application and any system software are properly written. The design of an experiment will probably evolve during the development of models of the underlying phenomena, development of a strategy to map them to the testbed grid, development of the application software and development of any required system software.
During the design of an experiment one may need to develop new techniques specific to it, but a number of standard design features are already in use on the ORBIT Radio Grid Testbed, and many more are expected to be developed in due course. A number of design issues and approaches are discussed in ''ORBIT Testbed Software Architecture: Supporting Experiments as a Service''.
Mapping
When designing an experiment, one often has to translate a real-world network topology into specific experimental node assignments. An ORBIT paper ''Wireless Link SNR Mapping Onto An Indoor Testbed'' addresses the issue of mapping a network topology with given SNR requirements onto the actual ORBIT testbed.
This paper illustrates how one might use analysis and simulation to develop some aspects of experimental design. To emulate communication links of specified quality on the ORBIT Radio Grid Testbed it was found through analysis and simulation that when the interferer node is located on the corner across from the access point, one can map a communication link with a given received signal-to-interference-and-noise-ratio (SINR) to the corresponding link signal-to-noise-ratio (SNR) with the mapping range on the order of 57dB and average mapping error on the order of 2dB.
Repeatability
Ideally, experimental results should be repeatable, and ORBIT Radio Grid Testbed experiment results should be no different, despite difficulties with complex interactions among physical, medium-access and network layers in wireless environments. This issue is addressed in ''Addressing Repeatability in Wireless Experiments using ORBIT Testbed''. Current results of any available manual or automated calibration procedures should be recorded as part of ones experimental data.
Design Validation
If one is only modifying a few parameters in an already well-understood experiment, by reviewing its experimental design one should be able to verify that each of its models of underlying phenomena correspond to physical reality, and by inspecting its application and system software verify that it accurately implements the underlying models and each of the higher-level protocols. Besides reviewing the design and inspecting the code, one might want to plan to use independent methods like analysis of measurements from RF or spectrum analyzers and passive packet monitors to confirm its intended operation.
Using the ORBIT Radio Grid Testbed
The ORBIT Radio Grid Testbed was designed to be used as a service. This section describes how to use the ORBIT Radio Grid Testbed as a service. At the highest level, the input to a well-designed experiment on the ORBIT testbed is a script and the output is a database of measurements. This system architecture enables its efficient operation as a service and makes it possible to run experiments remotely.
PC or Workstation Requirements
You will need access to a PC or workstation that can run SSH to login to an ORBIT application server, submit the script for the experiment, use MySQL to browse the experimental results, and run Excel or Matlab to interpret the results. You may also be able to monitor the progress of your experiment using a browser.
Getting an Account
You will need an ORBIT user login (distinct from a WINLAB user login) to use the testbed. At the very least you will need it to login into an ORBIT server to submit a script to run an experiment and to access an ORBIT server to get the set of measurements recorded during the experiment in order to interpret the results of the experiment.
Scheduling
Only one ORBIT Radio Grid Testbed experiment can be run at a time. Details of how the testbed will be made available for local and remote experiments are under discussion.
Audit and Security Issues
Users working on a specific project will be given access to ORBIT Radio Grid Testbed resources using role-based access control. Roles and privileges are expected to change over time or even as an experiment progresses. The primary resource would be time on the testbed. Users are expected to backup their own measurements on databases. These controls and policies are under development and are subject to change.
Developing the Script for an Experiment
A script is used to run an ORBIT Radio Grid Testbed experiment. The script contains the configuration of the experiment, specifies its application and operating system software, describes its initialization and any parameters, and describes each phase of its operation.
ORBIT Radio Grid Testbed scripts are written in Ruby, an easily understood, extensible, dynamic, object-oriented scripting language. Ruby has been extended into the testbed user's domain with a number of methods. Besides its extensibility and object-orientation, Ruby is concise and consistent. It is a dynamic language supporting closure, that is, a Ruby block can use its original scope information even if the environment in which it was defined would otherwise have disappeared.
An ORBIT Radio Grid Testbed script therefore is written in Ruby primarily using ORBIT-specific methods. A brief description or Ruby follows, then information on the ORBIT-specific methods used to define and control an ORBIT Radio Grid Testbed experiment, then a note on script coding techniques.
Ruby Resources
There are two primary resources for the Ruby programming language. The first is the Ruby Web site. It contains current information on Ruby and a link to download it for free. The second is an excellent tutorial and reference book, Programming Ruby The Pragmatic Programmer's Guide, Second Edition, Dave Thomas with Chad Fowler and Andy Hunt, Pragmatic Bookshelf, October 2004, ISBN:0-9745140-5-5, 864 pages; see http://www.pragmaticprogrammer.com/titles/ruby.
Ruby
Ruby's syntax is similar to other object-oriented languages like Java or C++. In Ruby everything that is represented is an object, even numbers like 1. Functions or procedures are termed methods and are called using the name of the object, a period, and the name of the method, e.g., node.prototype. Methods in ruby are usually small.
Variables associated with objects are termed properties and are referred to by the name of the object, a period, and the name of the property, e.g., w.essid. The formatting of ones use of the language is at the discretion of the programmer, but consistency is helpful, especially in the use of indentation and the placement of the opening and closing braces for code blocks.
ORBIT-specific Methods
Four ORBIT-specific methods are available for debugging help: error( ), warn( ), info( ), and debug( ). How do they work? Like assertions? What arguments do they take? Besides these debugging methods, there are three classes of ORBIT-specific methods discussed below: implicit, Experiment, and NodeSet.
Among the implicit methods two are used to define resources:
defProperty(name, value, description)
defNodes(setName, nodeList) {}
Some ORBIT-specific methods work with node sets to specify a set of nodes for each member of which the associated block of code is executed:
nodes(setName) {}
allNodes {}
nodes("_ALL_") {}
Other ORBIT-specific methods use events which delay execution of the associated block of code and sequential execution until that event is true:
whenAll(setTest, nodeTest, interval = 5) {}
whenAllInstalled()
whenAll("_ALL_", "apps/app/status[text()='INSTALLED.OK']") {}
wait time_in_sec
The Experiment methods and properties are ORBIT-specific methods and properties associated with the Experiment object. They have the form Experiment.method. These methods include:
name = "tutorial-1a"
project = "orbit:tutorial"
props.propName = ""
Done
It is anticipated that the 'Experiment' prefix will be removed in future versions.
Other ORBIT-specific methods are NodeSet methods that set properties for the specified set of nodes. They have the form
NodeSet ( nodes(setName).method )
image =
pxeImage =
prototype(name) {|node|
}
onNodesUp {|node|
}
startApplication(appName), startApplications
stopApplication(appName), stopApplications
resource
net.if_name
The defNodes method declares the nodes used in the experiment:
defNodes(setName:string, nodeList) {}
There are various ways to declare node list:
A single node: [x,y]
multiple nodes: [[x1,y1], [x2, y2], [x3, y3]]
ranges of nodes: [x1..x2, y1..y2]
the entire grid: [1..20, 1..20]
with other node sets: [nodeSet1, nodeSet2]
These methods are useful for declaring common functionality over multiple sets. The various network parameters that may be configured within a defNodes or allNodes block are shown in Figure 9 below, here the first and second Ethernet interfaces are e0 and e1, and the first and second wireless interfaces are w0 and w1.
net e0, e1 arp = true|false En/disable ARP forward = true|false Enable forwarding ip = address/netmask IP address of interface up = true|false En/disable interface route w0, w1 All the above channelI = 1..11; 36, 40, 44, 48, 52, 56, 60, 64, 149, 153, 157, 161 essid = string frequencyI = 2.412 2.462 (5 Mhz steps); 5.18Ghz (20Mhz steps) mode = master|managed|ad-hocI|monitorI rtsA = packetSizeThreshold [bytes] rateI = 1, 5, 11; 6, 9, 12, 18, 24, 36, 48, 54 tx_power = -12 .. 15 dBm (intel), 0 .. 20 dBm (atheros) type = a/b/g
Figure 9. Configurable Network Parameters
ORBIT Script Coding Techniques
There are many known techniques for defining an experiment with parameters that do not change over the course of the experiment and for controlling experiments with parameters that change while the experiment runs. These techniques are available in the standard scripts for the baseline ORBIT experiments [where?]. It is also possible to change parameters based on the experiment's own measured behavior, e.g., to run until the observed packet error rate exceeds a given value, see ''ORBIT Testbed Software Architecture: Supporting Experiments as a Service''.
Developing Application Software
The experimenter may need to develop application software, for example, to emulate the operation of a new network protocol. An ORBIT Radio Grid Testbed application operates in the environment shown in Figure 10 below. An application needs to be integrated into this application harness. Once an application is developed it needs to have an application description developed and to be loaded onto an application server and assigned an id. The process for developing this application description and loading the application software is as follows ?.
Figure 10. Experiment Execution Architecture (see page 12 of http://www.orbit-lab.org/doc/tutorial).
Development Tools for Application Software
Three major tools are provided to the ORBIT experimenter to develop an application in C on Linux: ORBIT Traffic Generator/Receiver (OTG/OTR), Libmac and the ORBIT Measurement Framework (OML). The ORBIT traffic generator/receiver (OTG/OTR) illustrated in Figure 11 below is a modular traffic generator/receiver that is integrated with OML and Libmac. OTG/OTR has pluggable traffic models and transport protocols, see ''ORBIT Measurements Framework and Library (OML): Motivations, Design, Implementation, and Features''.
Figure 11. ORBIT Traffic Generator/Receiver (OTG/OTR)(on page 53 of http://www.orbit-lab.org/doc/tutorial).
Libmac is a user-space C library that brings the wireless driver application program interface (API) into user space allowing one to get or set values of driver parameters. This library supports operation for a variable number of parameters per call. The call mac_get_params(struct mac_params *h, struct mac_ifinfo_list *if_ptr, int argc, ) requests the wireless device driver to GET parameter information. The call mac_set_params(struct mac_params *h, struct mac_ifinfo_list *if_ptr, int argc, ) requests device driver to SET parameter information.
The ORBIT Measurement Framework (OML) was discussed in the first section of this tutorial as one of the ORBIT Services. It is a distributed software framework enabling real-time collection of data, and, as such, has a client application programming interface (API). A developer can use this client API through a web interface to define the measurement points and parameters for his or her application. Measurement points and their frequency of collection are an important part of ones experimental plan.
The definition of a measurement point is illustrated in Figure 12 below. Measurement point definitions are saved as an XML-based configuration file, and source code for the measurement client is automatically generated that contains application-specific methods that handle type-safe data collection. This source code can be compiled and linked with the application as illustrated by a Makefile in Figure 13 and sample application code in Figure 14 below.
<measurement-points> <measurement-point id="group-1"> <metric id="rssi" type="float"/> <metric id="noise" type="float"/> </measurement-point> <measurement-point id="group-2"> <metric id="throughput" type="int"/> </measurement-point> </measurement-points>
Figure 12. Defining Measurement Points
Makefile: $(INC_DIR)/oml_%.h : $(ETC_DIR)/%.xml mkdir -p $(INC_DIR) wget -q http://www.orbit-lab.org/oml/client_wrapper\ --post-file $< -O - \ | tar -C $(BUILD_DIR) -xzf oml_foo.h: int oml_group1(float rssi, float noise); int oml_group2(int throughput);
Figure 13. Makefile and include file for Measurement Points
oml_init(&argc, &argv, NULL);
if (r_data->send_option == 1) {
buffer->rssi = recv_packet_params.rssi ;
buffer->noise = recv_packet_params.noise;
oml_group1(buffer->rssi, buffer->noise);
} else {
log(LOG_ERR, "Unknown receive option! \n");
}
lost_packets = pck_id.seqnum - old - 1;
oml_group2(lost_packets);
Figure 14. Application Code Sample
Because not all measurements are needed and not all measurement samples are needed, OML supports preprocessing or filtering at source to reduce the amount of reported and recorded data. Filters are defined by experimenter, and experimenter-provided filters are supported. Figure 15 below illustrates the client-side data flow. Collection of and access to the recorded data requires the use of a database schema. OML automatically generates the appropriate schema as diagrammed in Figure 16 below.
Figure 15. Client-side Data Flow (see page 66 of http://www.orbit-lab.org/doc/tutorial).
Figure 16. Server-side DB Schema Generation (see page 67 of http://www.orbit-lab.org/doc/tutorial).
Developing Device Drivers
A possibly difficult aspect of having to develop a completely new application would be to have to develop a device driver for Linux or for another operation system to support a new communications device. Even a simple modification to an existing device driver requires extensive testing. Once a device driver has been developed it needs to be loaded onto an application server. The process for loading a device driver is as follows ?.
ORBIT Sandboxes
When developing a new application or modifying system software and especially when developing a device driver, it is best to have an electromagnetically isolated hardware environment. An ORBIT Sandbox contains two ORBIT Radio Nodes and one console node with their 802.11 a/b/g cards connected by coaxial cable. Each ORBIT Sandbox is supported by a shared services node. ORBIT Sandboxes allow multiple projects to develop multiple ORBIT Radio Grid Testbed experiments in parallel without interfering with experiments currently running on the testbed.
Loading Your Own OS
The ORBIT Radio Nodes usually run Debian Linux. If you need to load a different operating system, perhaps to use certain drivers or perhaps to run a harder real-time application, it can be done. However, the support libraries and services might need to be adapted to work with your operating system. The operating system needs to be loaded onto an application server. The process for loading an operating system is as follows ?.
Running an Experiment
Besides being operated as a service, another important design goal of the ORBIT Radio Grid Testbed is for experiments with it to be able to be operated remotely. For the time being though it is best if the experimenter is present at the site to observe the testbed and its operation.
Once you have developed an experimental script and any new application or system software required to run it, from the command line on your computer or an SSH tool, create a secure shell connection to the ORBIT Lab console. Figure 17 below contains a sample session for the "Hello World" experiment.
myMachine> ssh console.orbit-lab.org console> nodeAgent test:exp:tutorial-1a INFO init: NodeHandler Version 1.69 INFO init: Experiment ID: 2005-06-07-14-09-38 INFO Experiment: load test:exp:tutorial-1a INFO Experiment: DONE! INFO run: Experiment finished after 2:56
Figure 17. "Hello World" Experiment Console Session
How to Tell If An Experiment is Working Properly
Because measurements are available in the database as they are received, it is possible to monitor an experiment as it is running using the MySQL browser as shown in Figure 18. One can monitor the Frisbee, pxe and xml data using a browser and specific port addesses. What are they? Which ports? One also might design some sanity checks into your experiment, that is, initial modes of operation of certain patterns of parameters that have analytically predictable results.
Figure 18. Real-time monitoring (see page 68 of http://www.orbit-lab.org/doc/tutorial).
Analyzing Results
It is important to understand how measurements were collected to be able to interpret them. The ORBIT Measurement Framework (OML) that was described above provides tools to insert points to tap available information and to effectively collect that information in a timely manner. A number of different tools are available to interpret experimental results. The choice of tools depends upon availability and the nature of the measurements. Microsoft Excel can be used to analyze an experiment as shown in Figure 19 below. Matlab can also be used. Sample Matlab code is shown in Figure 20 and the resulting graph in Figure 21.
Figure 19. Usage of Microsoft Excel (page 69 of of http://www.orbit-lab.org/doc/tutorial).
function nsf(dbServer, dbUser, dbPW, database);
% Part where we retrieve data from the database;
mysql('open',dbServer, dbUser, dbPW);
mysql('use', database);
output = struct('time',[],'thr_all',[],'node',[]);
[output.time, output.thr_all, output.node] = mysql('select timestamp, throughput, node_id from group2');
[thru1_4, time1_4, thru3_1, time3_1] = sort_mysql(output);
% Finally, the plotting part
subplot(2,1,1);
plot(time1_4, thru1_4, '-*');
title('Throughput On Obstructed Link');
xlabel('Time (sec)'); ylabel('Throuhput (bps)'); grid on;
subplot(2,1,2);
plot(time3_1, thru3_1, '-*');
title('Throughput On Monitor Node'); xlabel('Time (sec)');
ylabel('Throuhput (bps)'); grid on;
Figure 20. Matlab Code
Figure 21. Matlab Graph (see graph on page 70 of http://www.orbit-lab.org/doc/tutorial).
