wiki:DSC/zdc_framework

Version 23 (modified by seskar, 11 years ago) ( diff )

—

Match Execution Framework

Table of Contents

  1. DARPA Spectrum Challenge
    1. Hurdles
    2. Wildcard tournament
    3. Preliminary round
    4. Final round
  2. DARPA Spectrum Challenge Q&A
    1. Tournament Q&A
    2. Hurdles Q&A
  3. Tutorial for Hurdle3-like experiment
    1. Objective
    2. Hardware / Software Resources Used
    3. Setting up the nodes
    4. Running the experiment script
  4. Tournament game example !#1
    1. Objective
    2. Hardware / Software Resources Used
    3. Setting up the nodes
    4. Running the experiment script
  5. Match Execution Framework
    1. Objective
    2. Hardware / Software Resources Used
    3. Image Validation and Submission
    4. Setting Up the Arena
    5. Running the Experiment Script
    6. Running Matches on Sandboxes
  6. Bot Images
  7. Tournament Scripts
    1. Objective
    2. Running Competitive Matches
    3. Running Cooperative Matches
  8. Match Visualization
    1. Competitive Match Visualization
    2. Cooperative Match Visualization
  9. Calibration Reports (2014-03-13)
    1. Primary Competitive
    2. Secondary Competitive
    3. Primary Cooperative
    4. Secondary Cooperative
    5. Full Arena
    6. Calibration Procedure
  10. Image Validation Procedure (Preliminary Torunament)
  11. Image Validation Procedure (Final Tournament)
    1. Validate With Basic Script
    2. Validate With Tournament Scripts
    3. Additional Arena Testing

Objective

Use an OEDL script to execute either a competitive-style or cooperative-style game for both Wildcard and Preliminary matches. Even though the use of the framework is illustrated by using 2 house bots (or 3 in the case of a cooperative game), in the actual competition, the teams will use their own radio implementations. An additional purpose of the framework is to provide certification to the images that teams submit for the Wildcard and Preliminary matches: teams must use the script in this framework to certify their image before submission. Certification is required for both competitive and cooperative style matches.

The OEDL script uses three pair of nodes: dsc-teamA, dsc-teamB, dsc-teamC (the actual topology descriptions are system:topo:dsc-team{A,B,C}). The competitive matches will use teamA and teamC nodes, while cooperative matches will use teamA, teamB and teamC nodes (contestant radios will be only on teamA nodes for the Wildcard tournament and for image certification, while house bots will be using teamB and teamC nodes); configurations should be considered to be fixed and will be changed only in the case of hardware failures.

Hardware / Software Resources Used

  1. Four grid nodes with USRP N210s (two for each team) for competitive match and six nodes with USRP N210s for cooperative match (two for each team).
  2. dsc-wbot1.ndz,dsc-wbot2.ndz,dsc-wbot3.ndz: house bot disk images with the prerequisite software to configure the USRPs and execute a house radio bot.
  3. team-image.ndz: disk image that contains the team's radio.
  4. Scoring packet server: This is the server that resides on the infrastructure machine and is used as the source of data packets as well as the sink for teams to submit received packets for scoring.
  5. The OEDL experiment script to execute the matches.

Teams must ensure that their images contain /root/dsc-code_tx and /root/dsc-code_rx. Only these names will be used for testing and scoring. Further, these executables/scripts must be able to receive as input three parameters:

  • -f 1700000000 : option for specifying the center frequency (in this example 1.7GHz)
  • -s idb2 : option for specifying the packet server host name (in this example idb2)
  • —mode COMP or —mode COOP : option that specifies whether the match style is a competitive or cooperative match.

Please note that idb2 and 1700000000 are provided merely for illustration purposes and that the teams must allow for the values passed to be entered by the evaluators. Finally, no modifications to this execution format is allowed, and no additional fields will be used during evaluation.

Teams should also implement their radios such that packets are drawn from port 5123, while packets are delivered to port 5125 on the packet server specified with the -s argument. An illustration of this can be seen in the Python code in the table below:

Transmitter Receiver 
class dsc_pkt_src(object):
    def __init__(self, server, port=5123 ):
        self.pkt_size = 1440 # 1440 bytes of data  
        self.pkt_server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        self.pkt_server_socket.connect((server,port))
        self.MESSAGE = struct.pack('!l', self.pkt_size)
        self.pkt_server_socket.send(self.MESSAGE)

    def read(self):
           try:
              data = self.pkt_server_socket.recv(self.pkt_size)
           except socket.error:
              print "Connection to packet server closed"
              return ''
           self.pkt_server_socket.send(self.MESSAGE)
        return data

class dsc_pkt_sink(object):    
      def __init__(self, server, port=5125):       
          self.pkt_sink_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
          self.pkt_sink_socket.connect((server,port))

      def send(self, payload):
          try:
             self.pkt_sink_socket.recv(4)
             self.pkt_sink_socket.send(payload)
          except socket.error:
             print "Connection to packet sink closed"

Teams are also required to label their image with the image name they intend to use for submission, which should be stored in file named /.orbit_image on the root of the filesystem before node is saved. For example, if the team intends to submit an image with image name orbit-bot.ndz the /.orbit_image file should contain orbit-bot.

Packet Server

The packet server application running on the infrastructure machine has

  • a packet source at port 5123 which serves fixed (1440 bytes). Clients can request packets from server with the following handshaking process after a TCP socket connection has been established:
    1. client sends server the request with packet size as 4 byte word (ignored for the matches).
    2. server sends client a single data packet (fixed at 1440 bytes)

The packet source sends out the first packet to each team only after it receives packet requests from all the teams (2 teams for competitive match and 3 for co-operative) and the start signal from the experiment script. It closes transmit socket on the first packet request after the end-of-file is reached or at the 180 second timeout. It will report the score via an email to whoever has a grid reservation (if at least one packet is received correctly).

  • a packet sink at port 5125 which receive packets from client and verifies that it matches one of the sent packets. Clients can send data to server after a connection has been established:
    1. server sends client a ready packet (4 byte word).
    2. client sends server data for scoring.

  • a control interface used by the experiment script to prepare the packet server for a match by setting its mode, duration etc., and to send a start signal so the packet source can send packets to the teams.

Image Validation and Submission

For the image to be considered for the tournament it must be successfully validated in both the Competitive and Cooperative modes. Successful execution of the system:exp:dsc-match script (i.e. with at least one error-free packet received by the scoring server) is considered validation of the image for that match type. Please note that the email sent by the scoring server to contestants (as well as the ORBIT team) contains all the information about images that were used for the match. It is essential that teams validate their final image without any modifications. The best way to ensure that is to simply save the final image, re-image the nodes and execute the scripts (without any further login/access to the nodes). This guarantees that the image checksum will be the same with what was reported in the validation email. You may run the validation script as often as desired and on multiple images. If the team successfully validates multiple images, the ORBIT team needs to be informed of the actual image to be used, otherwise the last successfully validated image (i.e. passing both type of matches) will be used.

Each team is also advised that the file /.orbit_image should contain the final image name (or team name) in order to facilitate proper reporting by the framework.

Setting Up the Arena

  • To get started, first make a reservation on the ​Orbit Scheduler for using the Grid.
  • After logging into grid console, make sure all nodes are turned off (in the example below, turn off all the nodes in the grid): 
    username@console.grid:~$ omf tell -a offh -t system:topo:all
  • Verify state of node before continuing. Make sure nodes are in the POWEROFF state. 
    username@console.grid:~$ omf stat
  • Image the two nodes to be used for the first team's radios (team A nodes): 
    username@console.grid:~$ omf load -i dsc-wbot1.ndz -t system:topo:dsc-teamA

In the case of Wildcard matches these two nodes will receive the competitor's image (and hence replace the house bot image name with the team's image name).

  • Image the two nodes to be used for the second team's radios (in this case team C nodes): 
    username@console.grid:~$ omf load -i dsc-wbot2.ndz -t system:topo:dsc-teamC
  • FOR COOPERATIVE MATCHES: image the third pair of nodes to be used for the third team's radios (in this case team B nodes): 
    username@console.grid:~$ omf load -i dsc-wbot3.ndz -t system:topo:dsc-teamB
  • Turn nodes back on and verify they are in POWERON state 
    username@console.grid:~$ omf tell -a on -t system:topo:dsc-teamA
username@console.grid:~$ omf tell -a on -t system:topo:dsc-teamC
  • FOR COOPERATIVE MATCHES: turn on the third pair of nodes 
    username@console.grid:~$ omf tell -a on -t system:topo:dsc-teamB

Running the Experiment Script

The experiment script will execute /root/dsc-code_tx and /root/dsc-code_rx on the source and sink respectively (please note that there are no suffixes), for both (all three) pairs of nodes. Teams must ensure that their images contain /root/dsc-code_tx and /root/dsc-code_rx. No modifications to this execution format is allowed. The sender module draws packets from a packet server, while the sink module delivers packets to the same packet server.

Use the system:exp:dsc-match OEDL script to run the match. The script will start the packet server, start the transmitter and receiver for all teams and release the packet source, wait for 180 seconds and then terminate the match.

For competitive matches, the command line is: 

username@console.grid:~$ omf exec system:exp:dsc-match -- --mode COMP

Competitive match output

  • The output of the competitive match should look similar to the following: 
    username@console.grid:~# omf exec system:exp:dsc-match -- --mode COMP

 INFO NodeHandler: OMF Experiment Controller 5.4 (git 5385228)
 INFO NodeHandler: Slice ID: default_slice (default)
 INFO NodeHandler: Experiment ID: default_slice-2013-07-19t00.06.04.002-04.00
 INFO NodeHandler: Message authentication is disabled
 INFO Experiment: load system:exp:stdlib
 INFO property.resetDelay: resetDelay = 220 (Fixnum)
 INFO property.resetTries: resetTries = 1 (Fixnum)
 INFO Experiment: load system:exp:eventlib
 INFO Experiment: load system:exp:winlib
 INFO Experiment: load dsc-challenge.rb
 INFO property.freq: freq = 600000000 (Fixnum)
 INFO property.server: server = "idb2" (String)
 INFO property.port: port = 5123 (Fixnum)
 INFO property.runtime: runtime = 180 (Fixnum)
 INFO property.mode: mode = "COMP" (String)
 INFO property.team1: team1 = "dsc-teamA" (String)
 INFO property.team2: team2 = "dsc-teamC" (String)
 INFO property.team3: team3 = "dsc-teamB" (String)
 INFO Topology: Loaded topology 'system:topo:dsc-teamA'.
 INFO Topology: Loaded topology 'system:topo:dsc-teamC'.
 INFO exp: Connecting to packet server
 INFO stdlib: Waiting for nodes (Up/Down/Total): 0/4/4 - (still down: node1-1.grid.orbit-lab.org,node20-19.grid.orbit-lab.org,node2-1.grid.orbit-lab.org) [0 sec.]
 INFO ALL_UP_AND_INSTALLED: Event triggered. Starting the associated tasks.
 INFO exp: Preparing packet server
 INFO exp: Request from Experiment Script: Wait for 2s....
 INFO exp: Start rx and tx applications
 INFO exp: Request from Experiment Script: Wait for 10s....
 INFO exp: Sending signal to packet server to start serving packets
 INFO exp: Running the experiment
 INFO exp: Request from Experiment Script: Wait for 180s....
 INFO exp: Getting the score
 INFO exp: Team 1: Sent 12817440 Received 12578400 in 179997 ms
 INFO exp: Team 2: Sent 14400000 Received 10418400 in 179997 ms
 INFO exp: Stopping the applications
 INFO exp: Request from Experiment Script: Wait for 2s....
 INFO NodeHandler: 
 INFO NodeHandler: Shutting down experiment, please wait...
 INFO NodeHandler: 
 INFO run: Experiment default_slice-2013-07-19t00.06.04.002-04.00 finished after 3:30

username@console.grid:~#

For cooperative matches, the command line is: 

username@console.grid:~$ omf exec system:exp:dsc-match -- --mode COOP

Cooperative match output

  • The output of the cooperative match should look similar to the following: 
    username@console.grid:~# omf exec system:exp:dsc-match -- --mode COOP

 INFO NodeHandler: OMF Experiment Controller 5.4 (git 5385228)
 INFO NodeHandler: Slice ID: default_slice (default)
 INFO NodeHandler: Experiment ID: default_slice-2013-07-19t00.15.31.080-04.00
 INFO NodeHandler: Message authentication is disabled
 INFO Experiment: load system:exp:stdlib
 INFO property.resetDelay: resetDelay = 220 (Fixnum)
 INFO property.resetTries: resetTries = 1 (Fixnum)
 INFO Experiment: load system:exp:eventlib
 INFO Experiment: load system:exp:winlib
 INFO Experiment: load dsc-challenge.rb
 INFO property.freq: freq = 600000000 (Fixnum)
 INFO property.server: server = "idb2" (String)
 INFO property.port: port = 5123 (Fixnum)
 INFO property.runtime: runtime = 180 (Fixnum)
 INFO property.mode: mode = "COOP" (String)
 INFO property.team1: team1 = "dsc-teamA" (String)
 INFO property.team2: team2 = "dsc-teamC" (String)
 INFO property.team3: team3 = "dsc-teamB" (String)
 INFO Topology: Loaded topology 'system:topo:dsc-teamA'.
 INFO Topology: Loaded topology 'system:topo:dsc-teamC'.
 INFO Topology: Loaded topology 'system:topo:dsc-teamB'.
 INFO exp: Connecting to packet server
 INFO stdlib: Waiting for nodes (Up/Down/Total): 3/3/6 - (still down: node1-2.grid.orbit-lab.org,node1-1.grid.orbit-lab.org,node20-19.grid.orbit-lab.org) [0 sec.]
 INFO ALL_UP_AND_INSTALLED: Event triggered. Starting the associated tasks.
 INFO exp: Preparing packet server
 INFO exp: Request from Experiment Script: Wait for 2s....
 INFO exp: Start rx and tx applications
 INFO exp: Request from Experiment Script: Wait for 10s....
 INFO exp: Sending signal to packet server to start serving packets
 INFO exp: Running the experiment
 INFO exp: Request from Experiment Script: Wait for 180s....
 INFO exp: Getting the score
 INFO exp: Team 1: Sent 12817440 Received 9578880 in 179998 ms
 INFO exp: Team 2: Sent 14400000 Received 10342080 in 179998 ms
 INFO exp: Team 3: Sent 5796000 Received 1241280 in 179998 ms
 INFO exp: Stopping the applications
 INFO exp: Request from Experiment Script: Wait for 2s....
 INFO NodeHandler: 
 INFO NodeHandler: Shutting down experiment, please wait...
 INFO NodeHandler: 
 INFO run: Experiment default_slice-2013-07-19t00.15.31.080-04.00 finished after 3:32

username@console.grid:~#

After each run, the team will receive notification email ONLY if at least one packet was successful received by the packet server. Team image will be considered for the wildcard evaluation if it was successfully verified (i.e. notification email was received) for both competitive and cooperative matches and if image filename and MD5 checksum matches the ones in the email.

Competitive match notification email

Subject: DSC Match Result

Please find the scoring report for the competitive match created at Fri Jul 19 00:16:09 -0400 2013:

Team C:  Exp. Run Time = 180000, RX bytes = 10418400, TX bytes = 14400000, Sucess Rate = 72.35 
 Image: dsc-wbot2.ndz (Owner: root, Created: Thu Jul 18 12:18:55 -0400 2013, Md5: 5ed3d3b94535d56b6964ad0e58a4e1db )
 Receiver = node1-1.grid.orbit-lab.org, Imaged by: seskar, Imaged on: 2013-07-18 at  22:25:36

Team A:  Exp. Run Time = 180000, RX bytes = 12578400, TX bytes = 12817440, Sucess Rate = 98.135 
 Image: dsc-wbot1.ndz (Owner: root, Created: Thu Jul 18 12:18:35 -0400 2013, Md5: d783eb15b8e12fb95cd03985fd75c1a6 )
 Receiver = node2-1.grid.orbit-lab.org, Imaged by: seskar, Imaged on: 2013-07-18 at  22:21:04

Cooperative match notification email

Subject: DSC Match Result

Please find the scoring report for the cooperative match created at Fri Jul 19 00:23:09 -0400 2013:

Team B:  Exp. Run Time = 180000, RX bytes = 1241280, TX bytes = 5796000, Sucess Rate = 21.4161 
 Image: dsc-wbot3.ndz (Owner: root, Created: Wed Jul 17 21:26:22 -0400 2013, Md5: 209dad194473954803e2b6ed1b2ea8a6 )
 Receiver = node1-2.grid.orbit-lab.org, Imaged by: seskar, Imaged on: 2013-07-18 at  22:37:27

Team C:  Exp. Run Time = 180000, RX bytes = 10342080, TX bytes = 14400000, Sucess Rate = 71.82 
 Image: dsc-wbot2.ndz (Owner: root, Created: Thu Jul 18 12:18:55 -0400 2013, Md5: 5ed3d3b94535d56b6964ad0e58a4e1db )
 Receiver = node1-1.grid.orbit-lab.org, Imaged by: seskar, Imaged on: 2013-07-18 at  22:25:36

Team A:  Exp. Run Time = 180000, RX bytes = 9578880, TX bytes = 12817440, Sucess Rate = 74.7332 
 Image: dsc-wbot1.ndz (Owner: root, Created: Thu Jul 18 12:18:35 -0400 2013, Md5: d783eb15b8e12fb95cd03985fd75c1a6 )
 Receiver = node2-1.grid.orbit-lab.org, Imaged by: seskar, Imaged on: 2013-07-18 at  22:21:04

Turn the nodes off 

username@console.grid:~$ omf tell -a offh -t system:topo:all

Running Matches on Sandboxes

In order to facilitate code development, all three sanboxes (sb2,sb3 and sb7) are haveg simplified version of the OEDL script and packet server. Please do note that, since they have only one pair of nodes, these environments are really not capable of generating interference, but are rather meant for code testing. Appropriately, the actual match script is named dsc-test and the sequence of commands to execute it is somewhat simplified and consist of:

  • Loading the image: 
      omf load -i team-image-name.ndz
  • Turning the nodes on: 
      omf tell -a on
  • Executing the "match"": 
      omf exec system:exp:dsc-test -- --mode COMP
    or 
      omf exec system:exp:dsc-test -- --mode COOP
  • The output of the 30 second matches should look similar to the following:

Competitive match output on the sanbox

console.sb2:~# omf exec system:exp:dsc-test -- --mode COMP --runtime 30

 INFO NodeHandler: OMF Experiment Controller 5.4 (git 5385228)
 INFO NodeHandler: Slice ID: default_slice (default)
 INFO NodeHandler: Experiment ID: default_slice-2013-08-12t20.55.32.533-04.00
 INFO NodeHandler: Message authentication is disabled
 INFO Experiment: load system:exp:stdlib
 INFO property.resetDelay: resetDelay = 230 (Fixnum)
 INFO property.resetTries: resetTries = 1 (Fixnum)
 INFO Experiment: load system:exp:eventlib
 INFO Experiment: load system:exp:winlib
 INFO Experiment: load system:exp:dsc-test
 INFO property.freq: freq = 650000000 (Fixnum)
 INFO property.server: server = "10.12.0.10" (String)
 INFO property.port: port = 5123 (Fixnum)
 INFO property.runtime: runtime = 30 (Fixnum)
 INFO property.mode: mode = "COMP" (String)
 INFO exp: Connecting to packet server
 INFO stdlib: Waiting for nodes (Up/Down/Total): 1/1/2 - (still down: node1-1.sb2.orbit-lab.org) [0 sec.]
 INFO ALL_UP_AND_INSTALLED: Event triggered. Starting the associated tasks.
 INFO exp: Preparing packet server
 INFO exp: Request from Experiment Script: Wait for 2s....
 INFO exp: Start rx and tx applications
 INFO exp: Request from Experiment Script: Wait for 5s....
 INFO exp: Running the experiment
 INFO exp: Request from Experiment Script: Wait for 30s....
 INFO exp: Getting the score
 INFO exp: Sent 2161440 Received 2131200 in 30 ms
 INFO exp: Stopping the applications
 INFO exp: Request from Experiment Script: Wait for 2s....
 INFO NodeHandler: 
 INFO NodeHandler: Shutting down experiment, please wait...
 INFO NodeHandler: 
 INFO run: Experiment default_slice-2013-08-12t20.55.32.533-04.00 finished after 0:52

Cooperative match output on the sandbox

console.sb2:~# omf exec system:exp:dsc-test -- --mode COOP --runtime 30

 INFO NodeHandler: OMF Experiment Controller 5.4 (git 5385228)
 INFO NodeHandler: Slice ID: default_slice (default)
 INFO NodeHandler: Experiment ID: default_slice-2013-08-12t21.00.56.411-04.00
 INFO NodeHandler: Message authentication is disabled
 INFO Experiment: load system:exp:stdlib
 INFO property.resetDelay: resetDelay = 230 (Fixnum)
 INFO property.resetTries: resetTries = 1 (Fixnum)
 INFO Experiment: load system:exp:eventlib
 INFO Experiment: load system:exp:winlib
 INFO Experiment: load system:exp:dsc-test
 INFO property.freq: freq = 650000000 (Fixnum)
 INFO property.server: server = "10.12.0.10" (String)
 INFO property.port: port = 5123 (Fixnum)
 INFO property.runtime: runtime = 30 (Fixnum)
 INFO property.mode: mode = "COOP" (String)
 INFO exp: Connecting to packet server
 INFO ALL_UP_AND_INSTALLED: Event triggered. Starting the associated tasks.
 INFO exp: Preparing packet server
 INFO exp: Request from Experiment Script: Wait for 2s....
 INFO exp: Start rx and tx applications
 INFO exp: Request from Experiment Script: Wait for 5s....
 INFO exp: Running the experiment
 INFO exp: Request from Experiment Script: Wait for 30s....
 INFO exp: Getting the score
 INFO exp: Sent 2161440 Received 1990080 in 30 ms
 INFO exp: Stopping the applications
 INFO exp: Request from Experiment Script: Wait for 2s....
 INFO NodeHandler: 
 INFO NodeHandler: Shutting down experiment, please wait...
 INFO NodeHandler: 
 INFO run: Experiment default_slice-2013-08-12t21.00.56.411-04.00 finished after 0:52

Please note that the sandboxes use different frequencies in order to minimize the interference, are not going to produce the scoring email (the score is printed on the screen only) and that your code must support the scoring server hostname option (since the scoring server is running on respective consoles of each of the sandboxes).

NOTES:

Note: See TracWiki for help on using the wiki.