Changes between Version 8 and Version 9 of Internal/Infrastructure/OMF/GridServices/Inventory


Ignore:
Timestamp:
Jun 5, 2008, 5:59:44 PM (16 years ago)
Author:
Joseph F. Miklojcik III
Comment:

—

Legend:

Unmodified
Added
Removed
Modified

  • Internal/Infrastructure/OMF/GridServices/Inventory

    v8 v9  
    88 * It should have an (optional) query qualifier (with interface to CMC) to return only the functional node set.
    99 * It should have ...
    10 
    11 
    12 It is important to consider the Inventory database schema part of the experimental interface.  The advantage of building a web service for Inventory is that it can provide commonly used queries that incorporate data from other OMF web services (namely the CMC, as in "available" in "give me all available intel nodes").  At the same time, it will be easier for experiment scripts to build custom SQL queries that implement complex criteria than to utilize a more abstract web service.  Furthermore, arguably, SQL is RESTful in the first place.
    1310
    1411== Service Configuration File ==
    … …  
    3027}}}
    3128
    32 == Inventory Gathering ==
    33 
    34 Inventory, is the "experiment" that uses a special much larger image, which can have a relatively large number of drivers for a wide variety of devices.  Inventory is used to do the slower more difficult job of generating informative scans of the pci and usb busses.
    35 
    36 As of 10/16/2007, we run the inventory every Wednesday during maintenance.  Plan is to, in addition, run Inventory whenever there is a change in hardware configuration.  There exists a full [source:/Inventory/inventory inventory package] that inserts "all" information into the database.
    37 
    38   The gathering procedure uses:
    39  
    40  * lspci
    41  * lsusb
    42  * dmidecode
    43  
    4429=== Known Device IDs ===
    4530
    46 Devices that are discovered by gathering procedure are uniquely identified by either [http://www.linux-usb.org/usb.ids USB ids] or [http://www.pcidatabase.com/vendors.php?sort=id PCI ids]. The ids for most important devices are listed in the ID table: 
     31Devices that are discovered by gathering procedure are uniquely identified by either [http://www.linux-usb.org/usb.ids USB ids] or [http://www.pcidatabase.com/vendors.php?sort=id PCI ids]. The ids for most important devices are listed below: 
    4732
    4833|| Bus type || Vendor ID || Vendor || Device ID || Description ||
    … …  
    5843|| -"- || 0x8086 (32902) || [http://www.pcidatabase.com/vendor_details.php?id=1302 Intel Corp.] || 0x4223 (16931) || Intel PRO/Wireless 2915ABG Network Connectio ||
    5944|| -"- || 0x168c (5772) || [http://www.pcidatabase.com/vendor_details.php?id=174 Atheros Communications] || 0x0013 (19) || Atheros AR5212, AR5213 802.11a/b/g Wireless Adapter ||
    60 || -"- || 0x1106 (4358) || [http://www.pcidatabase.com/vendor_details.php?id=648 VIA Technology] || 0x3122 (12578) || VT8623 !CastleRock AGP 8X Controller (VGA) ||
    61 || -"- || -"- || -"- || 0x0571 (1393) || Bus Master IDE Controller ||
    62 || -"- || -"- || -"- || 0x3177 (12663) || VT8235 PCI to ISA Bridge ||
    63 || -"- || -"- || -"- || 0x3104 (12548) || VT6202 USB 2.0 Enhanced Host Controller ||
    64 || -"- || -"- || -"- || 0x3038 (12344) || USB&UHCI Controller ||
    65 || -"- || -"- || -"- || 0xb091 (45201) || VT8633 PCI-to-PCI Bridge (AGP) ||
    66 || -"- || -"- || -"- || 0x3123 (12579) || VT8623 CPU to PCI Bridge ||
    67 || USB || 0x3f0 (1008) || Hewlett-Packard  || 0x0024 (36) || KU-0316 Keyboard ||
    6845
    69 
    70 We should look into turning inventory image into PXE image and avoid imaging phase completely!
    7146
    7247== Inventory Database ==
    7348
    74 Inventory database lives on internal1 and consists of 6 tables:
     49The Inventory database is a Mysql database on {{{internal1}}}.  The schema was created to be general and scalable, so that it can support many different kinds of nodes and attached devices, and can hold inventory data for multiple testbeds.  In general names of tables are plural nouns.  When rows in a particular table represent physical objects or concrete ideas, as opposed to relationships or metadata, each row in a table will have a unique id.  When an id is referenced from another table, the name of the referencing column will end in {{{_id}}}.  Although it is not set in stone, the schema for the inventory database should not change significantly without advance notice on {{{orbit-dev}}}.  No data consistency constraints other than automatic ID number generation are implemented in the DBMS, only in the application code.  This keeps things flexible enough so that there are lots of options when you find 400 nodes can't all successfully lock tables and rows in the database at once.
    7550
    76  1. device_kinds
    77  2. device_tags
    78  3. inventories
    79  4. locations
    80  5. motherboards
    81  6. nodes
    82  7. peripherals
    83  8. testbeds
     51It is important to consider the Inventory database schema part of the experimental interface.  The advantage of building a web service for Inventory is that it can provide commonly used queries that incorporate data from other OMF web services (namely the CMC, as in "available" in "give me all available intel nodes").  At the same time, it will be easier for experiment scripts to build custom SQL queries that implement complex criteria than to utilize a more abstract web service.  Furthermore, arguably, SQL is RESTful in the first place.
    8452
    85 === device_kinds table ===
     53The database is used both as the authoritative source of data for the OMF inventory service, and for maintenance purposes.  Most of the data is maintained automatically, but some data (such as chassis ID number) can not be collected automatically and must be entered by hand.
    8654
    87 || Field        || Type        || Null || Key || Default || Extra          ||
    88 || id           || int(11)     || NO   || PRI || NULL    || auto_increment ||
    89 || inventory_id || int(11)     || NO   ||     || NULL    ||                ||
    90 || oui          || varchar(8)  || YES  ||     || NULL    ||                ||
    91 || bus          || varchar(16) || YES  ||     || NULL    ||                ||
    92 || vendor       || int(11)     || NO   ||     || NULL    ||                ||
    93 || device       || int(11)     || NO   ||     || NULL    ||                ||
     55'''These schemas are still changing, so assume the actual database is more authoritative than this document.'''
    9456
     57== Inventory Gathering ==
    9558
    96 === device_tags table ===
     59Inventory gathering is done as any other ORBIT experiment.  It uses a relatively large image, which includes a thorough set of drivers for a wide variety of devices.  A new node undergoes an Enrollment process when it is first deployed, implemented as a small PXE image with just enough of a payload to associate the node location with an IP address and report it.  In contrast, the Inventory process does the slower more complex job of generating informative scans of the pci and usb busses.
    9760
    98 || Field         || Type        || Null || Key || Default || Extra ||
    99 || tag           || varchar(64) || NO   ||     || NULL    ||       ||
    100 || peripheral_id || int(11)     || NO   ||     || NULL    ||       ||
     61As of 10/16/2007, we run the inventory every Wednesday during maintenance.  We may also run it when there are significant hardware changes.
    10162
    102 === motherboards table ===
    103 || Field       || Type        || Null || Key || Default           || Extra          || Description                                   ||
    104 || id          || varchar(64) || NO   || PRI ||                   ||                || UUID of the motherboard                       ||
    105 || node_id     || varchar(64) || YES  || UNI || NULL              ||                || Link to 'id' in nodes table                   ||
    106 || sn   || varchar(16) || NO   || UNI ||                   ||                || manufacturer serial number of the motherboard ||
    107 || hd_sn        || varchar(16) || NO   || UNI ||                   ||                || Hard drive serial number                      ||
    108 || cpu_type         || varchar(X)  || YES  ||     || NULL              ||                || CPU Type                                      ||
    109 || cpu_speed       || int(11)     || YES  ||     || 0                 ||                || CPU speed in MHz                              ||
    110 || memory      || int(11)     || YES  ||     || 0                 ||                || Memory size in MB                             ||
    111 || hd_size        || int(11)     || YES  ||     || 0                 ||                || Hard disk size in bytes                          ||
    112 || updated_on  || timestamp   || NO   ||     || CURRENT_TIMESTAMP ||                ||                                               ||
    113 || updated_by  || varchar(64) || NO   ||     || ||                ||                                               ||
     63The gathering procedure uses {{{lspci}}}, {{{lsusb}}}, {{{dmidecode}}} and {{{sysfs}}}.  The most frustrating and difficult part of maintaining the code is keeping up with changes in {{{/sys}}} over even minor kernel revisions.
    11464
    115 
    116 (NOTE: 'node_id' is NULL when this motherboard is not installed on any node, i.e. new parts that just got in, or stored extra/spare parts)
    117 
    118 We could also move the hard-drive info in a separate table if we allow hard-drive swapping between motherboards.
    119 
    120 === nodes table ===
    121 || Field       || Type        || Null || Key || Default || Extra          || Description ||
    122 || id          || varchar(64) || NO   || PRI ||         ||                || UUID of the node (i.e. the chassis). ||
    123 || chassis_sn  || varchar(16) || NO   || UNI ||         ||                || Manufacturer serial number of the node's chassis ||
    124 || location_id || varchar(64) || YES  || UNI || NULL    ||                || Link to 'id' in 'locations' table ||
    125 || updated_on  || timestamp   || NO   ||     || CURRENT_TIMESTAMP ||                ||                                               ||
    126 || updated_by  || varchar(64) || NO   ||     ||  ||                ||                                               ||
    127 
    128 (NOTE: 'location_id' is NULL when this chassis is not installed at any location, i.e. new parts that just got in, or stored extra/spare parts)
    129 
    130 === locations table ===
    131 
    132 || Field       || Type        || Null || Key || Default           || Extra          || Description ||
    133 || id          || varchar(64) || NO   || PRI ||         ||                || UUID of the location ||
    134 
    135 
    136 || x           || int(11)     || NO   ||     || 0                 ||                || ||
    137 || y           || int(11)     || NO   ||     || 0                 ||                || ||
    138 || z           || int(11)     || NO   ||     || 0                 ||                || ||
    139 || unit        || int(11)     || NO   ||     || 0                 ||                || ||
    140 || testbed_id  || varchar(64) || NO   ||     || 0                 ||                || Link to 'id' in 'testbeds' table ||
    141 || updated_on  || timestamp   || NO   ||     || CURRENT_TIMESTAMP ||                ||                                               ||
    142 || updated_by  || varchar(64) || NO   ||     ||  ||                ||                                               ||
    143 
    144 
    145 === testbeds (resources) table ===
    146 
    147 || Field      || Type        || Null || Key || Default || Extra  || Description ||
    148 || id         || varchar(64)  || NO  || PRI ||         ||        || UUID of the testbed ||
    149 || domain     || varchar(4)  || NO   || UNI ||         ||        || ||
    150 || control_ip || varchar(12) || NO   || UNI ||         ||        || ||
    151 || data_ip    || varchar(12) || NO   || UNI ||         ||        || ||
    152 || cm_ip      || varchar(12) || NO   ||     ||         ||        || ||
    153 || latitude   || int(11)     || NO   ||     || 0       ||        || ||
    154 || longitude  || int(11)     || NO   ||     || 0       ||        || ||
    155 || elevation  || int(11)     || NO   ||     || 0       ||        || ||
    156 || updated_on  || timestamp   || NO   ||     || CURRENT_TIMESTAMP ||                ||                                               ||
    157 || updated_by  || varchar(64) || NO   ||     ||  ||                ||                                               ||
    158 
    159 == Notes ==
    160 
    161 The design goal of this schema is to allow the double use of the Inventory database as:
    162   * a source of information for user experiment scripts
    163   * a 'real' hardware inventory giving operators information on which piece of hardware (chassis, motherboard) is used (or not) in which testbed/location.
    164 
    165 The entries in the ''testbeds'', ''locations'', ''nodes'' tables are manually created and updated by operators, when:
    166   * a new testbed is being deployed
    167   * a new location is added to the testbed (e.g. physical place-holder creation on a sandbox testbed for future addition of a third node)
    168   * a new purchased chassis (i.e. empty node box) is delivered, or mounted to a new location, or switched from a location to another one
    169 
    170 We do not expect these events to happen very often, thus it should be ok to make the operator responsible for creating/updating the related entries. (furthermore he/she could also use some scripts to do this job...)
    171 
    172 The entries in the ''motherboards'' table are also manually created upon delivery of a new purchased motherboard. The only field that needs to be manually filled by the operator is the ''node_id'', which will happen when the operator installs a new motherboard inside a node/chassis. All the other fields are automatically populated by the Inventory process (i.e. the scripts in the inventory package).
    173 
    174 The ''interfaces'' and ''devices'' tables are created and updated as in the previous schema.
    175 
    176 These schemas have a tendency to change, so assume the actual database is more authoritative than this document.
     65The current inventory code can all be found in the image {{{inventory.ndz}}}.  Source is controlled by {{{git}}}.  At the time of this writing there is no functioning {{{git}}} server at WINLAB, so just mail patches to {{{jfm3}}} at {{{winlab.rutgers.edu}}}.
     66