## Setting Up A Load Sensor in Grid Engine

Overview
Grid Engine contains a certain set of load parameters which it tracks automatically. Should it be necessary to track a load value not tracked by Grid Engine, a load sensor can be used. A load sensor is a small script which simply outputs one or more name-value pairs to standard out. A name-value pair consists of a resource name, and its current value.
The example below illustrates how to set up a load sensor to track the amount of /tmp space on each Grid Engine host. Load sensors to monitor any desirable value can be written by using this as a template. Once a load sensor is added, the new resource can be used as a load threshhold, or consumable resource. The steps for adding a load sensor are as follows:
Step 1: Define the resource
Step 2: Configure the resource
Step 3: View/Verify the resource
Step 4: Request the resource

Step 1: Define the resource attributes in the host complex
First, modify the complex called "host" or "global", depending on the type of resource, host-specific or clusterwide. To modify the complex, click on "Complexes Configuration" on the main toolbar of qmon. Select "host" or "global", then click on "Modify". If the global host does not exist, it must be created by clicking on "Add"(the name, global, must be used). In the example below, host-specific resources are added to the "host" complex.

       name   shortcut   type     value  relop requestable consumable default
       ----------------------------------------------------------------------
       tmpfree tmpfree   MEMORY    0       <=    YES         YES        0
       tmptot  tmptot    MEMORY    0       <=    YES         NO         0
       tmpused tmpused   MEMORY    0       >=    NO          NO         0

This says: There is a complex attribute called "tmpfree" with the shortcut "tmpfree" of type memory. The "value" is supplied by the load sensor. It is requestable ("yes"), and it is consumable ("yes"). The "default" should be set to 0.
When using qmon, do not forget to press the "Add" button after entering each line. When all the lines are entered (all are in the table below), press the "Ok" button to close this window.
The complex may be viewed at the command line by running the following:

    % qconf -sc host
       name      shortcut   type     value relop requestable consumable default
       ------------------------------------------------------------------------
       tmpfree   tmpfree    MEMORY   0     <=    YES         YES        0
       tmptot    tmptot     MEMORY   0     <=    YES         NO         0
       tmpused   tmpused    MEMORY   0     >=    NO          NO         0

Step 2: Configure the "global" host in the cluster configuration
In this case, the load sensor should run on all hosts (each host will return a different value). Therefore, the pseudo host "global" needs to be configured. In the case of a floating license, a single host would be configured to run our load sensor (since the value would be the same and all hosts returning this same value would be redundant).
In the main qmon window, click on "Cluster Configuration". Highlight "global", then click on "Modify". On the "General Settings" tab, add the path and name of the load sensor program to the load sensor box. As an example/template, tmpspace.sh is included below. Once OK is pressed, the load sensor will be automatically started on each host. This may take several minutes.
Note: The resource names output by the load sensor must be the same as the names added to the host or global complex. Grid Engine takes the smaller of the two values when determining the current value.

Step 3: View the new global resources
The new global resource can be viewed by running the following:

       % qhost -F tmpfree,tmptot,tmpused

       HOSTNAME ARCH      NPROC LOAD  MEMTOT   MEMUSE   SWAPTO   SWAPUS
       ----------------------------------------------------------------
       BALROG   solaris6  2     1.47  1.0G     974.0M   150.0M   130.0M
           Host Resource(s):      hl:tmpfree=337.744000M
           hl:tmptot=338.808000M
           hl:tmpused=1.014709M

See the qhost(1) man page for more information.

Step 4: Requesting a resource
Include the -l switch on the command line to request a resource:

     % qsub -l tmpfree=100 myjob.sh

This will dispatch the job only to those machines whose tmpfree value is greater than or equal to 100 MB.

Note on Using a Load Sensor for Floating Licenses
In order to track the number of floating licenses used outside of Grid Engine, a load sensor may be used in conjunction with a consumable resource. The lessor of the Consumable Resources or the load sensor value will be used to prevent license oversubscription. The load sensor in this case only needs to run on a single host. To do this, output the string 'global' in place of the machine name in the load sensor.

    # ----------------< tmpspace.sh >-----------------------------
    #!/bin/sh
    # Grid Engine will automatically start/stop this script on exec hosts, if
    # configured properly. See the application note for configuration
    # instructions or contact support@gridware.com
    # fs to check
    FS=/tmp

    if [ "$SGE_ROOT" != "" ]; then
       root_dir=$SGE_ROOT

    # invariant values
    myarch=`$root_dir/util/arch`
    myhost=`$root_dir/utilbin/$myarch/gethostname -name`

    ende=false
    while [ $ende = false ]; do

       # ----------------------------------------
       # wait for an input
       #
       read input
       result=$?
       if [ $result != 0 ]; then
          ende=true
          break
       fi

       if [ "$input" = "quit" ]; then
          ende=true
          break
       fi

       # ----------------------------------------
       # send mark for begin of load report
       # NOTE: for global consumable resources not attached
       # to each machine (ie. floating licenses), the load
       # sensor only needs to be run on one host. In that case,
       # echo the string 'global' instead of '$myhost'.
       echo "begin"

       dfoutput="`df -k $FS | tail -1`"
       tmpfree=`echo $dfoutput | awk '{ print $4}'`
       tmptot=`echo $dfoutput | awk '{ print $2}'`
       tmpused=`echo $dfoutput | awk '{ print $3}'`

       echo "$myhost:tmpfree:${tmpfree}k"
       echo "$myhost:tmptot:${tmptot}k"
       echo "$myhost:tmpused:${tmpused}k"

       echo "end"
    done

    #----------------------< CUT HERE >--------------------------------