Jobs submitted to WS-GRAM can now be described in JSDL (in addition to the original JDD schema). A separete job factory service is deployed by the grid resource provider to allow job submissions for each job description type. Thus, a client must target the appropriate factory endpoint for their job description type.

Job description variables are special strings in a job description that are replaced by the GRAM service with values that the client-side does not a priori know. Job description variables can be used in any path-like string or URL specified in the job description. An example of a variable is ${GLOBUS_USER_HOME}, which represents the path to the HOME directory on the file system where the job is executed. The set of variables is fixed in the gram service implementation. This is different from previous implementations of RSL substitutions in GT2 and GT3, where a user could define a new variable for use inside a job description document. This was done to preserve the simplicity of the job description XML schema (relatively to the GT3.2 RSL schema), which does not require a specialized XML parser to serialize a job description document.

Details of the RSL variables are in job description doc

A submission ID may be used in the GRAM protocol for reliability in the face of message faults or other transient errors in order to ensure that at most one instance of a job is executed, i.e. to prevent accidental duplication of jobs under rare circumstances with client retry on failure. By default, the globusrun-ws program will generate a submission ID (uuid). One can override this behavior by supplying a submission ID as a command line argument.

If a user is unsure whether a job was submitted successfully, he should resubmit using the same ID as was used for the previous attempt.

It is possible to specify in a job description that the job be put on hold when it reaches a chosen state (see GRAM Approach documentation for more information about the executable job state machine, and see the job description XML schema documentation for information about how to specify a held state). This is useful for instance when a GRAM client wishes to directly access output files written by the job (as opposed to waiting for the stage-out step to transfer files from the job host). The client would request that the file cleanup process be held until released, giving the client an opportunity to fetch all remaining/buffered data after the job completes but before the output files are deleted.

This is used by globusrun-ws in order to ensure client-side streaming of remote files in batch mode.

GRAM4 services implement a WS Rendezvous mechanism to perform synchronization between job processes in a multiprocess job and between subjobs in a multijob. The job application can in fact register binary information, for instance process information or subjob information, and get notified when all the other processes or subjobs have registered their own information. This is for instance useful for parallel jobs which need to rendezvous at a "barrier" before proceeding with computations, in the case when no native application API is available to help do the rendezvous.

In order to generate a valid proxy file, use the grid-proxy-init tool available under $GLOBUS_LOCATION/bin:

    % bin/grid-proxy-init
    Your identity: /O=Grid/OU=GlobusTest/OU=simpleCA.mymachine/OU=mymachine/CN=John Doe
    Enter GRID pass phrase for this identity:
    Creating proxy ................................. Done
    Your proxy is valid until: Tue Oct 26 01:33:42 2004
    

There are three different uses of delegated credentials:

The globusrun-ws client can either delegate these credentials automatically for a particular job, or it can reuse pre-delegated credentials (see next paragraph) through the use of command-line arguments for specifying the credentials' EPR files. Please see the GT 4.1.1 GRAM4 Command-line Reference for details on these command-line arguments.

It is possible to use delegation Delegation Service Command Reference to obtain and refresh delegated credentials in order to use them when submitting jobs to GRAM4. This, for instance, enables the submission of many jobs using a shared set of delegated credentials. This can significantly decrease the number of remote calls for a set of jobs, thus improving performance.

Unfortunately there is no option yet to print the list of local resource managers supported by a given GRAM service installation. Such information must currently be provided out of band to the user. The GRAM name of local resource managers for which GRAM support has been installed can be obtained by looking at the GRAM configuration on the GRAM server-side machine, as explained in "Local resource manager configuration" under Configuring GRAM4.

The GRAM name of the local resource manager can be used with the factory type option of the job submission command-line tool to specify which factory resource to use when submitting a job.

Use the globusrun-ws program to submit a simple job without writing a job description document. Use the -c argument, a job description will be generated assuming the first arg is the executable and the remaining are arguments. For example:

       % globusrun-ws -submit -c /bin/touch touched_it
       Submitting job...Done.
       Job ID: uuid:4a92c06c-b371-11d9-9601-0002a5ad41e5
       Termination time: 04/23/2005 20:58 GMT
       Current job state: Active
       Current job state: CleanUp
       Current job state: Done
       Destroying job...Done.
    

Confirm that the job worked by verifying the file was touched:

       % ls -l ~/touched_it 
       -rw-r--r--  1 smartin globdev 0 Apr 22 15:59 /home/smartin/touched_it

       % date
       Fri Apr 22 15:59:20 CDT 2005
    

Note: you did not tell globusrun-ws where to run your job, so the default of localhost was used.

Use globusrun-ws to submit the same touch job, but this time specify the contact string.

       % globusrun-ws -submit -F https://lucky0.mcs.anl.gov:8443/wsrf/services/ManagedJobFactoryService -c /bin/touch touched_it
       Submitting job...Done.
       Job ID: uuid:3050ad64-b375-11d9-be11-0002a5ad41e5
       Termination time: 04/23/2005 21:26 GMT
       Current job state: Active
       Current job state: CleanUp
       Current job state: Done
       Destroying job...Done.
    

Try the same job to a remote host. Type globusrun-ws -help to learn the details about the contact string.

The specification of a job to submit is to be written by the user in a job description XML file.

Here is an example of a simple job description:

    <job>
        <executable>/bin/echo</executable>
        <argument>this is an example_string </argument>
        <argument>Globus was here</argument>
        <stdout>${GLOBUS_USER_HOME}/stdout</stdout>
        <stderr>${GLOBUS_USER_HOME}/stderr</stderr>
    </job>
    

Tell globusrun-ws to read the job description from a file, using the -f argument:

    % bin/globusrun-ws -submit -f test_super_simple.xml
    Submitting job...Done.
    Job ID: uuid:c51fe35a-4fa3-11d9-9cfc-000874404099
    Termination time: 12/17/2004 20:47 GMT
    Current job state: Active
    Current job state: CleanUp
    Current job state: Done
    Destroying job...Done.
    

Note the usage of the substitution variable ${GLOBUS_USER_HOME} which resolves to the user home directory.

Here is an example with more job description parameters:

    <?xml version="1.0" encoding="UTF-8"?>
    <job>
        <executable>/bin/echo</executable>
        <directory>/tmp</directory>
        <argument>12</argument>
        <argument>abc</argument>
        <argument>34</argument>
        <argument>this is an example_string </argument>
        <argument>Globus was here</argument>
        <environment>
            <name>PI</name>
            <value>3.141</value>
        </environment>
        <stdin>/dev/null</stdin>
        <stdout>stdout</stdout>
        <stderr>stderr</stderr>
        <count>2</count>
    </job>
    

Note that in this example, a <directory> element specifies the current directory for the execution of the command on the execution machine to be /tmp, and the standard output is specified as the relative path stdout. The output is therefore written to /tmp/stdout:

    % cat /tmp/stdout
    12 abc 34 this is an example_string  Globus was here
    

In order to do file staging one must add specific elements to the job description and delegate credentials appropriately (see Section 3.2, “Delegating credentials”). The file transfer directives follow the RFT syntax, which allows only for third-party transfers. Each file transfer must therefore specify a source URL and a destination URL. URLs are specified as GridFTP URLs (for remote files) or as file URLs (for files local to the service--these are converted internally to full GridFTP URLs by the service).

For instance, in the case of staging a file in, the source URL would be a GridFTP URL (for instance gsiftp://job.submitting.host:2811/tmp/mySourceFile ) resolving to a source document accessible on the file system of the job submission machine (for instance /tmp/mySourceFile ). At run-time the Reliable File Transfer service used by the MEJS on the remote machine would reliably fetch the remote file using the GridFTP protocol and write it to the specified local file (for instance file:///${GLOBUS_USER_HOME}/my_transfered_file, which resolves to ~/my_transfered_file). Here is how the stage-in directive would look like:

<fileStageIn>
    <transfer>
        <sourceUrl>gsiftp://job.submitting.host:2811/tmp/mySourceFile</sourceUrl>
        <destinationUrl>file:///${GLOBUS_USER_HOME}/my_transfered_file</destinationUrl>
    </transfer>
</fileStageIn>

Note: additional RFT-defined quality of service requirements can be specified for each transfer. See the RFT documentation for more information.

Here is an example job description with file stage-in and stage-out:

    <job>
        <executable>my_echo</executable>
        <directory>${GLOBUS_USER_HOME}</directory>
        <argument>Hello</argument>
        <argument>World!</argument>
        <stdout>${GLOBUS_USER_HOME}/stdout</stdout>
        <stderr>${GLOBUS_USER_HOME}/stderr</stderr>
        <fileStageIn>
            <transfer>
                <sourceUrl>gsiftp://job.submitting.host:2811/bin/echo</sourceUrl>
                <destinationUrl>file:///${GLOBUS_USER_HOME}/my_echo</destinationUrl>
            </transfer>
        </fileStageIn>
        <fileStageOut>
            <transfer>
                <sourceUrl>file:///${GLOBUS_USER_HOME}/stdout</sourceUrl>
                <destinationUrl>gsiftp://job.submitting.host:2811/tmp/stdout</destinationUrl>
            </transfer>
        </fileStageOut>
        <fileCleanUp>
            <deletion>
                <file>file:///${GLOBUS_USER_HOME}/my_echo</file>
            </deletion>
        </fileCleanUp>
    </job>
    

Note that the job description XML does not need to include a reference to the schema that describes its syntax. As a matter of fact it is possible to omit the namespace in the GRAM job description XML elements as well. The submission of this job to the GRAM services causes the following sequence of actions:

Basic support is provided for specifying custom extensions to the job description. There are plans to improve the usability of this feature, but at this time it involves a bit of work.

Specifying the actual custom elements in the job description is trivial. Simply add any elements that you need between the beginning and ending extensions tags at the bottom of the job description as in the following basic example:

    <job>
        <executable>/home/user1/myapp</executable>
        <extensions>
            <mySillyData>
                <florgsplat>on</florgsplat>
                <tumblebuffel>off</tumblebuffel>
                <headontight>no</headontight>
            </mySillyData>
        </extensions>
    </job>
    

To handle this data, you will have to alter the appropriate perl scheduler script (i.e. fork.pm for the Fork scheduler, etc...) to parse the data returned from the $description->extensions() sub.

To allow for customization of values, such as paths, on a per-job basis; a job description substitution variable named "GLOBUS_JOB_ID" can be used.

For example:

    <job>
        <executable>/bin/date<executable>
        <stdout>/tmp/stdout.${GLOBUS_JOB_ID}<stdout>
        <stderr>/tmp/stderr.${GLOBUS_JOB_ID}<stderr>
        <fileStageOut>
            <transfer>
                <sourceUrl>file:///tmp/stdout.${GLOBUS_JOB_ID}<sourceUrl>
             <destinationUrl>gsiftp://mymachine.mydomain.com/out.${GLOBUS_JOB_ID}<destinationUrl>
            <transfer>
        <fileStageOut>
    <job>
    

The job description XML schema allows for specification of a multijob i.e. a job that is itself composed of several executable jobs, which we will refer to as subjobs (note: subjobs cannot be multijobs, so the structure is not recursive). This is useful for instance in order to bundle a group of jobs together and submit them as a whole to a remote GRAM installation.

Note that no relationship can be specified between the subjobs of a multijob. The subjobs are submitted to job factory services in their order of appearance in the multijob description.

Within a multijob description, each subjob description must come along with an endpoint for the factory to submit the subjob to. This enables the at-once submission of several jobs to different hosts. The factory to which the multijob is submitted acts as an intermediary tier between the client and the eventual executable job factories.

Here is an example of a multijob description:

<?xml version="1.0" encoding="UTF-8"?>
<multiJob xmlns:gram="http://www.globus.org/namespaces/2004/10/gram/job" 
     xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/03/addressing">
    <factoryEndpoint>
       <wsa:Address>
          https://localhost:8443/wsrf/services/ManagedJobFactoryService
      </wsa:Address>
      <wsa:ReferenceProperties>
          <gram:ResourceID>Multi</gram:ResourceID>
      </wsa:ReferenceProperties>
    </factoryEndpoint>
    <directory>${GLOBUS_LOCATION}</directory>
    <count>1</count>

    <job>
       <factoryEndpoint>
         <wsa:Address>https://localhost:8443/wsrf/services/ManagedJobFactoryService</wsa:Address>
         <wsa:ReferenceProperties>
             <gram:ResourceID>Fork</gram:ResourceID>
         </wsa:ReferenceProperties>
       </factoryEndpoint>
       <executable>/bin/date</executable>
       <stdout>${GLOBUS_USER_HOME}/stdout.p1</stdout>
       <stderr>${GLOBUS_USER_HOME}/stderr.p1</stderr>
       <count>2</count>
    </job>

    <job>
       <factoryEndpoint>
         <wsa:Address>https://localhost:8443/wsrf/services/ManagedJobFactoryService</wsa:Address>
            <wsa:ReferenceProperties>
               <gram:ResourceID>Fork</gram:ResourceID>
            </wsa:ReferenceProperties>
       </factoryEndpoint>
       <executable>/bin/echo</executable>
       <argument>Hello World!</argument>        
       <stdout>${GLOBUS_USER_HOME}/stdout.p2</stdout>
       <stderr>${GLOBUS_USER_HOME}/stderr.p2</stderr>
       <count>1</count>
    </job>

</multiJob>

Notes:

In order to submit a multijob description, use a job submission GT 4.1.1 GRAM4 Command-line Reference and specify the Managed Job Factory resource to be Multi. For instance, submitting the multijob description above using globusrun-ws, we obtain:

    % bin/globusrun-ws -submit -f test_multi.xml
    Delegating user credentials...Done.
    Submitting job...Done.
    Job ID: uuid:bd9cd634-4fc0-11d9-9ee1-000874404099
    Termination time: 12/18/2004 00:15 GMT
    Current job state: Active
    Current job state: CleanUp
    Current job state: Done
    Destroying job...Done.
    Cleaning up any delegated credentials...Done.
    

A multijob resource is created by the factory and exposes a set of WSRF resource properties different than the resource properties of an executable job. The state machine of a multijob is also different since the multijob represents the overall execution of all the executable jobs it is composed of.

N/A

N/A

The specification of a job to submit is to be written by the user in a JSDL compliant job description file. Here is an example of a simple job description:

<?xml version="1.0" encoding="UTF-8"?>
<jsdl:JobDefinition xmlns:jsdl="http://schemas.ggf.org/jsdl/2005/11/jsdl" 
                    xmlns:jsdl-posix="http://schemas.ggf.org/jsdl/2005/11/jsdl-posix">
    <jsdl:JobDescription>
        <jsdl:Application>
            <JobName>Test Job</JobName>
            <Description>Simple Job without staging</Description>
            <JobAnnotation>With Posix application element</JobAnnotation>
            <JobProject>Test project</JobProject>
            <jsdl-posix:POSIXApplication >
                <jsdl-posix:Executable>/bin/echo</jsdl-posix:Executable>
                <jsdl-posix:Argument>hello, world!</jsdl-posix:Argument>
                <jsdl-posix:Output>${GLOBUS_USER_HOME}/stdout</jsdl-posix:Output>
                <jsdl-posix:Error>${GLOBUS_USER_HOME}/stderr</jsdl-posix:Error>
            </jsdl-posix:POSIXApplication>
        </jsdl:Application>
    </jsdl:JobDescription>
</jsdl:JobDefinition>

The above job description could also use HPCProfileApplication instead of PosixApplication. The only thing that needs to be done is to replace the PosixApplication element in the above job description with the following

<jsdl-hpcp:HPCProfileApplication>
    <jsdl-hpcp:Executable>/bin/echo</jsdl-hpcp:Executable>
    <jsdl-hpcp:Argument>hello, world!</jsdl-hpcp:Argument>
    <jsdl-hpcp:Output>${GLOBUS_USER_HOME}/stdout</jsdl-hpcp:Output>
    <jsdl-hpcp:Error>${GLOBUS_USER_HOME}/stderr</jsdl-hpcp:Error>
</jsdl-hpcp:HPCProfileApplication>

Note the usage of the substitution variable ${GLOBUS_USER_HOME} which resolves to the user home directory.

Submit the job with the following command but fill in values for host and port before:

% java -DGLOBUS_LOCATION=${GLOBUS_LOCATION} \
    org.globus.exec.client.GlobusRun \
    -factory https://<host>:<port>/wsrf/services/v4_2/ManagedJobFactoryService \
    -file <jsdl-file>

In order to do file staging one must add specific elements to the job description and delegate credentials appropriately. The file transfer directives follow the JSDL syntax. Each file transfer must therefore specify either a source URL or a target URL, depending on the fact if a file should be staged in before job execution or staged out after job execution. URLs are specified as GridFTP URLs.

For instance, in the case of staging a file in, the source URL would be a GridFTP URL (for instance gsiftp://job.submitting.host:2811/tmp/mySourceFile ) resolving to a source document accessible on the file system of the job submission machine (for instance /tmp/mySourceFile ). At run-time the Reliable File Transfer service used by the MEJS on the remote machine would reliably fetch the remote file using the GridFTP protocol and write it to the specified local file (for instance /tmp/mySourceFile).

Here is how the stage-in directive would look like:

<jsdl:DataStaging>
    <jsdl:FileName>/tmp/my_SourceFile</jsdl:FileName>
    <jsdl:CreationFlag>overwrite</jsdl:CreationFlag>
    <jsdl:Source>
        <jsdl:URI>gsiftp://job.submitting.host:2811/tmp/mySourceFile</jsdl:URI>
    </jsdl:Source>
</jsdl:DataStaging>

Here is an example job description with file stage-in and stage-out:

<?xml version="1.0" encoding="UTF-8"?>
<!-- simple job, file staging in and out -->
<jsdl:JobDefinition xmlns:jsdl="http://schemas.ggf.org/jsdl/2005/11/jsdl" 
                    xmlns:jsdl-rft="http://www.globus.org/gram/2006/12/jsdl-rft"
                    xmlns:jsdl-posix="http://schemas.ggf.org/jsdl/2005/11/jsdl-posix">
    <jsdl:JobDescription>
        <jsdl:Application>
            <JobName>Test Job</JobName>
            <Description>Job with staging in and out</Description>
            <JobAnnotation>With Posix application element</JobAnnotation>
            <JobProject>Test project</JobProject>
            <jsdl-posix:POSIXApplication >
                <jsdl-posix:Executable>/tmp/my_echo</jsdl-posix:Executable>
                <jsdl-posix:Argument>hello, world!</jsdl-posix:Argument>
                <jsdl-posix:Output>/tmp/stdout</jsdl-posix:Output>
                <jsdl-posix:Error>/tmp/stderr</jsdl-posix:Error>
            </jsdl-posix:POSIXApplication>
        </jsdl:Application>

        <jsdl:DataStaging>
            <jsdl:FileName>/tmp/my_echo1</jsdl:FileName>
            <jsdl:CreationFlag>overwrite</jsdl:CreationFlag>
            <jsdl:Source>
                <jsdl:URI>gsiftp://127.0.0.1:2811/bin/echo</jsdl:URI>
            </jsdl:Source>
        </jsdl:DataStaging>

        <jsdl:DataStaging>
            <jsdl:FileName>/tmp/stdout</jsdl:FileName>
            <jsdl:CreationFlag>overwrite</jsdl:CreationFlag>
            <jsdl:Target>
                <jsdl:URI>gsiftp://127.0.0.1:2811/tmp/stdout_staged</jsdl:URI>
            </jsdl:Target>
        </jsdl:DataStaging>

        <jsdl:DataStaging>
            <jsdl:FileName>/tmp/stderr</jsdl:FileName>
            <jsdl:CreationFlag>overwrite</jsdl:CreationFlag>
            <jsdl:Target>
                <jsdl:URI>gsiftp://127.0.0.1:2811/tmp/stderr_staged</jsdl:URI>
            </jsdl:Target>
        </jsdl:DataStaging>
              
    </jsdl:JobDescription>
</jsdl:JobDefinition>

The above job description could also use HPCProfileApplication instead of PosixApplication. The only thing that needs to be done is to replace the PosixApplication element in the above job description with the following

<jsdl-hpcp:HPCProfileApplication>
    <jsdl-hpcp:Executable>/tmp/my_echo</jsdl-hpcp:Executable>
    <jsdl-hpcp:Argument>hello, world!</jsdl-hpcp:Argument>
    <jsdl-hpcp:Output>/tmp/stdout</jsdl-hpcp:Output>
    <jsdl-hpcp:Error>/tmp/stderr</jsdl-hpcp:Error>
</jsdl-hpcp:HPCProfileApplication>

Submit the job with the following command:

% java -DGLOBUS_LOCATION=${GLOBUS_LOCATION} \
    org.globus.exec.client.GlobusRun \
    -factory https://<host>:<port>/wsrf/services/v4_2/ManagedJobFactoryService \
    -deleg limited \
    -file <jsdl-file>

N/A

N/A

N/A

The following are specific supported extensions to the WS-GRAM job description schema. They do not require any modification of the resource manager/scheduler adapter Perl modules.

<multiJob>
    ...
    <job>
        ...
    </job>
    <job>
        ...
    </job>
    <extensions>
        <multiAuthzSubject>/DC=org/DC=doegrids/OU=People/CN=John Doh 123456</multiAuthzSubject>
    </extensions>
</multiJob>
        

<extensions>
    <resourceAllocationGroup>
        <!-- Optionally select hosts by type and number... -->
        <hostType>...</hostType>
        <hostCount>...</hostCount>

        <!-- *OR* by host names -->

        <hostName>...</hostName>
        <hostName>...</hostName>
        . . .


        <!-- With a total CPU count for this group... -->
        <cpuCount>...</cpuCount>

        <!-- *OR* an explicit number of CPUs per node... -->
        <cpusPerNode>...</cpusPerNode>
        . . .


        <!-- And a total process count for this group... -->
        <processCount>...</processCount>

        <!-- *OR* an explicit number of processes per node... -->
        <processesPerNode>...</processesPerNode>
    </resourceAllocationGroup>
</extensions>
        

<!-- #PBS -l nodes=1:ppn=10 -->
<!-- 10 processes -->
<extensions>
    <resourceAllocationGroup>
        <cpuCount>10</cpuCount>
        <processCount>10</processCount>
    </resourceAllocationGroup>
</extensions>

<!-- #PBS -l nodes=activemural:ppn=10+5:ia64-compute:ppn=2 -->
<!-- 1 process (process default) -->
<extensions>
    <resourceAllocationGroup>
        <hostType>activemural</hostType>
        <cpuCount>10</cpuCount>
    </resourceAllocationGroup>
    <resourceAllocationGroup>
        <hostType>ia64-compute</hostType>
        <hostCount>5</hostCount>
        <cpusPerHost>2</cpusPerHost>
    </resourceAllocationGroup>
</extensions>

<!-- #PBS -l nodes=vis001:ppn=5+vis002:ppn=5+comp014:ppn=2+comp015:ppn=2 -->
<!-- 15 total processes -->
<extensions>
    <resourceAllocationGroup>
        <hostName>vis001</hostName>
        <hostName>vis002</hostName>
        <cpuCount>10</cpuCount>
        <processesPerHost>5</processesPerHost>
    </resourceAllocationGroup>
    <resourceAllocationGroup>
        <hostName>comp014</hostName>
        <hostName>comp015</hostName>
        <cpusPerHost>2</cpusPerHost>
        <processCount>5</processCount>
    </resourceAllocationGroup>
</extensions>
        

The following are general constructs that are supported by the ExtensionsHandler.pm Perl module. Although no modifications to ExtensionsHandler.pm are required, you will need to edit the appropriate resource manager/scheduler adapter Perl module as neccessary to affect the submission of jobs to the local resource manager/batch scheduler.

The WS-GRAM job description schema includes a section for extending the job description with custom elements. To make sense of this in the resource manager adapter Perl scripts, a Perl module named Globus::GRAM::ExtensionsHandler is provided to turn these custom elements into paramters that the adapter scripts can understand.

It should be noted that although non-GRAM XML elements only are allowed in the <extensions> element of the job description, the extensions handler makes no distinction based on namespace. Thus, <foo:myparam> and <bar:myparam> will both be treated as just <myparam>.

Familiarity with the adapter scripts is assumed in the following subsections.

<extensions>
    <myparam>yahoo!</myparam>
</extensions>
        

            'yahoo!'
        

<extensions>
    <myparams>Hello</myparams>
    <myparams>World!</myparams>
</extensions>
        

            [ 'Hello', 'World!' ]
        

<extensions>
    <myvars name="pi">3.14159</myvars>
    <myvars name="mole">6.022 x 10^23</myvars>
</extensions>
        

[ [ 'pi', '3.14159'], ['mole', '6.022 x 10^23'] ]
        

See the System Administrator's Guide section on Configuring GRAM4 for information on how to customize the resource manager/scheduler adapter Perl modules