GT 4.2.1 GRAM4 : System Administrator's Guide

In order to use GRAM4, the container must be started with Transport Level security. The "-nosec" option should *not* be used with globus-start-container.

GRAM4 requires that the sudo command is installed and functioning on the service host where GRAM4 software will execute.

Authorization rules will need to be added to the sudoers file to allow the GRAM4 service account to execute (without a password) the scheduler adapter in the accounts of authorized GRAM users. For sudo configuration details, see the Configuring section.

Platform Note: On AIX, sudo is not installed by default, but it is available as source and rpm here: AIX 5L Toolbox for Linux Applications

GRAM4 depends on a local mechanism for starting and controlling jobs. Included in the GRAM4 software is a Fork scheduler, which requires no special software installed to execute jobs on the local host. However, to enable GRAM4 to execute and manage jobs to a batch scheduler, the scheduler software must be installed and configured prior to configuring GRAM4.

GRAM4 depends on scheduler adapters to translate the GRAM4 job description document into commands understood by the local scheduler, as well as monitor the jobs.

Scheduler adapters included in the GT 4.2.1 release are: PBS, Condor, LSF

Third party schedulers adapters available for GT 4.2.1 are: Sun Grid Engine

For configuration details, see "Configuring scheduler adapters" in the Configuring section.

Though staging directives are processed by RFT (see next section), RFT uses GridFTP servers underneath to do the actual data movement. As a result, there must be at least one GridFTP server that shares a file system with the execution nodes. There is no special process to get staged files onto the execution node before the job executable is run. See the "Non-default GridFTP server" section of Configuring GRAM4 for details on how to configure GRAM4 for your GridFTP servers used in your execution environment.

GRAM4 depends on RFT to perform file staging and cleanup directives in a job description. For configuration details, see System Administrator's Guide. Important: Jobs requesting these functions will fail if RFT is not properly setup.

When the credentials of the service account and the job submitter are different (multi user mode), then GRAM will prepend a call to sudo to the local adapter callout command. Important: If sudo is not configured properly, the command and thus job will fail.

As root, here are the two lines to add to the /etc/sudoers file for each GLOBUS_LOCATION installation, where /opt/globus/GT4.2.1 should be replaced with the GLOBUS LOCATION for your installation:

# Globus GRAM entries
globus  ALL=(username1,username2) 
NOPASSWD: /opt/globus/GT4.2.1/libexec/globus-gridmap-and-execute \
    -g /etc/grid-security/grid-mapfile \
    /opt/globus/GT4.2.1/libexec/globus-job-manager-script.pl *
globus  ALL=(username1,username2) 
NOPASSWD: /opt/globus/GT4.2.1/libexec/globus-gridmap-and-execute \
    -g /etc/grid-security/grid-mapfile \
    /opt/globus/GT4.2.1/libexec/globus-gram-local-proxy-tool *

The globus-gridmap-and-execute program is used to ensure that GRAM only runs programs under accounts that are in the grid-mapfile. In the sudo configuration, it is the first program called. It looks up the account in the grid-mapfile and then runs the requested command. It is redundant if sudo is properly locked down. This tool could be replaced with your own authorization program.

The GRAM4 scheduler adapters included in the release tarball are: PBS, Condor and LSF. To install, follow these steps (shown for pbs):

% configure --prefix=$GLOBUS_LOCATION --enable-wsgram-pbs ...
% make
% make install

Using PBS as the example, make sure the scheduler commands are in your path (qsub, qstat, pbsnodes).

For PBS, another setup step is required to configure the remote shell for rsh access:

% cd $GLOBUS_LOCATION/setup/globus
% ./setup-globus-job-manager-pbs --remote-shell=rsh

The last thing is to define the GRAM and GridFTP file system mapping for PBS. A default mapping in this file is created to allow simple jobs to run. However, the actual file system mappings for your compute resource should be entered to ensure:

Done! You have added the PBS scheduler adapters to your GT installation.

To run the container using just a user proxy, instead of host creds, edit the $GLOBUS_LOCATION/etc/globus_wsrf_core/global_security_descriptor.xml file, and either comment out the credentials section...

<?xml version="1.0" encoding="UTF-8"?>
<securityConfig xmlns="http://www.globus.org">
<!--
<credential>
<key-file value="/etc/grid-security/containerkey.pem"/>
<cert-file value="/etc/grid-security/containercert.pem"/>
<credential>
-->
<gridmap value="/etc/grid-security/grid-mapfile"/>
<securityConfig>

or replace the credentials section with a proxy file location...

<?xml version="1.0" encoding="UTF-8"?>
<securityConfig xmlns="http://www.globus.org">
<proxy-file value="<PATH TO PROXY FILE>"/>
<gridmap value="/etc/grid-security/grid-mapfile"/>
<securityConfig>

Running in personal mode (user proxy), another GRAM configuration setting is required. For GRAM to authorize the RFT service when performing staging functions, it needs to know the subject DN for verification. Here are the steps:

% cd $GLOBUS_LOCATION/setup/globus
% ./setup-gram-service-common --staging-subject=
"/DC=org/DC=doegrids/OU=People/CN=Stuart Martin 564720"

You can get your subject DN by running this command:

% grid-cert-info -subject
            

By default, the GridFTP server is assumed to run as root on localhost:2811. If this is not true for your site then change it by editing the GridFTP host and/or port in the GRAM and GridFTP file system mapping config file: $GLOBUS_LOCATION/etc/globus_wsrf_gram/globus_gram_fs_map_config.xml.

By default, the globus services will assume 8443 is the port the Globus container is using. However the container can be run under a non-standard port, for example:

% globus-start-container -p 4321

If you wish to specify a non-standard gridmap file in a multi-user installation, two basic configurations need to be changed:

Example: global_security_descriptor.xml

...
<gridmap value="/opt/grid-mapfile"/>
...

sudoers

...
# Globus GRAM entries
globus  ALL=(username1,username2) 
NOPASSWD: /opt/globus/GT4.2.1/libexec/globus-gridmap-and-execute \
    -g /opt/grid-mapfile \
    /opt/globus/GT4.2.1/libexec/globus-job-manager-script.pl *
globus  ALL=(username1,username2) 
NOPASSWD: /opt/globus/GT4.2.1/libexec/globus-gridmap-and-execute \
    -g /opt/grid-mapfile \
    /opt/globus/GT4.2.1/libexec/globus-gram-local-proxy-tool *...

RFT is used by GRAM to stage files in and out of the job execution environment. In the default configuration, RFT is hosted in the same container as GRAM and is assumed to have the same service path and standard service names. This need not be the case. For example, the most likely alternative scenario is that RFT would be hosted seperately in a container on a different machine. In any case, both the RFT and the Delegation Service endpoints need to be adjustable to allow this flexibility. The following options can be passed to the setup-gram-service-common script to affect these settings:

--staging-protocol=<protocol> \
--staging-host=<host> \
--staging-port=<port> \
--staging-service-path=<RFT and Delegation factory service path> \
--staging-factory-name=<RFT factory service name> \
--staging-delegation-factory-name=<name of Delegation factory service used by RFT>

For example:

    % setup-gram-service-common \
    --staging-protocol=https
    --staging-host=machine.domain.net
    --staging-delegation-factory-name=machine.domain.net
        

will internally cause the GRAM service code to construct the following EPR addresses.

http://machine.domain.net:8444/wsrf/services/ReliableFileTransferFactoryService
http://machine.fakedomain.net:8444/wsrf/services/DelegationFactoryService
        

	Important
	Jobs submitted by Condor-G will fail if RFT is not located in the same container as GRAM4, because Condor-G assumes that RFT is located in the same container like GRAM4.

Note that, by default, two authorization checks are done during an invocation of WS-GRAM:

The second gridmap check can be avoided by adding a system property to the environment variable GLOBUS_OPTIONS before starting the container:

export GLOBUS_OPTIONS="$GLOBUS_OPTIONS -Dorg.globus.exec.disablegge=true"

This however does not mean, that no authorization check at all is done. The container still checks if the client is authorized as defined in $GLOBUS_LOCATION/etc/globus_wsrf_core/global_security_descriptor.xml but there's no further authorization check when calling the Perl modules. It's up to the GT4 container administrator to decide whether or not to have that additional authorization check. Note that a change in the sudo configuration is required in that case because globus-gridmap-and-execute will not be executed. /opt/globus/GT4.2.1 should be replaced with the value of $GLOBUS_LOCATION for your installation:

# Globus GRAM entries
globus  ALL=(username1,username2) 
NOPASSWD: /opt/globus/GT4.2.1/libexec/globus-job-manager-script.pl *
globus  ALL=(username1,username2) 
NOPASSWD: /opt/globus/GT4.2.1/libexec/globus-gram-local-proxy-tool *

The configuration of WSRF resources and application-level service configuration not related to service deployment is contained in JNDI files. The JNDI-based GRAM configuration is of two kinds:

% ls etc | grep globus_wsrf_gram_

globus_wsrf_gram_Fork
globus_wsrf_gram_LSF
globus_wsrf_gram_Multi

<service name="ManagedJobFactoryService">
    <!-- LRM configuration:  Fork -->
    <resource
        name="ForkResourceConfiguration"
        type="org.globus.exec.service.factory.FactoryResourceConfiguration">
        <resourceParams>
            [...]
            <parameter>
                <name>
                    localResourceManagerName
                </name>
                <value>
                    Fork
                </value>
            </parameter>           
            <!-- Site-specific scratchDir
                    Default: ${GLOBUS_USER_HOME}/.globus/scratch
            <parameter>
                <name>
                    scratchDirectory
                </name>
                <value>
                    ${GLOBUS_USER_HOME}.globus/scratch
                </value>
            </parameter>           
            -->
        </resourceParams>
    </resource>
</service>

The file $GLOBUS_LOCATION/etc/globus_wsrf_gram/managed-job-factory-security-config.xml contains the Core security configuration for the GRAM ManagedJobFactory service:

The file $GLOBUS_LOCATION/etc/globus_wsrf_gram/managed-job-security-config.xml contains the Core security configuration for the GRAM job resources:

Note: GRAM does not override the container security credentials defined in $GLOBUS_LOCATION/etc/globus_wsrf_core/global_security_descriptor.xml. These are the credentials used to authenticate all service requests.

The file $GLOBUS_LOCATION/etc/globus_wsrf_gram/server-config.wsdd contains information necessary to deploy and instantiate the GRAM services in the Globus container.

Three GRAM services are deployed:

Each service deployment information contains the name of the Java service implementation class, the path to the WSDL service file, the name of the operation providers that the service reuses for its implementation of WSDL-defined operations, etc. More information about the service deployment configuration information can be found in Configuring Java WS Core.

In addition to the service configuration described above, there are scheduler-specific configuration files for the Scheduler Event Generator modules. These files consist of name=value pairs separated by newlines. These files are:

With a default GT 4.2.1 installation, the GRAM4 service is automatically registered with the default WS MDS Index Service running in the same container for monitoring and discovery purposes.

This is how auto-registration is configured:

There is a jndi resource defined in $GLOBUS_LOCATION/etc/globus_wsrf_gram/jndi-config.xml as follows :

 <resource name="mdsConfiguration"
    type="org.globus.wsrf.impl.servicegroup.client.MDSConfiguration">
    <resourceParams>
    <parameter> 
    <name>reg</name>
    <value>true</value>
    </parameter>
    <parameter> 
    <name>factory</name>
    <value>org.globus.wsrf.jndi.BeanFactory</value>
    </parameter>
    </resourceParams>
</resource>

To configure the automatic registration of GRAM4 to the default WS MDS Index Service, change the value of the parameter <reg> as follows:

<Content xsi:type="agg:AggregatorContent"
    xmlns:agg="http://mds.globus.org/aggregator/types">        
    <agg:AggregatorConfig xsi:type="agg:AggregatorConfig">
        <agg:GetResourcePropertyPollType
            xmlns:glue="http://mds.globus.org/glue/ce/1.1">
            <!-- Specifies that the index should refresh information
                    every 60000 milliseconds (once per minute) -->
            <agg:PollIntervalMillis>60000</agg:PollIntervalMillis>
            <!-- specifies the resource property that should be
                    aggregated, which in this case is the GLUE cluster
                    and scheduler information RP -->
            <agg:ResourcePropertyName>glue:GLUECE</agg:ResourcePropertyName>  
        </agg:GetResourcePropertyPollType>
    </agg:AggregatorConfig> 
    <agg:AggregatorData/>
</Content>

If a third party needs to register an GRAM4 service manually, see Registering with mds-servicegroup-add in the WS MDS Aggregator Framework documentation.

The GT4 container can run out of memory with large number of jobs. To avoid this:

We recommend the following thread settings as part of the global configuration in $GLOBUS_LOCATION/etc/globus_wsrf_core/server-config.wsdd:

<globalConfiguration:
    ...
    <parameter name="containerThreads" value="20"/>
    <parameter name="containerThreadsMax" value="40"/>
    <parameter name="containerThreadsHighWaterMark" value="10"/>
    ...
</globalConfiguration>

For more information, see global configurations under Java WS Core.

<resource name="auditConfiguration" type="org.globus.exec.service.exec.utils.audit.AuditConfiguration">
    <resourceParams>
        ....
        <parameter>
            <name>enableAuditLogging</name>
            <value>false</value>
        </parameter>
        <parameter>
            <name>auditVersion</name>
            <value>1</value>
        </parameter>
        <parameter>
            <name>fallbackStorageDirectory</name>
            <value>/opt/gt421/share/globus_wsrf_gram/</value>
        </parameter>
        <parameter>
            <name>dbUploadRetryInterval</name>
            <value>300</value>
        </parameter>
    </resourceParams>
</resource>

<resource name="auditDatabase" type="javax.sql.DataSource">
    <resourceParams>
        ....
        <parameter>
            <name>driverClassName</name>
            <value>driver class name</value>
        </parameter>
        <parameter>
            <name>url</name>
            <value>db connection url</value>
        </parameter>
        <parameter>
            <name>username</name>
            <value>user to access the database</value>
        </parameter>
        <parameter>
            <name>password</name>
            <value>password to access the database</value>
        </parameter>
    </resourceParams>
 </resource>

We support 3 database systems: MySQL, PostgreSQL, Derby. The following table gives an overview which values must be used for the parameters url and driverClassName in the above JNDI configuration for the various db systems. Derby is configured as the default DB system.

Audit records are stored in a database which must be set up once.

host:~ feller$ mysql -u root -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 16
Server version: 5.0.37 MySQL Community Server (GPL)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> create database auditDatabase;
Query OK, 1 row affected (0.09 sec)
 
mysql> GRANT ALL ON auditDatabase.* to globus@localhost identified by "foo";
Query OK, 0 rows affected (0.32 sec)

mysql> exit
Bye
host:~ feller$ mysql -u globus -p auditDatabase < ${GLOBUS_LOCATION}/share/globus_wsrf_gram/gram_audit_schema_mysql.sql 
Enter password: 
host:~ feller$

# Connect as postgres admin
create database auditDatabase\g
create user gt4auditload with encrypted password '<password1>'\g
create user gt4auditview with encrypted password '<password2>'\g
\c auditDatabase
\i gram_audit_schema_postgres-8.0.sql
grant insert on gram_audit_table to gt4auditload\g
grant select on gram_audit_table to gt4auditview\g
\q

hostssl   auditDatabase  gt4auditload   <containerhostip> 255.255.255.255 md5
host      auditDatabase  gt4auditload   <containerhostip> 255.255.255.255 md5
hostssl   auditDatabase  gt4auditview   <containerhostip> 255.255.255.255 md5
host      auditDatabase  gt4auditview   <containerhostip> 255.255.255.255 md5

Simple string extension elements are converted into single-element arrays with the name of the unqualified tag name of the extension element as the array's key name in the Perl job description hash. Simple string extension elements can be considered a special case of the string array construct in the next section.

For example, adding the following element to the <extensions> element of the job description as follows:

    <extensions>
        <myparam>yahoo!</myparam>
    </extensions>

will cause the $description->myparam() to return the following value:

    'yahoo!'

String arrays are a simple iteration of the simple string element construct. If you specify more than one simple string element in the job description, these will be assembled into a multi-element array with the unqualified tag name of the extension elements as the array's key name in the Perl job description hash.

For example:

<extensions>
  <myparams>Hello</myparams>
  <myparams>World!</myparams>
</extensions>

will cause the $description->myparams() to return the following value:

[ 'Hello', 'World!' ]

Name/value extension elements can be thought of as string arrays with an XML attribute 'name'. This will cause the creation of a two-dimensional array with the unqualified extension element tag name as the name of the array in the Perl job description hash.

For example:

<extensions>
  <myvars name="pi">3.14159</myvars>
  <myvars name="mole">6.022 x 10^23</myvars>
</extensions>

will cause the $description->myvars() to return the following value:

        [ [ 'pi', '3.14159'], ['mole', '6.022 x 10^23'] ]

Node selection constraints in PBS can be specified in one of the following ways:

The former will be more portable, but the latter will appeal to those familiar with specifying node constraints for PBS jobs.

To specify PBS node selection constraints explicitly, one can simply constuct a single, simple string extension element named nodes with a value that conforms to the #PBS -l nodes=... PBS job description directive. The Globus::GRAM::ExtensionsHandler module will make this available to the PBS adapter script by invoking $description->{nodes}. The updated PBS adapter package checks for this value and will create a directive in the PBS job description using this value.

To use the generic construct for specifying node selection constraints, use the resourceAllocationGroup element:

        <extensions>
        <resourceAllocationGroup>
        <!-- Optionally select hosts by type and number... -->
        <hostType>...</hostType>
        <hostCount>...</hostCount>
        
        <!-- *OR* by host names -->
        
        <hostName>...</hostName>
        <hostName>...</hostName>
        . . .
        
        
        <!-- With a total CPU count for this group... -->
        <cpuCount>...</cpuCount>
        
        <!-- *OR* an explicit number of CPUs per node... -->
        <cpusPerHost>...</cpusPerHost>
        . . .
        
        
        <!-- And a total process count for this group... -->
        <processCount>...</processCount>
        
        <!-- *OR* an explicit number of processes per node... -->
        <processesPerHost>...</processesPerHost>
        </resourceAllocationGroup>
        </extensions>

Extension elements specified according to the above pseudo-schema will be converted to an appropriate nodes parameter which will be treated as if an explicit nodes extension element were specified.

Multiple resourceAllocationGroup elements may be specified. This will simply append the constraints to the nodes paramater with a '+' separator.

	Note
	You cannot specify both hostType/hostCount and hostName elements. Similarly, one cannot specify both processCount and processesPerHost elements.

Here are some examples of using resourceAllocationGroup:

        <!-- #PBS -l nodes=1:ppn=10 -->
        <!-- 10 processes -->
        <extensions>
        <resourceAllocationGroup>
        <cpuCount>10</cpuCount>
        <processCount>10</processCount>
        </resourceAllocationGroup>
        </extensions>
        
        <!-- #PBS -l nodes=activemural:ppn=10+5:ia64-compute:ppn=2 -->
        <!-- 1 process (process default) -->
        <extensions>
        <resourceAllocationGroup>
        <hostType>activemural</hostType>
        <cpuCount>10</cpuCount>
        </resourceAllocationGroup>
        <resourceAllocationGroup>
        <hostType>ia64-compute</hostType>
        <hostCount>5</hostCount>
        <cpusPerHost>2</cpusPerHost>
        </resourceAllocationGroup>
        </extensions>
        
        <!-- #PBS -l nodes=vis001:ppn=5+vis002:ppn=5+comp014:ppn=2+comp015:ppn=2 -->
        <!-- 15 total processes -->
        <extensions>
        <resourceAllocationGroup>
        <hostName>vis001</hostName>
        <hostName>vis002</hostName>
        <cpuCount>10</cpuCount>
        <processesPerHost>5</processesPerHost>
        </resourceAllocationGroup>
        <resourceAllocationGroup>
        <hostName>comp014</hostName>
        <hostName>comp015</hostName>
        <cpusPerHost>2</cpusPerHost>
        <processCount>5</processCount>
        </resourceAllocationGroup>
        </extensions>

This module logs various things to the log file specified in the logfile extension element. If you place this element at the start of the extensions for which you are creating support, then you can look at the specified log file to get some idea of what the handler is doing. You can add new logging lines by using the $self->log() function. This simply takes a string that gets appended to the log file with a prefix of "<date string> EXTENSIONS HANDLER:".

There are three main subroutines that are used to handle parsing events and process them accordingly:

More handlers can be specified for other specific events when creating the XML::Parser instance in new() (see the XML::Parser documentation for details).

The following list describes what the three main subroutines currently do. Modify the subroutines as necessary to achieve your specific goal.

Each adapter and each extension's purpose is different, so there aren't any specific instructions for modifying the resource manager/scheduler adapter module. It is suggested that you spend some time trying to understand what the adapter does and how before making your changes.

Any new hash entries you created in ExtensionsHandler.pm (see the "Customizing ExtensionsHandler.pm" section above) can be accessed by calling $description->entryname() from the adapter module, where 'entryname' is the name of the entry that was added.

See the construct documentation above for more details on generic constructs that are already supported in ExtensionsHandler.pm. This is often an easier route to implementing your extensions than creating a custom construct.

The following information applies to Java WS Core and those services built on it.

Logging in the Java WS Core is based on the Jakarta Commons Logging API. Commons Logging provides a consistent interface for instrumenting source code while at the same time allowing the user to plug-in a different logging implementation. Currently we use Log4j as a logging implementation. Log4j uses a separate configuration file to configure itself. Please see Log4j documentation for details on the configuration file format.

   #for debug level logging from org.globus.package.FooClass 
   log4j.category.org.globus.package.name.FooClass=DEBUG
   #for warnings from org.some.warn.package
   log4j.category.org.some.warn.package=WARN
   

The specific logger to edit will be log4j.logger.sysadmin in $GLOBUS_LOCATION/container-log4j.properties. There you can configure the following properties:

log4j.appender.infoCategory=org.apache.log4j.RollingFileAppender
   log4j.appender.infoCategory.Threshold=INFO
   log4j.appender.infoCategory.File=var/containerLog
   log4j.appender.infoCategory.MaxFileSize=10MB
   log4j.appender.infoCategory.MaxBackupIndex=2

Above implies the logging file is rolling with each file size limited to 10MB and the logging information is stored in $GLOBUS_LOCATION/var/containerLog.

The sample log file contains many log entries for various scenarios in the Java WS container.