In order to use GRAM4, the container must be started with Transport Level security. The "-nosec" option should *not* be used with globus-start-container.

GRAM4 requires that the sudo command is installed and functioning on the service host where GRAM4 software will execute.

Authorization rules will need to be added to the sudoers file to allow the GRAM4 service account to execute (without a password) the scheduler adapter in the accounts of authorized GRAM users. For sudo configuration details, see the Configuring section.

Platform Note: On AIX, sudo is not installed by default, but it is available as source and rpm here: AIX 5L Toolbox for Linux Applications

GRAM4 depends on a local mechanism for starting and controlling jobs. Included in the GRAM4 software is a Fork scheduler, which requires no special software installed to execute jobs on the local host. However, to enable GRAM4 to execute and manage jobs to a batch scheduler, the scheduler software must be installed and configured prior to configuring GRAM4.

GRAM4 depends on scheduler adapters to translate the GRAM4 job description document into commands understood by the local scheduler, as well as monitor the jobs.

Scheduler adapters included in the GT 4.1.1 release are: PBS, Condor, LSF

Third party schedulers adapters available for GT 4.1.1 are: Sun Grid Engine

For configuration details, see "Configuring scheduler adapters" in the Configuring section.

Though staging directives are processed by RFT (see next section), RFT uses GridFTP servers underneath to do the actual data movement. As a result, there must be at least one GridFTP server that shares a file system with the execution nodes. There is no special process to get staged files onto the execution node before the job executable is run. See the "Non-default GridFTP server" section of Configuring GRAM4 for details on how to configure GRAM4 for your GridFTP servers used in your execution environment.

GRAM4 depends on RFT to perform file staging and cleanup directives in a job description. For configuration details, see GT 4.1.1 Reliable File Transfer (RFT) Service: System Administrator's Guide. Important: Jobs requesting these functions will fail if RFT is not properly setup.

WS-GRAM stores information about job resources in a relational database system to ensure that no information about running jobs is lost in case of a container shutdown and restart. The default database system is Derby; the necessary database is created during installation of the GT. Database schemas for MySQL and PostgreSQL are provided too.

When the credentials of the service account and the job submitter are different (multi user mode), then GRAM will prepend a call to sudo to the local adapter callout command. Important: If sudo is not configured properly, the command and thus job will fail.

As root, here are the two lines to add to the /etc/sudoers file for each GLOBUS_LOCATION installation, where /opt/globus/GT4.1.1 should be replaced with the GLOBUS LOCATION for your installation:

# Globus GRAM entries
                globus  ALL=(username1,username2) 
                NOPASSWD: /opt/globus/GT4.1.1/libexec/globus-gridmap-and-execute 
                -g /etc/grid-security/grid-mapfile
                /opt/globus/GT4.1.1/libexec/globus-job-manager-script.pl *
                globus  ALL=(username1,username2) 
                NOPASSWD: /opt/globus/GT4.1.1/libexec/globus-gridmap-and-execute 
                -g /etc/grid-security/grid-mapfile
                /opt/globus/GT4.1.1/libexec/globus-gram-local-proxy-tool *
            

The globus-gridmap-and-execute program is used to ensure that GRAM only runs programs under accounts that are in the grid-mapfile. In the sudo configuration, it is the first program called. It looks up the account in the grid-mapfile and then runs the requested command. It is redundant if sudo is properly locked down. This tool could be replaced with your own authorization program.

The GRAM4 scheduler adapters included in the release tarball are: PBS, Condor and LSF. To install, follow these steps (shown for pbs):

    % cd $GLOBUS_LOCATION\gt4.1.1-all-source-installer
                
                % make gt4-gram-pbs
                
                % make install
                
            

Using PBS as the example, make sure the scheduler commands are in your path (qsub, qstat, pbsnodes).

For PBS, another setup step is required to configure the remote shell for rsh access:

                
                % cd $GLOBUS_LOCATION/setup/globus
                
                % ./setup-globus-job-manager-pbs --remote-shell=rsh
                
            

The last thing is to define the GRAM and GridFTP file system mapping for PBS (see the below). A default mapping in this file is created to allow simple jobs to run. However, the actual file system mappings for your compute resource should be entered to ensure:

Done! You have added the PBS scheduler adapters to your GT installation.

Note for future GT builds with scheduler adapters: scheduler adapters can be enabled by adding --enable-wsgram-pbs to the configure line when building the entire toolkit.

                % configure --prefix=$GLOBUS_LOCATION --enable-wsgram-pbs ...
                % make
                % make install
            

The database that stores the information about job resources is configured in the JNDI container registry in $GLOBUS_LOCATION/etc/globus_wsrf_gram/jndi-config.xml. By default Derby, which is shipped as part of the GT, is used as DMBS. The necessary tables are created during installation of the GT and no additional configuration must be done.

Aside from Derby, persistence data can also be stored in MySQL or PostgreSQL. For information about how to configure MySQL or PostgreSQL, see the Non-default Configuration section on this page.

In case the Derby database should be cleared or recreated the following can be done:

To run the container using just a user proxy, instead of host creds, edit the $GLOBUS_LOCATION/etc/globus_wsrf_core/global_security_descriptor.xml file, and either comment out the credentials section...

                <?xml version="1.0" encoding="UTF-8"?>
                <securityConfig xmlns="http://www.globus.org">
                <!--
                <credential>
                <key-file value="/etc/grid-security/containerkey.pem"/>
                <cert-file value="/etc/grid-security/containercert.pem"/>
                <credential>
                -->
                <gridmap value="/etc/grid-security/grid-mapfile"/>
                <securityConfig>
            

or replace the credentials section with a proxy file location...

                <?xml version="1.0" encoding="UTF-8"?>
                <securityConfig xmlns="http://www.globus.org">
                <proxy-file value="<PATH TO PROXY FILE>"/>
                <gridmap value="/etc/grid-security/grid-mapfile"/>
                <securityConfig>
            

Running in personal mode (user proxy), another GRAM configuration setting is required. For GRAM to authorize the RFT service when performing staging functions, it needs to know the subject DN for verification. Here are the steps:

                % cd $GLOBUS_LOCATION/setup/globus
                % ./setup-gram-service-common --staging-subject=
                "/DC=org/DC=doegrids/OU=People/CN=Stuart Martin 564720"
            

You can get your subject DN by running this command:

                % grid-cert-info -subject
            

By default, the GridFTP server is assumed to run as root on localhost:2811. If this is not true for your site then change it by editing the GridFTP host and/or port in the GRAM and GridFTP file system mapping config file: $GLOBUS_LOCATION/etc/gram-service/globus_gram_fs_map_config.xml.

By default, the globus services will assume 8443 is the port the Globus container is using. However the container can be run under a non-standard port, for example:

                % globus-start-container -p 4321
            

If you wish to specify a non-standard gridmap file in a multi-user installation, two basic configurations need to be changed:

Example: global_security_descriptor.xml

                ...
                
                <gridmap value="/opt/grid-mapfile"/>
                
                ...
            

sudoers

                ...
                
                # Globus GRAM entries
                globus  ALL=(username1,username2) 
                NOPASSWD: /opt/globus/GT4.1.1/libexec/globus-gridmap-and-execute 
                -g /opt/grid-mapfile
                /opt/globus/GT4.1.1/libexec/globus-job-manager-script.pl *
                globus  ALL=(username1,username2) 
                NOPASSWD: /opt/globus/GT4.1.1/libexec/globus-gridmap-and-execute 
                -g /opt/grid-mapfile
                /opt/globus/GT4.1.1/libexec/globus-gram-local-proxy-tool *
                
                ...
            

RFT is used by GRAM to stage files in and out of the job execution environment. In the default configuration, RFT is hosted in the same container as GRAM and is assumed to have the same service path and standard service names. This need not be the case. For example, the most likely alternative scenario is that RFT would be hosted seperately in a container on a different machine. In any case, both the RFT and the Delegation Service endpoints need to be adjustable to allow this flexibility. The following options can be passed to the setup-gram-service-common script to affect these settings:

            --staging-protocol=<protocol>
            --staging-host=<host>
            --staging-port=<port>
            --staging-service-path=<RFT and Delegation factory service path>
            --staging-factory-name=<RFT factory service name>
            --staging-delegation-factory-name=<name of Delegation factory service used by RFT>
        

for example

            % setup-gram-service-common \
            --staging-protocol=http
            --staging-host=somemachine.fakedomain.net
            --staging-port=8444
            --staging-service-path=/tomcat/services/
            --staging-factory-name=MyReliableFileTransferFactoryService
            --staging-delegation-factory-name=MyDelegationFactoryServiceForRFT
        

will internally cause the GRAM service code to construct the following EPR addresses:

            http://somemachine.fakedomain.net:8444/tomcat/services/MyReliableFileTransferFactoryService
            
            http://somemachine.fakedomain.net:8444/tomcat/services/MyDelegationFactoryServiceForRFT
        

Aside from Derby, database schemas for MySQL and PostgreSQL are provided. They can be found in $GLOBUS_LOCATION/share/globus_wsrf_gram/. After creating the database, the JNDI configuration of WS-GRAM must be adapted in order to use the non-default database.

Note, that there's more than one configuration section for ManagedJobFactoryService in the JNDI registry:

The database configuration must be specified for each factory, except for the BES factory which can't be used at the moment due to it's prototype status. The database settings must be specified in the section for the ManagedJobFactoryService and the v4_2/ManagedJobFactoryService. The important parameters are driverClassName, url, username and password.

The following describes how to configure MySQL as DBMS. Similar settings must be done if PostgreSQL is used.

The file $GLOBUS_LOCATION/etc/gram-service/jndi-config.xml contains configuration information that is common to every local resource manager.

More precisely, the configuration data it contains pertains to the implementation of the GRAM WSRF resources (factory resources and job resources), as well as initial values of WSRF resource properties that are always published by any Managed Job Factory WSRF resource.

The data is categorized by service, because according to WSRF, in spite of the service/resource separation of concern, a given service will use only one XML Schema type of resource. In practice it is therefore clearer to categorize the configuration resource implementation by service, even if theoretically speaking a given resource implementation could be used by several services. For more information, refer to the Java WS Core documentation.

Here is the decomposition, in JNDI objects, of the common configuration data, categorized by service. Each XYZHome object contains the same Globus Core-defined information for the implementation of the WSRF resource, such as the Java implementation class for the resource (resourceClass datum), the Java class for the resource key (resourceKeyType datum), etc.

When a SOAP call is made to a GRAM factory service in order to submit a job, the call is actually made to a GRAM service-resource pair, where the factory resource represents the local resource manager to be used to execute the job.

There is one directory gram-service-<manager>/ for each local resource manager supported by the GRAM installation.

For instance, let's assume the command line:

% ls etc | grep gram-service-

gives the following output:

gram-service-Fork
gram-service-LSF
gram-service-Multi

In this example, the Multi, Fork and LSF job factory resources have been installed. Multi is a special kind of local resource manager which enables the GRAM services to support multijobs.

The JNDI configuration file located under each manager directory contains configuration information for the GRAM support of the given local resource manager, such as the name that GRAM uses to designate the given resource manager. This is referred to as the GRAM name of the local resource manager.

For instance, $GLOBUS_LOCATION/etc/gram-service-Fork/jndi-config.xml contains the following XML element structure:

    <service name="ManagedJobFactoryService">
        <!-- LRM configuration:  Fork -->
        <resource
            name="ForkResourceConfiguration"
            type="org.globus.exec.service.factory.FactoryResourceConfiguration">
            <resourceParams>
                [...]
                <parameter>
                    <name>
                        localResourceManagerName
                    </name>
                    <value>
                        Fork
                    </value>
                </parameter>           
                <!-- Site-specific scratchDir
Default: ${GLOBUS_USER_HOME}/.globus/scratch
                <parameter>
                    <name>
                        scratchDirectory
                    </name>
                    <value>
                        ${GLOBUS_USER_HOME}.globus/scratch
                    </value>
                </parameter>           
                -->
            </resourceParams>
        </resource>        
    </service>

In the example above, the name of the local resource manager is Fork. This value can be used with the GRAM command line client in order to specify which factory resource to use when submitting a job. Similarly, it is used to create an endpoint reference to the chosen factory WS-Resource when using the GRAM client API.

In the example above, the scratchDirectory is set to ${GLOBUS_USER_HOME}/.globus/scratch. This is the default setting. It can be configured to point to an alternate file system path that is common to the compute cluster and is typically less reliable (auto purging), while offering a greater amount of disk space (thus "scratch").

By default, the GLUECE: resource property (which contains GLUE data) is sent to the default Index Service:

You can configure which resource properties are sent in GRAM4's registration.xml file, $GLOBUS_LOCATION/etc/gram-service/registration.xml. The following is the relevant section of the file (as it is set by default):

            <Content xsi:type="agg:AggregatorContent"
            xmlns:agg="http://mds.globus.org/aggregator/types">
            
            <agg:AggregatorConfig xsi:type="agg:AggregatorConfig">
            
            <agg:GetResourcePropertyPollType
                xmlns:glue="http://mds.globus.org/glue/ce/1.1">
            <!-- Specifies that the index should refresh information
            every 60000 milliseconds (once per minute) -->
            <agg:PollIntervalMillis>60000</agg:PollIntervalMillis>
            
            <!-- specifies the resource property that should be
            aggregated, which in this case is the GLUE cluster
            and scheduler information RP -->
            
            <agg:ResourcePropertyName>glue:GLUECE</agg:ResourcePropertyName>
            
            </agg:GetResourcePropertyPollType>
            </agg:AggregatorConfig> 
            <agg:AggregatorData/>
            </Content>
        

NOTE: if you are using one of the generic constructs described in the Section 6.2, “Additional Extension Constructs” section of the WS-GRAM User's Guide, skip to the subsection titled Customizing the Adapter Module.

For starters, this module logs various things to the log file specified in the logfile extension element. If you place this element at the start of the extensions you are creating support for, then you can look at the specified log file to get some idea of what the handler is doing. You can add new logging lines by using the $self->log() function. This simply takes a string that gets appended to the log file with a prefix of "<date string> EXTENSIONS HANDLER:".

There are three main subroutines that are used to handle parsing events and process them accordingly: Char(), StartTag(), and EndTag(). More handlers can be specified for other specific events when creating the XML::Parser instance in new() (for details, see the Section 6, “Technology Dependencies” section of the WS-GRAM Release Notes for a link to the XML::Parser documentation). Descriptions of what the three main subroutines do currently are layed out bellow. Modify the subroutines as neccesary to achieve your goal.

Char() doesn't do anything but collect CDATA found between the current element start and end tags. You can access the CDATA for the current element by using $self->{CDATA}.

StartTag() is responsible for collecting the attributes associated with the element. It also increments the counter which keeps track of the number of child elements under the current extension element, and pushes the current element name onto the @scope queue for later use.

EndTag() is used for taking the CDATA collected by Char() and creating new Perl job description hash entries. This is most likely where you will need to do most of your work when adding support for new extension elements. Two useful variables are $currentScope and $parentScope. These indicate the current element that is being parsed and the parent of the element being parsed respectively. This is useful for establishing a context from which to work from. The @scope queue is poped at the end of this subroutine.

Each adapter and each extension's purpose is different, so there aren't any specific instructions for modifying the resource manager/scheduler adapter module. It is suggested that you spend some time trying to understand what the adapter does and how before making your changes.

Any new hash entries you created in ExtensionsHandler.pm (see the "Customizing ExtensionsHandler.pm" section above) can be accessed by calling $description->entryname() from the adapter module, where 'entryname' is the name of the entry that was added.

See the Section 6.2, “Additional Extension Constructs” section of the WS-GRAM User's Guide for details on generic constructs that are already supported in ExtensionsHandler.pm. This is often an easier route to implmenting your extensions than creating a custom construct.