Transcript Resource Management Overview

Globus GRAM for Developers
Stuart Martin, Peter Lane
Argonne National Lab
Session Overview
Q: What is this session about?
A:This presentation will cover the features, interface,
architecture, performance, and future plans of the
Globus Toolkit v4 Web Services Grid Resource
Allocation and Management (GRAM4) component.

Four-part discussion (~ 20 mins/each)

Overview of GRAM Model

How to use client software

How to administer servers

Future plans
2
GRAM: Part 1
Overview of GRAM Model…
3
What is GRAM?

GRAM is a Globus Toolkit component


GRAM is a unifying remote interface to
Resource Managers


For Grid job management
Yet preserves local site security/control
GRAM is for stateful job control

Reliable operation

Asynchronous monitoring and control

Remote credential management

File staging via RFT and GridFTP
4
Grid Job Management Goals
Provide a service to securely:
 Create an environment for a job
 Stage files to/from environment
 Cause execution of job process(es)




Via various local resource managers
Monitor execution
Signal important state changes to client
Enable client access to output files

Streaming access during execution
5
Job Submission Model



Create and manage one job on a resource
Submit and wait
Not with an interactive TTY



File based stdin/out/err
Supported by all batch schedulers
More complex than RPC


Optional steps before and after submission message
Job has complex lifecycle



Staging, execution, and cleanup states
But not as general as Condor DAG, etc.
Asynchronous monitoring
6
Job Submission Options


Optional file staging

Transfer files “in” before job execution

Transfer files “out” after job execution
Optional file streaming


Monitor files during job execution
Optional credential delegation

Create, refresh, and terminate delegations

For use by job process

For use by GRAM to do optional file staging
7
Job Submission Monitoring

Monitor job lifecycle

GRAM and scheduler states for job


Job execution status


StageIn, Pending, Active, Suspended, StageOut,
Cleanup, Done, Failed
Return codes
Multiple monitoring methods

Simple query for current state

Asynchronous notifications to client
8
Secure Submission Model

Secure submit protocol
PKI authentication
 Authorization and mapping



Further authorization by scheduler


Based on Grid ID
Based on local user ID
Secure control/cancel
Also PKI authenticated
 Owner has rights to his jobs and not others’

9
Secure Execution Model


After authorization…
Execute job securely

User account “sandboxing” of processes


Initialization of sandbox credentials



According to mapping policy and request details
Client-delegated credentials
Adapter scripts can be customized for site needs
 AFS, Kerberos, etc
Multiple levels of audit possible



Container
Sudo
Local scheduler
10
Secure Staging Model


Before and after sandboxed execution…
Perform secure file transfers

Create RFT request




To local or remote RFT service
PKI authentication and delegation
In turn, RFT controls GridFTP
 Using delegated client credentials
GridFTP



PKI authentication
Authorization and mapping by local policy files
further authorization by FTP/unix perms
11
Users/Applications:
Job Brokers, Portals, Command line tools, etc.
GRAM WSDLs
+
Job Description
Schema
(executable, args,
env, …)
GRAM4
WS standard
interfaces for
subscription,
notification,
destruction
Resource Managers:
PBS, Condor, LSF, SGE, Loadleveler, Fork
12
GRAM4
Approach
compute element and service host(s)
compute element
Delegation
sudo
xfer request
client
GRAM
services
GRAM
adapter
GridFTP
RFT
local sched.
user job
FTP control
FTP data
GridFTP
remote storage element(s)
13
Other Approach Highlights

Scalability improvements
(discussed next)

sudo/auth_and_exec
to limit damage risk from software failures
 to improve audit capabilities


Extensibility

Retain: scheduler adapter structure


To extend for new platforms
Improved: authorization callouts

To better integrate with site practices
14
Usage Scenarios: the Ideal
“GRAM should add little to no overhead
compared to an underlying batch system”

Submit as many jobs to GRAM as is possible
to the underlying scheduler



Goal - 10,000 jobs to a batch scheduler
Goal – efficiently fill the process table for fork
scheduler
Submit/process jobs as fast to GRAM as is
possible to the underlying scheduler

Goal - 1 per second
15
Usage Scenarios: the Attempt

Efforts and features towards the goal



Allow job brokers the freedom to optimize

E.g. Condor-G is smarter than globusrun-ws

Protocol steps made optional and shareable
Reduced cost for GRAM service on host

Single WSRF host environment

Better job status monitoring mechanisms
More scalable/reliable file handling

GridFTP and RFT instead of globus-url-copy

Removal of non-scalable GASS caching
16
Production Quality

Service performance

Throughput



Max concurrency



Number of jobs (/bin/date) GRAM can process per minute
100
Total jobs a GRAM service can manage at one time without
failure
32,000
Job burst



Many simultaneous job submissions
Are the error conditions acceptable?
Job should be rejected, before overloading the service
container or service host
17
Production Quality

Service Stability & Recovery

Service uptime


Under a moderate load, how long can the GRAM
service process jobs without failure / reboot?
Job recovery


After reboot, processing/monitoring resumes for
submitted jobs
Clients resume control of jobs
18
Reasonable Applications Today

High throughput job sets: two approaches
1.
Use GRAM for every application task

2.

Jobs durations > 1 minute
Use GRAM for starting user/VO services

Course-grain jobs handle task/transaction flow

As in Condor glide-ins
MPICH-G4 (MPIG)

Large-scale multi-site/grid MPI jobs

Co-allocation but no co-reservation yet

Estimated release - Q4 2006
19
GRAM: Part 2
How to use client software…
20
How to use Client Software

Command line programs

WSDL interface
21
Command Line Programs

globusrun-ws


grid-proxy-init



Query a services resource properties
globus-url-copy


Creates client-side user proxy
wsrf-query


Submit and monitor gram jobs
Transfer files to remote hosts
globus-credential-delegate
globus-credential-refresh

Credential management to remote hosts
22
globusrun-ws

Written in C (C WS Core)


Supports GRAM multi-jobs or single jobs


Submission, monitoring, cancellation
Credential management


Faster startup and execution
Automatic or user-supplied delegation
Streaming of job stdout/err during execution

Advanced use of GridFTP client library
23
Simple Job: Step 1

Create a user proxy


Your temporary grid credential
Command Example:
% grid-proxy-init
Your identity:
/DC=org/DC=doegrids/OU=People/CN=Stuart Martin 564728
Enter GRID pass phrase for this identity:
Creating proxy......................... Done
Your proxy is valid until: Fri Jan 7 21:35:31 2005
24
Simple Job: Step 2

Submit job to a GRAM service
default factory EPR
 generate job RSL to default localhost


Command example:
% globusrun-ws -submit -c /bin/touch touched_it
Submitting job...Done.
Job ID: uuid:002a6ab8-6036-11d9-bae60002a5ad41e5
Termination time: 01/07/2005 22:55 GMT
Current job state: Active
Current job state: CleanUp
Current job state: Done
Destroying job...Done.
25
Complete Factory Contact

Override default EPR

Select a different host/service

Use “contact” shorthand for convenience


Relies on proprietary knowledge of EPR format!
Command example:
% globusrun-ws -submit –F \
https://140.221.65.193:4444/wsrf/services\
/ManagedJobFactoryService \
-c /bin/touch touched_it
26
Read RSL from File

Command:
% globusrun-ws -submit -f touch.xml

Contents of touch.xml file:
<job>
<executable>/bin/touch</executable>
<argument>touched_it</argument>
</job>
27
Batch Job Submissions
% globusrun-ws -submit -batch -o job_epr -c /bin/sleep 50
Submitting job...Done.
Job ID: uuid:f9544174-60c5-11d9-97e3-0002a5ad41e5
Termination time: 01/08/2005 16:05 GMT
% globusrun-ws -monitor -j job_epr
job state: Active
Current job state: CleanUp
Current job state: Done
Requesting original job description...Done.
Destroying job...Done.
28
Batch Job Submissions
% globusrun-ws -submit -batch -o job_epr -c /bin/sleep 50
Submitting job...Done.
Job ID: uuid:f9544174-60c5-11d9-97e3-0002a5ad41e5
Termination time: 01/08/2005 16:05 GMT
% globusrun-ws -status -j job_epr
Current job state: Active
% globusrun-ws -status -j job_epr
Current job state: Done
% globusrun-ws -kill -j job_epr
Requesting original job description...Done.
Destroying job...Done.
29
Common/useful options

globusrun-ws -J


globusrun-ws -S


Perform delegation as necessary for job’s file staging
globusrun-ws -s


Perform delegation as necessary for job
Stream stdout/err during job execution to the
terminal
globusrun-ws -self

Useful for testing, when you have started the service
using your credentials instead of host credentials
30
Staging job
<job>
<executable>/bin/echo</executable>
<directory>/tmp</directory>
<argument>Hello</argument>
<stdout>job.out</stdout>
<stderr>job.err</stderr>
<fileStageOut>
<transfer>
<sourceUrl>file:///tmp/job.out</sourceUrl>
<destinationUrl>
gsiftp://host.domain:2811/tmp/stage.out
</destinationUrl>
</transfer>
</fileStageOut>
</job>
31
RFT Options
<fileStageOut>
<transfer>
<sourceUrl>file:///tmp/job.out</sourceUrl>
<destinationUrl>
gsiftp://host.domain:2811/tmp/stage.out
</destinationUrl>
<rftOptions>
<subjectName>
/DC=org/DC=doegrids/OU=People/CN=Stuart Martin 564728
</subjectName>
<parallelStreams>4</parallelStreams>
</rftOptions>
</transfer>
</fileStageOut>
32
RSL Variable

Enables late binding of values


Values resolved by GRAM service
System-specific variables

${GLOBUS_USER_HOME}

${GLOBUS_LOCATION}

${GLOBUS_SCRATCH_DIR}

Alternative directory that is shared with compute node

Typically providing more space than user’s HOME dir
33
RSL Variable Example
<job>
<executable>/bin/echo</executable>
<argument>HOME is ${GLOBUS_USER_HOME}</argument>
<argument>SCRATCH = ${GLOBUS_SCRATCH_DIR}</argument>
<argument>GL is ${GLOBUS_LOCATION}</argument>
<stdout>${GLOBUS_USER_HOME}/echo.stdout</stdout>
<stderr>${GLOBUS_USER_HOME}/echo.stderr</stderr>
</job>
34
RSL Extensions Support

4.0.3 does not support extension by default


Update packages are available to add
extension support

http://www.globus.org/toolkit/downloads/development/

globus_gram_job_manager-7.14 plus dependencies
All 4.1.x releases support extensions by
default
35
RSL Extensions Example
<job>
<executable>/bin/echo</executable>
<extensions>
<email_address>[email protected]</email_address>
<extensions>
</job>


Simple string extension elements are converted into
single-element arrays
Code example in pbs.pm:
if($description->email_address() ne '')
{
print JOB '#PBS -M ', \
$description->email_address(), "\n";
}
36
How to use Client Software

Command line programs

WSDL interface
37
ManagedJobFactory portType

createManagedJob operation


Creates either an MMJR or MEJR
Input:





Initial Termination Time
Job ID
 UUID of the job resource, for job reliability/recoverability
Subscribe Request
 Client can include a request to subscribe for job state notifications
with the job submission to avoid an extra operation call
Job Description / RSL
 Either a single or multi-job description
Output:



newTerminationTime
managedJobEndpoint
subscriptionEndpoint
- new termination time of the job resource
- EPR of the newly created job resource
- EPR of the notification subscription
38
ManagedJob portType


Base port type for the MEJS and MMJS
release operation

Release a holdState set in the job description




Only one hold state can be set/released
Input: None
Output: None
State change notifications




State - job state (Active, Pending, Done, Cleanup…)
Fault - fault causing a Failed state (if applicable)
Exit Code - exit code of the job process
Holding - boolean indicating if the job is in a hold state
39
ManagedJob portType
On destroy, or soft state termination…
The MJS will cleanup everything
1.
2.
Stop any outstanding tasks

Cancel/terminate the execution

Destroy RFT stage in, out requests
Process CleanUp state

Submit request to RFT to remove files/directories


3.
RSL attribute fileCleanUp
Remove job user proxy file
Destroy job resource
40
ManagedExecutableJobService


Executes the requested job process(es)
specified in the RSL
Resource Properties
(ManagedExecutableJobPortType)










serviceLevelAgreement
state
faults
localUserId
userSubject
holding
stdoutURL
stderrURL
credentialPath
exitCode
-
- the RSL / Job Description
the current job state
the fault causing a Failed state
the username of the resource owner
the GSI subject of the resource owner
boolean indiciating the job is holding
the GridFTP URL to the stdout file
the GridFTP URL to the stderr file
the local path to the user proxy file
the exit code of the job proces (if applicable)
41
ManagedMultiJobService

Processes a multi-job RSL
submits the sub-jobs to the specified
ManagedJobFactoryService.
 Sub-jobs cannot be multi-jobs themselves.


Resource Properties
(ManagedMultiJobPortType)







serviceLevelAgreement
- the multi-job RSL / Job Description
state
- the current overall state
faults
- the fault causing a Failed state
localUserId
- the username of the resource owner
userSubject
- the GSI subject of the resource owner
holding
- boolean indiciating all jobs are holding
subJobEndpoint
- list of endpoints to the sub-jobs
42
Our Goals

Highly functional interface





Expressive job description language
Basic command line clients


grid service WSDLs
C API
Java API
Should be useable from shell scripts
Collaborate with others to create more capable and
complete clients

E.g. Condor-G, TG’s Science Gateways, Portals
43
GRAM: Part 3
How to administer servers…
44
4.0 Quickstart Guide

Consult this guide first for basic GT setup





Setting up first machine
Setting up second machine
Setting up a compute cluster - PBS
www.globus.org/
toolkit/docs/4.0/admin/docbook/quickstart.html
Then consult GRAM admin guide for additional details

www.globus.org/toolkit/docs/4.0/admin/docbook/ch11.html
45
Typical GRAM service setup

Host credentials




Gridmap file




For client/service authentication
For client authorization of the service
Existing GT2/GT3 host certs can be used
Entries for each user allowed to execute job’s
Maps the grid ID to a local user account
Same syntax as GT2, GT3 gridmap files
Installed sudo

Method for GRAM to runs commands in the user’s
account
46
sudo configuration

sudo policies

Done by hand by root
Runas_Alias GRAMUSERS = ! root, ! wheel, …
globus ALL=(GRAMUSERS) NOPASSWD:
/sandbox/globus/install/libexec/globus-gridmap-and-execute
/sandbox/globus/install/libexec/globus-job-manager-script.pl *
globus ALL=(GRAMUSERS) NOPASSWD:
/sandbox/globus/install/libexec/globus-gridmap-and-execute
/sandbox/globus/install/libexec/globus-gram-local-proxy-tool *

globus-gridmap-and-execute


Redundant if sudo is locked down tightly
Enforce that GRAM only targets accounts in gridmap



So sudo policy need not enumerate all GRAM users at large/dynamic sites
In fact, you can audit this tool and change GRAMUSERS to ALL if you like…
Replace this with your own authz tool (callout)
47
Local Resource Manager
Adapters

GT provides/supports 4 RM adapters


3rd party RM adapters exist





PBS, LSF, Condor, Fork
SGE, LoadLeveler, GridWay
Tell us about yours and we’ll add to GT web pages!
All 4 RM adapters are included in all binary and
source installers
Only Fork is configured automatically
Configuring an RM adapter

Add configure arguments

./configure --enable-wsgram-pbs …
48
File staging functionality

GridFTP Server

Could be run on a separate host from GRAM service
container to improve performance / scalability

cpu intensive

globus_gram_fs_map_config.xml


Config the GridFTP server(s) to use for local file staging
RFT

Requires PostgreSQL DB setup

Usability: 4.1.x Defaults to embedded DB (Derby)
49
GRAM / GridFTP file system mapping





Associates compute resources and GridFTP servers
Maps shared filesystems of the gram and gridftp hosts, e.g.
 Gram host mounts homes at /pvfs/home
 gridftp host mounts same at /pvfs/users/home
GRAM resolves file:/// staging paths to local GridFTP URLs
 File:///pvfs/home/smartin/file1... resolves to:
 gsiftp://host.domain:2811/pvfs/users/home/smartin/file1
$GL/etc/gram-service/globus_gram_fs_map_config.xml
Client will need to know mappings to stage files separately
from WS GRAM
50
Non-default Setup

./setup-gram-service-common



GridFTP Server config



To change GRAM configuration
Run in $GLOBUS_LOCATION/setup
Default is for localhost, port 2811
--gridftp-server=gsiftp://gridftp.host.org:1234
RFT Service config




Default is localhost, port 8443
--stage-protocol=https
--staging-host=host.domain.org
--staging-port=4321
51
Setup: Container Credentials

Default: host credentials



/etc/grid-security/containercert.pem
/etc/grid-security/containerkey.pem
To configure for a user proxy

Update container global security descriptor


Tell GRAM the subject to expect for authorization of
the RFT service



Comment out <credential> element
$GL/etc/globus_wsrf_core/global_security_descriptor.xml
./setup-gram-service-common
--staging-subject=
"/DC=org/DC=doegrids/OU=People/CN=Stuart Martin
564720”
Use “-self” argument with globusrun-ws
Default GT auth in 4.1.1 will be “host” *or* “self”
52
GRAM: Part 4

Future Plans
53
4.2 Series WS GRAM


4.1.x is dev series for eventual stable 4.2.x stable
series
4.1.0 released July 06







RSL extension support
globus-job-*-ws scripts included by default
Improved service throttling controls
Persistence data stored in DB
resource manager adapter API
Removed unnecessary dependencies to Pre-WS
GRAM
4.1.1 (no target date yet)


Initial support for JSDL jobs
Service auditing to DB
54
WS GRAM Standards Compliance

JSDL





Target is 4.1.1 (definitely 4.2.0)
Will preserve current interface, so 4.0.x job
descriptions will work just fine
Adding new createManagedJobFromJSDLDocument
operation
Globusrun-ws will choose appropriate create
operation based on job description contents
OGSA-BES


Target is 4.4 (spec is not finished, so 4.2 is unlikely)
Will preserve 4.0.x interface as well
55
Service Auditing

Follow along on bugzilla “roadmap” item



Prototype written and deployed on TeraGrid




http://bugzilla.globus.org/bugzilla/show_bug.cgi?id=4409
Add yourself to cc list
In evaluation phase
provides the capability for a TG grid user to get TG usage info
using a grid job id (from GRAM)
Audit DB entries provide join between grid job id and local TG
accounting DB
Will be included in 4.1.x series to be included in 4.2

Probably disable by default in GT releases
56
Advanced Reservation



Investigation is underway
No firm plans yet, but high on our priority list
Follow along on bugzilla “roadmap” item

http://bugzilla.globus.org/bugzilla/show_bug.cgi?id=4045
57
Performance testing
with OSG

Test scenario



Solved reliability issue with default condor-g jobs


Included in 4.0.3
Found/fixed bugs in RFT which effected performance by appox
250% for staging jobs



submit large (3500) job run through condor-g to WS GRAM to
LRM condor
Job is create unique job dir; 2MB stageIn, 2MB stageOut, cleanup
job dir
From 5.2 jpm to 13 jpm
Patches to 4.0.3 will be made available soon
We plan on writing up results and provide config
recommendation for GT container and condor-g
58
WS GRAM Usage Statistics

July 6 thru Aug 6th 2006



651517 jobs submitted
25 unique domains (e.g. .edu, .org, .gov)
356 unique IPs (Container installations with WS GRAM)
59
Documentation

4.0.x GRAM documentation



http://www.globus.org/toolkit/docs/4.0/execution/w
sgram/
4.1.x GRAM documentation


Guides: admin, user, developer, overview, public interface
http://www.globus.org/toolkit/docs/4.1/execution/w
sgram/
Main 4.0.x documentation


http://www.globus.org/toolkit/docs/4.0/
Download, release notes, links to all GT projects/
components
60
Writing New RM Adapters

http://www.globus.org/toolkit/docs/4.0/ex
ecution/wsgram/developer/schedulertutorial.html

Scheduler perl modules (e.g. pbs.pm)


Submitting jobs, canceling jobs, setup and packaging
Scheduler Event Generator (SEG)

Monitoring events from the scheduler for all job for all
users; it runs under a privileged account
61
Bugzilla

If you’ve found a bug (not a question!)


http://bugzilla.globus.org/
GRAM product, wsrf* components
62
Globus Development

GlobDev - Open development

Globus governance model based on Apache



GRAM project



Developers (committers) control direction of software
components (projects)
http://dev.globus.org
http://dev.globus.org/wiki/GRAM
Email lists: gram-user, gram-dev, gram-announce, gramcommit
GT project

gt-user, gt-dev
63
Thanks to the GRAM developers!





Peter Lane - ANL
Joe Bester - ANL
Ravi Madduri - ANL
Martin Feller - UofC
Plus the entire GT dev team
64
Meet the Developers Session
at Globus Alliance Booth (152A-P7)
September 12
8:00am - 9:00am "Java WS Core and Security (C, Java)" -- Olle Mulmo, Jarek
Gawor, Rachana Anantakrishnan
11:30am -12:30pm "RLS" -- Rob Schuler, Ann Chervenak
12:30pm -1:30pm "MDS" -- Mike D'arcy, Laura Pearlman
3:00pm - 4:00pm ”Resource Management (GRAM, Virtual Workspaces
and Dynamic Accounts)" – Stu Martin, Peter Lane, Tim Freeman, Kate
Keahey
6:00pm - 7:00pm "C WS Core" -- Joe Bester
7:00pm - 8:00pm "Python WS Core" -- Joshua Boverhof
September 13
8:00am - 9:00am "GridShib" -- Von Welch, Ton Scavo, Tim Freeman
11:30am - 12:30pm "GT Installation and Administration" -- Charles Bacon
12:30pm - 1:30pm "MyProxy" -- Jim Basney
3:00pm - 4:00pm "GridFTP, XIO, RFT" -- John Bresnahan, Ravi Madduri
65
COME CELEBRATE WITH US!
In appreciation of your support of all things
Globus over the past decade, you are cordially
invited to the Globus 10th Birthday Party.
When: Monday, September 11, 2006 - 7:00pm, immediately
following Ian Foster’s Globus State of the Union Keynote.
Where: The convention center concourse, in the center of
the GlobusWORLD / GridWorld conference activity.
What: Food, drinks, music, friends and lots of fun!
66