Transcript Grid Computing High-Performance Grid Computing and Research Networking Presented by Selim Kalayci

High-Performance Grid Computing and
Research Networking
Grid Computing
Presented by Selim Kalayci
Instructor: S. Masoud Sadjadi
http://www.cs.fiu.edu/~sadjadi/Teaching/
sadjadi At cs Dot fiu Dot edu
1
Acknowledgements

The content of many of the slides in this lecture
notes have been adopted from the online resources
prepared previously by the people listed below. Many
thanks!

Henri Casanova





Principles of High Performance Computing
http://navet.ics.hawaii.edu/~casanova
[email protected]
Ian Foster
Presentations&Tutorials from

www.globus.org
2
Agenda







Grid Computing
Grid Middleware - Globus
Security in Globus
Data Management
Execution Management
Monitoring
Metaschedulers - Gridway
3
Multiple Computers



Adding CPUs to a single computer becomes
very expensive
How about multiple computers together?
Linux Clusters (60% of Top-500 list)
Blue/Gene: 30K computers
4
Beyond the machine room?

Need more capacity than available at (most) single sites





Everyone would like a 10K-node 100GHz cluster
Very expensive (cooling, power)
More economical to have multiple sites
Need to locate available resources now
Data/Instruments are inherently distributed
Campus
Machine Room
Nation
5
Grid Computing
A dynamic multi-institutional network of computers that come together to
share resources for the purpose of coordinated problem solving.
resource
application
Achieved through:
institutional
boundary
1. Open general-purpose protocols
2. Standard interfaces
6
Layers in Grid
7
A Grid Checklist




coordinates resources that are not subject to centralized control
…
… using standard, open, general-purpose protocols and
interfaces …
… to deliver nontrivial qualities of service.
Virtual Organizations
 Group of individuals or institutions defined by sharing rules
to share the resources of “Grid” for a common goal.
 Example: Application service providers, storage service
providers, databases, crisis management team, consultants.
8
How is a grid different?

Grids focus on site autonomy

Grids involve heterogeneity


Grids involve more resources than just
computers and networks
Grids focus on the user
9
Agenda







Grid Computing
Grid Middleware - Globus
Security in Globus
Data Management
Execution Management
Monitoring
Metaschedulers - Gridway
10
Grid Infrastructure

Distributed management




Of physical resources
Of software services
Of communities and their policies
Unified treatment



Build on Web services framework
Use WS-RF, WS-Notification (or WSTransfer/Man) to represent/access state
Common management
abstractions & interfaces
11
Globus is Open Source
Grid Infrastructure

Implement key Web services standards


Software for Grid infrastructure




Service-enable new & existing resources
E.g., GRAM on computer, GridFTP on storage
system, custom application services
Uniform abstractions & mechanisms
Tools to build applications that exploit Grid
infrastructure


State, notification, security, …
Registries, security, data management, …
Enabler of a rich tool & service ecosystem
12
GLOBUS TOOLKIT 4 – GT4



Open source toolkit developed by The Globus Alliance
that allows us to build Grid applications.
Organized as a collection of loosely coupled
components.
Consists of services, programming libraries, and
development tools.
13
GT Domain Areas

Core runtime


Security


Provision, deploy, & manage services
Data management


Apply uniform policy across distinct systems
Execution management


Infrastructure for building new services
Discover, transfer, & access large data
Monitoring

Discover & monitor dynamic services
14
GT4 Components
15
WSRF & WS-Notification

Naming and bindings (basis for virtualization)


Lifecycle (basis for fault resilient state mgmt)





Resource properties associated with resources
Operations for querying and setting this info
Asynchronous notification of changes to properties
Service groups (basis for registries, collective svcs)


Resources created by services following factory pattern
Resources destroyed immediately or scheduled
Information model (basis for monitoring, discovery)


Every resource can be uniquely referenced, and has one or more
associated services for interacting with it
Group membership rules & membership management
Base Fault type
16
Agenda







Grid Computing
Grid Middleware - Globus
Security in Globus
Data Management
Execution Management
Monitoring
Metaschedulers - Gridway
17
Security Services



Forms the underlying communication
medium for all the services
Secure Authentication and Authorization
Single Sign-on



User need not explicitly authenticate
himself every time a service is requested
Uniform Credentials
Ex: GSI (Globus Security Infrastructure)
18
Grid Security Infrastructure - GSI

Grid Security Infrastructure (GSI)

Use GSI as a standard mechanism for bridging
disparate security mechanisms




Doesn’t solve trust problem, but now things talk same
protocol and understand each other’s identity credentials
Basic support for delegation, policy distribution
Translate from other mechanisms to/from GSI as
needed
Convert from GSI identity to local identity for
authorization
19
Grid Security Infrastructure - GSI

Grid Security Infrastructure (GSI)

Based on standard PKI technologies



SSL protocol or WS-Security for authentication, message
protection
X.509 Certificates for asserting identity


CAs allow one-way, light-weight trust relationships (not just
site-to-site)
for users, services, hosts, etc.
Proxy Certificates

GSI extension to X.509 certificates for delegation, single signon
20
Gridmap file

A gridmap file at each site maps the grid id of a user to a local
id
 The grid id of the user is his/her subject in the grid user
certificate
 The local id is site-specific;
 multiple grid ids can be mapped to a single local id
 Usually a local id exists for each VO participating in that
grid effort
 The local ids are then used to implement site specific
policies
 Priorities etc.
21
Gridmap file entry


The gridmap-file is maintained by the site
administrator
Each entry maps a Grid DN (distinguished name of
the user; subject name) to local user names
#
#Distinguished Name
#
Local username
“/DC=org/DC=doegrids/OU=People/CN=Laukik Chitnis 712960”
“/DC=org/DC=doegrids/OU=People/CN=Richard Cavanaugh 710220”
“/DC=org/DC=doegrids/OU=People/CN=JangUk In 712961”
“/DC=org/DC=doegrids/OU=People/CN=Jorge Rodriguez 690211”
ivdgl
grid3
ivdgl
osg
22
How to create and use an
Identity (1)

Run the below command to generate a personal grid
identity certificate.
grid-cert-request

This will create the following files in
$HOME/.globus
usercert_request.pem (request to sign
certificate)
userkey.pem (private key - encrypted)
usercert.pem (public key - signed)
23
How to create and use an
Identity (2)

After you have created the request then you need to
mail it to the local certificate authority:
cat $HOME/.globus/usercert_request.pem | mail
[email protected] (or [email protected])

Then the CA will mail you back a signed certificate
which you will want to put into
$HOME/.globus/usercert.pem
(it can take up to a day for the CA to process the
request)
24
Commands to log in / logout

grid-proxy-init

This "logs you into" the globus system.

grid-proxy-info

Use this to see your status.

grid-proxy-destroy

Use this to log out.


A proxy is like a temporary ticket to use the Grid, default in the
above case being 12 hours.
Once this is done, you should be able to run “grid jobs”
 globus-job-run site-name command
25
Agenda







Grid Computing
Grid Middleware - Globus
Security in Globus
Data Management
Execution Management
Monitoring
Metaschedulers - Gridway
26
GT4 Data Management

Stage/move large data to/from nodes



Locate data of interest


Replica Location Service (RLS)
Replicate data for performance/reliability


GridFTP, Reliable File Transfer (RFT)
Alone, and integrated with GRAM
Distributed Replication Service (DRS)
Provide access to diverse data sources


File systems, parallel file systems, hierarchical
storage: GridFTP
Databases: OGSA DAI
27
GridFTP



What is GridFTP?
A secure, robust, fast, efficient, standards based, widely
accepted data transfer protocol
A Protocol

Multiple independent implementations can interoperate



This works. Both the Condor Project at Uwisand Fermi Lab have home
grown servers that work with ours.
Lots of people have developed clients independent of the Globus
Project.
We also supply a reference implementation:



Server
Client tools (globus-url-copy)
Development Libraries
28
Globus-url-copy


GridFTP-compliant client from the Globus team
Copy files from one URL to another URL



One URL is usually a gsiftp:// URL
Another URL is usually a file:/ URL
To move a file from remote GridFTP-enabled server to local machine
% globus-url-copy gsiftp://gcb.fiu.edu/tmp/jt
file:/home/skala001/jt

To put file onto server reverse URLs
% globus-url-copy file:/home/skala001/jt
gsiftp://gcb.fiu.edu/tmp/jt

Monitor performance using –vb flag
% globus-url-copy -vb gsiftp://gcb.fiu.edu/tmp/jt
file:/home/skala001/jt
29
Reliable File Transfer - RFT


WSRF compliant Fault-tolerant, High- performance
data transfer service
 Soft state.
 Notifications/Query
Reliability on top of high performance provided by
GridFTP.
 Fire and Forget.
 Integrated Automatic Failure Recovery.
 Network level failures.
 System level failures etc.
 Essentially a Data transfer scheduler with FIFO as
a Queue Policy.
30
RFT
RFT Client
SOAP
Messages
Notifications
(Optional)
RFT Service
GridFTP Server
Master
DSI
Protocol
Interpreter
GridFTP Server
Data
Channel
Data
Channel
IPC Link
IPC
Receiver
Protocol
Interpreter
Master
DSI
IPC Link
Slave
DSI
Data
Channel
Data
Channel
Slave
DSI
IPC
Receiver
31
Agenda







Grid Computing
Grid Middleware - Globus
Security in Globus
Data Management
Execution Management
Monitoring
Metaschedulers - Gridway
32
Execution Management

Common WS interface to schedulers


More generally: interface for process
execution management





Unix, Condor, LSF, PBS, SGE, …
Lay down execution environment
Stage data
Monitor & manage lifecycle
Kill it, clean up
A basis for application-driven provisioning
33
Grid Job Management Goals
Provide a service to securely:
 Create an environment for a job
 Stage files to/from environment
 Cause execution of job process(es)
 Via various local resource managers
 Monitor execution
 Signal important state changes to client
 Enable client access to output files
 Streaming access during execution
34
GRAM




GRAM:Globus Resource Allocation and Management
GRAM is a Globus Toolkit component
 For Grid job management
GRAM is a unifying remote interface to Resource
Managers
 Yet preserves local site security/control
GRAM is for stateful job control
 Reliable operation
 Asynchronous monitoring and control
 Remote credential management
 File staging via RFT and GridFTP
35
GT4 WS GRAM Architecture
Service host(s) and compute element(s)
Job events
Client
Delegate
Delegation
Transfer
request
RFT File
Transfer
SEG
Compute element
Local job control
sudo
GT4 Java Container
GRAM
GRAM
services
services
GRAM
adapter
GridFTP
FTP
control
Local
scheduler
User
job
FTP data
GridFTP
Remote
storage
element(s)
36
GT4 WS GRAM Architecture
Service host(s) and compute element(s)
Job events
Client
Delegate
Delegation
Transfer
request
RFT File
Transfer
SEG
Compute element
Local job control
sudo
GT4 Java Container
GRAM
GRAM
services
services
GRAM
adapter
GridFTP
FTP
control
Local
scheduler
User
job
FTP data
Delegated credential can be:
Made available to the application
GridFTP
Remote
storage
element(s)
37
GT4 WS GRAM Architecture
Service host(s) and compute element(s)
Job events
Client
Delegate
Delegation
Transfer
request
RFT File
Transfer
SEG
Compute element
Local job control
sudo
GT4 Java Container
GRAM
GRAM
services
services
GRAM
adapter
GridFTP
FTP
control
Local
scheduler
User
job
FTP data
Delegated credential can be:
Used to authenticate with RFT
GridFTP
Remote
storage
element(s)
38
GT4 WS GRAM Architecture
Service host(s) and compute element(s)
Job events
Client
Delegate
Delegation
Transfer
request
RFT File
Transfer
SEG
Compute element
Local job control
sudo
GT4 Java Container
GRAM
GRAM
services
services
GRAM
adapter
GridFTP
FTP
control
Local
scheduler
User
job
FTP data
Delegated credential can be:
Used to authenticate with GridFTP
GridFTP
Remote
storage
element(s)
39
A Simple Example

Command example:
% globusrun-ws -submit -c /bin/date
Submitting job...Done.
Job ID: uuid:002a6ab8-6036-11d9-bae6-0002a5ad41e5
Termination time: 01/07/2005 22:55 GMT
Current job state: Active
Current job state: CleanUp
Current job state: Done
Destroying job...Done.


A successful submission will create a new ManagedJob resource
with its own unique EPR for messaging
Use –o option to create the EPR file
% globusrun-ws -submit –o job.epr -c /bin/date
40
A Simple Example(2)

To see the output, use –s (stream) option
% globusrun-ws -submit –s -c /bin/date
Termination time: 06/14/2007 18:07 GMT
Current job state: Active
Current job state: CleanUp-Hold
Wed Jun 13 14:07:54 EDT 2007
Current job state: CleanUp
Current job state: Done
Destroying job...Done.
Cleaning up any delegated credentials...Done.

If you want to send the output to a file, use –so option
% globusrun-ws -submit –s –so job.out -c /bin/date
…
% cat job.out
Wed Jun 13 14:07:54 EDT 2007
41
A Simple Example(3)
Submitting your job to different schedulers
 Fork
% globusrun-ws -submit -Ft Fork -s -c /bin/date
(Actually, the default is Fork. So, you can skip it in
this case.)

SGE
% globusrun-ws -submit -Ft SGE -s -c /bin/hostname
42
Batch Job Submissions
% globusrun-ws -submit -batch -o job_epr -c
/bin/sleep 50
Submitting job...Done.
Job ID: uuid:f9544174-60c5-11d9-97e3-0002a5ad41e5
Termination time: 01/08/2005 16:05 GMT
% globusrun-ws -status -j job_epr
Current job state: Active
% globusrun-ws -status -j job_epr
Current job state: Done
% globusrun-ws -kill -j job_epr
Requesting original job description...Done.
Destroying job...Done.
43
Complete Factory Contact

Override default EPR


Select a different host/service
Use “contact” shorthand for convenience


Relies on proprietary knowledge of EPR format!
Command example:
%
globusrun-ws -submit –F gcb.fiu.edu\
-c /bin/date
44
Read RSL from File

Command:
% globusrun-ws -submit -f touch.xml

Contents of touch.xml file:
<job>
<executable>/bin/touch</executable>
<argument>touched_it</argument>
</job>
45
Resource Specification Language
(RSL)




RSL is the language used by the clients to
submit a job.
All job submission requests are described in
RSL, including the executable file and
arguments.
You can specify the type and capabilities of
resources to execute your job.
You can also coordinate Stage-in and Stageout operations through RSL.
46
Common/useful options

globusrun-ws -J


globusrun-ws -S


Perform delegation as necessary for job’s file
staging
globusrun-ws -s


Perform delegation as necessary for job
Stream stdout/err during job execution to the
terminal
globusrun-ws -self

Useful for testing, when you have started the
service using your credentials instead of host
credentials
47
Staging job
<job>
<executable>/bin/echo</executable>
<directory>/tmp</directory>
<argument>Hello</argument>
<stdout>job.out</stdout>
<stderr>job.err</stderr>
<fileStageOut>
<transfer>
<sourceUrl>file:///tmp/job.out</sourceUrl>
<destinationUrl>
gsiftp://host.domain:2811/tmp/stage.out
</destinationUrl>
</transfer>
</fileStageOut>
</job>
48
RSL Variable

Enables late binding of values


Values resolved by GRAM service
System-specific variables



${GLOBUS_USER_HOME}
${GLOBUS_LOCATION}
${GLOBUS_SCRATCH_DIR}


Alternative directory that is shared with compute node
Typically providing more space than user’s HOME dir
49
RSL Variable Example
<job>
<executable>/bin/echo</executable>
<argument>HOME is ${GLOBUS_USER_HOME}</argument>
<argument>SCRATCH =
${GLOBUS_SCRATCH_DIR}</argument>
<argument>GL is ${GLOBUS_LOCATION}</argument>
<stdout>${GLOBUS_USER_HOME}/echo.stdout</stdout>
<stderr>${GLOBUS_USER_HOME}/echo.stderr</stderr>
</job>
!!!/tmp/rslExample
50
GRAM Commands

Run a job using:
% globus-job-run localhost /bin/date

Submit to Fork:
% globus-job-run localhost/jobmanager-fork /bin/date

Submit a batch job using:
% globus-job-submit localhost /bin/sleep 50



globus-job-status
globus-job-get-output
globus-job-cancel
51
Running a Script in GRAM

Add this script to file “job”
#! /bin/csh -f
echo "Hello World from ";$GLOBUS_LOCATION/bin/globus-hostname
echo arg 1 = $1
echo arg 2 = $2
echo -n "sum is "
echo "$1+$2" | /usr/bin/bc –l

Change the permissions for “job”
% chmod +x job

Run the job
% globus-job-run localhost ./job 5 6

You should get
Hello World from
gcb.fiu.edu
arg 1 = 5
arg 2 = 6
sum is 11
!!!/tmp/job
52
Agenda







Grid Computing
Grid Middleware - Globus
Security in Globus
Data Management
Execution Management
Monitoring
Metaschedulers - Gridway
53
What is MDS4?

Grid-level monitoring system used most often for
resource selection and error notification



Uses standard interfaces to provide publishing of
data, discovery, and data access, including
subscription/notification


Aid user/agent to identify host(s) on which to run an
application
Make sure that they are up and running correctly
WS-ResourceProperties, WS-BaseNotification, WSServiceGroup
Functions as an hourglass to provide a common
interface to lower-level monitoring tools
54
MDS4 Components

Information providers



Higher level services




Index Service – a way to aggregate data
Trigger Service – a way to be notified of changes
Both built on common aggregator framework
Clients


Monitoring is a part of every WSRF service
Non-WS services are also be used
WebMDS
All of the tool are schema-agnostic, but
interoperability needs a well-understood common
language
55
Information Providers


GT4 information providers collect
information from some system and make it
accessible as WSRF resource properties
Growing number of information providers



Ganglia, CluMon, Nagios
SGE, LSF, OpenPBS, PBSPro, Torque
Many opportunities to build additional ones

E.g., network monitoring, storage systems, various
sensors
56
Information Providers


Data sources for the higher-level services
Some are built into services




Any WSRF-compliant service publishes some data
automatically
WS-RF gives us standard Query/Subscribe/Notify
interfaces
GT4 services: ServiceMetaDataInfo element
includes start time, version, and service type
name
Most of them also publish additional useful
information as resource properties
57
Information Providers:
GT4 Services

Reliable File Transfer Service (RFT)


Community Authorization Service (CAS)


Service status data, number of active transfers,
transfer status, information about the resource
running the service
Identifies the VO served by the service instance
Replica Location Service (RLS)


Note: not a WS
Location of replicas on physical storage systems
(based on user registrations) for later queries
58
Information Providers (2)

Other sources of data





Any executables
Other (non-WS) services
Interface to another archive or data store
File scraping
Just need to produce a valid XML
document
59
Information Providers:
Cluster and Queue Data

Interfaces to Hawkeye, Ganglia, CluMon,
Nagios




Basic host data (name, ID), processor information,
memory size, OS name and version, file system
data, processor load data
Some condor/cluster specific data
This can also be done for sub-clusters, not just at
the host level
Interfaces to PBS, Torque, LSF

Queue information, number of CPUs available and
free, job count information, some memory
statistics and host info for head node of cluster
60
Higher-Level Services

Index Service


Trigger Service


Caching registry
Warn on error conditions
All of these have common needs, and
are built on a common framework
61
MDS4 Index Service

Index Service is both registry and cache




Subscribes to information providers
In memory default approach



Datatype and data provider info, like a registry (UDDI)
Last value of data, like a cache
DB backing store currently being discussed to allow for very
large indexes
Can be set up for a site or set of sites, a specific set
of project data, or for user-specific data only
Can be a multi-rooted hierarchy

No *global* index
62
Container-wide Index



Each GT4 container has a local index
Collects information about services in that container
Each service registers to container index when
correctly configured
63
VO-wide indexes


Local indexes can be registered to VO wide indexes
Configfile at resource container or at VO index –
contains URL for resource or VO index
64
MDS4 Trigger Service



Subscribe to a set of resource
properties
Evaluate that data against a set of preconfigured conditions (triggers)
When a condition matches, action
occurs


Email is sent to pre-defined address
Website updated
65
Information models




Each information sources publishes
information in XML according to some
schema.
Some times the author of the information
source or the grid resource defines that
schema.
Some collaborative efforts to define common
schemas–for example GLUE for compute
information
Schema typically written in XSD, but not
required
66
GLUE schema




Grid Laboratory Uniform Environment
Schema developed by DataTAG for EU/USA
interoperability.
Modelled in UML
Implementations

XML version for MDS


Information collected from various cluster monitoring
systems
Also: LDAP and SQL versions (used by older
versions of MDS and other monitoring systems).
67
MDS user interfaces

General purpose UIs



Web browser based interface -WebMDS
Command line tools
Specialized clients

Brokers
68
WebMDS


Web-based
interface to display
monitoring
information
Easily extensible
for new data using
XSLT
69
MDS4 - Command Line


Xpath Queries to query the Index Service
To see all collected in the Index Service


To see the number of free nodes:


wsrf-query -s \
https://gcb.fiu.edu:8443/wsrf/services/DefaultIndexService
wsrf-query -s
https://gcb.fiu.edu:8443/wsrf/services/DefaultIndexService
"number(//*/glue:GLUECE//glue:ComputingElement/glue:State/@
glue:FreeCPUs)"
To see how many jobs are currently running:

wsrf-query -s
https://gcb.fiu.edu:8443/wsrf/services/DefaultIndexService
"number(//*[localname()='GLUECE']//glue:ComputingElement//glue:State/@glue:T
70
otalJobs)"
Configuring GRAM to use a
cluster monitoring system




GRAM extracts and publishes cluster information
from either Ganglia or Hawkeye
$GLOBUS_LOCATION/etc/globus_wsrf_mds_usef
ulrp/gluerp.xml
<defaultProvider> tag specifies whether to use
Ganglia or Hawkeye or none.
Uncomment appropriate example supplied in the
configfile
71
Agenda







Grid Computing
Grid Middleware - Globus
Security in Globus
Data Management
Execution Management
Monitoring
Metaschedulers - Gridway
72
Grid Meta-Scheduler




Local Schedulers is not fit for Grid environment
Meta-scheduler(s) should interact with lower-level
schedulers for scheduling decisions
Resources (Computational, Data, Network, etc.) and
Jobs are other entities, Meta-Scheduler should be
aware of and interact with
Meta-Scheduler uses existing Grid services
73
GridWay


Lightweight metascheduler on top of GT 2.4 – 4.x
Properties:






Support of GGF DRMA standard API for job submission and
management
Support for JSDL
Simple scheduling mechanisms but extensible
Interoperability between different grid infrastructures and
middlewares (Globus, EGEE, UNICORE…)
Allows job dependencies (workflow)
Supports job migration/adaptive execution (Grid- and
application-initiated)
74
GridWay Architecture
RFT
GRAM
DRMAA Library
MDS
GridWay Core
Job
pool
Execution of
jobs on LRM
Job control
operations
Request
Manager
Host
pool
Dispatch
Manager
Matchmaking,
execution and
migration
Transfer
Manager
CLI
Execution
Manager
RFT
GRAM
Information
Manager
Scheduler
Performance
Monitor
MDS
Resource
75
GridWay Modules



Request Manager
Interfaces with client commands
Dispatch Manager
Performs job scheduling
Information Manager Resource Monitoring and data
gathering


Execution Manager Executes job stages
Performance Monitor Evaluates the job performance
76
Scheduling Strategy






Dispatch manager wakes up at every scheduling
interval
Uses Resource Selector to select the host(s) to
submit the job
Resource Selector interfaces with Grid Information
Services, such as MDS
Resource Selector returns a candidate list of hosts to
submit the job by using a policy script
You can implement your own policy script, so it is
extensible
Dispatch Manager then submits the job to the
Execution Manager
77
GridWay Commands






gwd - start the daemon
gwhost - information about resources
gwps - information about jobs
gwuser - information about users
gwsubmit - submits job
gwkill - cancels a job
78