The Globus Toolkit: Building Grid Applications
Download
Report
Transcript The Globus Toolkit: Building Grid Applications
The Globus Grid
Programming Toolkit:
A User-level Tutorial
The Globus Project Team
ANL and USC/ISI
http://www.globus.org
Abstract
This tutorial is a practical introduction to programming for high-performance
distributed computing systems, or "computational grids," and the capabilities of the
Globus grid toolkit.
Emerging high-performance networks promise to enable a wide range of emerging
application concepts such as remote computing, distributed supercomputing, teleimmersion, smart instruments, and data mining. However, the development and use of
such applications is in practice very difficult and time consuming, because of the need
to deal with complex and highly heterogeneous systems. The Globus grid
programming toolkit is designed to help application developers and tool builders
overcome these obstacles to the construction of "grid-enabled” scientific and
engineering applications. It does this by providing a set of standard services for
authentication, resource location, resource allocation, configuration, communication,
file access, fault detection, and executable management. These services can be
incorporated into applications and/or programming tools in a "mix-and-match" fashion
to provide access to needed capabilities.
Our goal in this tutorial is both to introduce the capabilities of the Globus toolkit and to
show attendees how Globus services can be applied in specific applications. Hence,
the tutorial covers a mixture of grid programming principles and detailed case studies
of real applications.
Introduction 2
Tutorial Goals
Provide an introduction
To the structure of the Globus computational
grid
To the capabilities of the Globus toolkit
To pragmatic issues associated with using
the toolkit
Enable attendees
To start building & using Globus applications
To utilize Globus services
Introduction 3
Overview
Introduction to computational grids
High-level overview of the Globus toolkit
Four components:
Security and remote process creation
Running programs across multiple resources
Information services
Dynamic configuration and resource management
Case studies
Other Globus services, and future directions
Globus installation & administration
Introduction 4
Why “The Grid”?
New applications based on high-speed
coupling of people, computers, databases,
instruments, etc.
Computer-enhanced instruments
Collaborative engineering
Browsing of remote datasets
Use of remote software
Data-intensive computing
Very large-scale simulation
Large-scale parameter studies
Introduction 5
E.g.: Computer-Enhanced
Instruments for Microtomography
50 Mb/s -> 5 Gb/s -> 100 Gb/s
APS
beamline
@ Argonne
“100 Gflop/sec,
50 Mb/sec,
5 minutes;
rendering,
10 GB storage”
5 Mb/s -> 1 Gb/s -> 10 Gb/s
Coupling with
supercomputers
Interactive use of
beamline
Collaboration on
results
Parameter studies
for experiment
planning
Coupling with mass
store systems
Los Angeles Chicago
Introduction 6
E.g.: Tele-immersion
“5 Gflop/sec,
flowspecs,
design db”
Multiple access modalities
Multiple flows
Control
Simulation
Text
Tracking
Video
Haptics
Audio
Rendering
Leigh et al., UofI, Electronic Visualization Lab.
Database
Introduction 7
SF-Express: Distributed
Interactive Simulation
NCSA
Origin
Argonne
SP
Caltech
Exemplar
Maui
SP
“200 GB memory,
100 BIPs”
P. Messina et al., Caltech
Issues:
Resource discovery, scheduling
Configuration
Multiple comm methods
Message passing (MPI)
Scalability
Fault tolerance
Introduction 8
The Grid
“Dependable, consistent,
pervasive access to
[high-end] resources”
Dependable: Can provide
performance and functionality
guarantees
Consistent: Uniform interfaces
to a wide variety of resources
Pervasive: Ability to “plug in”
from anywhere
Introduction 9
Evolution of a Concept
Metacomputing: late 80s
Gigabit testbeds: early 90s
Focus on distributed computation
Research, primarily on networking
I-WAY: 1995
Demonstration of application feasibility
PACIs (National Technology Grid): 1998
NASA Information Power Grid: 1999
ASCI DISCOM: 1999; SSI: 2000?
Introduction 10
I-WAY
The Alliance National Technology Grid
National and International Grid Testbeds
NASA’s Information Power Grid
Introduction 11
Technical Challenges
Complex application structures, combining
aspects of parallel, multimedia, distributed,
collaborative computing
Dynamic varying resource characteristics,
in time and space
Need for high & guaranteed “end-to-end”
performance, despite heterogeneity and
lack of global control
Interdomain issues of security, policy,
payment
Introduction 12
Issues
•Authenticate once
•Specify simulation
(code, resources, etc.)
•Locate resources
•Negotiate authorization, Domain 1
acceptable use, etc.
•Acquire resources
Domain 2
•Initiate computation
•Steer computation
•Access remote datasets
•Collaborate on results
•Account for usage
Introduction 13
Architectural Approaches
Distributed systems: DCE, CORBA, Jini, etc.
Rich functionality eases app. development
Complexity hinders deployment
especially in absence of global control
Performance difficulties
Internet Protocol, Web tools
Simple protocols facilitate deployment
Missing functionality hinders app. development
Performance difficulties
Introduction 14
Standards & Commodity Tech
Where appropriate, exploit standards and
commodity technology in core infrastructure
LDAP, SSL, X.509, GSS-API, GAA-API, http,
ftp, XML, etc.
Provides leverage
Interface with other common standards
CORBA, Java/Jini, DCOM, Web, etc
While our core infrastructure may not be built
on one of these distributed architectures, we
must cleanly interface with them
Introduction 15
The Globus Project
Basic research in grid-related technologies
Development of Globus toolkit
Core services for grid-enabled tools & applns
Construction of large grid testbed: GUSTO
Resource management, QoS, networking,
storage, security, adaptation, policy, etc.
Largest grid testbed in terms of sites & apps
Application experiments
Tele-immersion, distributed computing, etc.
Introduction 16
Globus Approach
A toolkit and collection of services
addressing key technical problems
Bag of services model
Not a vertically integrated solution
Inter-domain issues, rather than clustering
Integration of intra-domain solutions
Distinguish between local and global
services
“IP hourglass” model
Introduction 17
Technical Focus & Approach
Information-rich environment
Enable incremental development of gridenabled tools and applications
Basis for configuration and adaptation
Support many programming models, tools,
applications
Deploy toolkit on national-scale testbed to
allow large-scale applications
Evolve in response to user requirements
Introduction 18
Globus Approach
Focus on architecture issues
Propose set of core services
as basic infrastructure
Applications
Diverse global services
Use to construct high-level,
domain-specific solutions
Design principles
Keep participation cost low
Enable local control
Support for adaptation
Core Globus
services
Local OS
Introduction 19
Layered Architecture
Applications
High-level Services and Tools
GlobusView
DUROC
Nexus
Gloperf
MPI
MPI-IO
CC++
Testbed Status
Nimrod/G
globusrun
Core Services
Metacomputing
Directory
Service
Condor
MPI
LSF
Easy
NQE
Globus
Security
Interface
Local
Services
GRAM
Heartbeat
Monitor
AIX
GASS
TCP
UDP
Irix
Solaris
Introduction 20
Core Globus Services
Communication infrastructure (Nexus, IO)
Information services (MDS)
Network performance monitoring (Gloperf)
Process monitoring (HBM)
Remote file and executable management (GASS
and GEM)
Resource management (GRAM)
Security (GSI)
Introduction 21
Sample of High-Level Services I
Communication & I/O libraries
Parallel languages
CC++, HPC++
Collaborative environments
MPICH, PAWS, RIO (MPI-IO), PPFS, MOL
CAVERNsoft, ManyWorlds
Others
MetaNEOS, NetSolve, LSA, AutoPilot,
WebFlow
Introduction 22
Sample High-Level Services II
Resource brokers and co-allocators
DUROC: co-allocation of multiple systems
Nimrod: high-throughput computing
Graphical system status display elements
GlobusView
MDS Browsers
Health & Status Monitors (HBM)
Network Monitors (Gloperf)
Introduction 23
“GUSTO”
Globus Ubiquitous
Supercomputing Testbed Organization
A collection of organizations committed to
creating a persistent computational grid
infrastructure
As of November 1998, 70 organizations in
3 continents and 8 countries
Introduction 24
16 sites, 330 computers, 3600 nodes, 2 Teraflop/s, 10 application partners
Introduction 25
GUSTO Testbed During SC’97
Introduction 26
GUSTO Computational Grid
Testbed: November 1998
Introduction 27
Where We Are (November 1998)
New results in security, resource
management, tools, fault detection, etc.
Globus v1.0 completed
All core services complete, relatively
robust, and documented
Available on most Unix platforms
Many tool projects are leveraging this
considerable investment in infrastructure
Interesting applications are emerging,
although mostly still in “demo” mode
Introduction 28
Where We Are (June 1999)
New results in QoS, security, resource
management, data management, tools, etc.
Globus v1.1 nearing completion
Available on most Unix platforms and Win32
Many tool projects are leveraging this
considerable investment in infrastructure
Documentation and deployment underway at
NCSA and NASA IPG
Always looking for interesting applications
Introduction 29
Changes from 1.0 to 1.1
Tutorial changes for 1.1 are denoted by
Name changes from Globus to Grid
Security and Information Service adopted as core
Grid infrastructure by several organizations
Globus Security Infrastructure -> Grid Security Infrastructure
Metacomputing Directory Service -> Grid Information Service
Affects naming of APIs and tools
Numerous small API fixes, additions, changes
Cleanup of programs/tools
A few new modules (I/O, error objects)
Introduction 30
Example Application Projects
Computed microtomography (ANL, ISI)
Hydrology (ISI, UMD, UT; also NCSA, Wisc.)
Interactive modeling and data analysis
Collaborative engineering (“tele-immersion”)
Real-time, collaborative analysis of data from
X-Ray source (and electron microscope)
CAVERNsoft @ EVL, Metro @ ANL
X-Ray crystallography (ANL, SUNY)
High-throughput computing for Shake ‘n Bake
Introduction 31
Example Application Expts (contd)
Distributed interactive simulation (CIT, ISI)
Remote visualization and steering for
astrophysics
Record-setting SF-Express simulation
Including trans-Atlantic experiments
Data-intensive computing experiments (with
LBNL and SLAC: “Clipper” project)
Introduction 32
For More Information on Globus
http://www.globus.org
Papers on all components
Tutorial and documents
Software
Application descriptions
Introduction 33
The Grid:
Blueprint for a New Computing Infrastructure
I. Foster, C. Kesselman (Eds), Morgan Kaufmann, 1999
Available July 1998;
ISBN 1-55860-475-8
22 chapters by expert
authors including Andrew
Chien, Jack Dongarra, Tom
DeFanti, Andrew
Grimshaw, Roch Guerin,
Ken Kennedy, Paul
Messina, Cliff Neuman, Jon
Postel, Larry Smarr, Rick
Stevens, and many others
“A source book for the history
of the future” -- Vint Cerf
http://www.mkp.com/grids
Introduction 34
Tutorial Approach
Four sections, each illustrates a basic
Globus technique
Laboratory material is available to allow
practice with the use of each technique
See http://www.globus.org/tutorial
Introduction 35