Transcript Grid, Globus Toolkit, and OGSA

Grid Computing and the
Open Grid Service Architecture
Ian Foster
Argonne National Laboratory
University of Chicago
http://www.mcs.anl.gov/~foster
2nd IEEE Intl Symp. on Network Computing & Applications, Boston, April 17, 2003
Partial Acknowledgements

3
Open Grid Services Architecture design
– Carl Kesselman, Karl Czajkowski @ USC/ISI
– Steve Tuecke @ANL
– Jeff Nick, Steve Graham, Jeff Frey @ IBM

Grid services collaborators at ANL
– Kate Keahey, Gregor von Laszewski
– Thomas Sandholm, Jarek Gawor, John Bresnahan

Globus Toolkit R&D also involves many fine
scientists & engineers at ANL, USC/ISI, and
elsewhere (see www.globus.org)

Strong links with many EU, UK, US Grid projects

Support from DOE, NASA, NSF, IBM, Microsoft
[email protected]
ARGONNE  CHICAGO
4
Overview

Grid: why and what

Evolution of Grid technology
– Open Grid Services Architecture

Future directions
– Towards lightweight VOs: dynamic trust
relationships
– Towards global knowledge communities:
virtual data and dynamic workspaces
[email protected]
ARGONNE  CHICAGO
Why the Grid?
(1) Revolution in Science

5
Pre-Internet
– Theorize &/or experiment, alone
or in small teams; publish paper

Post-Internet
– Construct and mine large databases of
observational or simulation data
– Develop simulations & analyses
– Access specialized devices remotely
– Exchange information within
distributed multidisciplinary teams
[email protected]
ARGONNE  CHICAGO
Why the Grid?
(2) Revolution in Business

6
Pre-Internet
– Central data processing facility

Post-Internet
– Enterprise computing is highly distributed,
heterogeneous, inter-enterprise (B2B)
– Business processes increasingly
computing- & data-rich
– Outsourcing becomes feasible =>
service providers of various sorts
[email protected]
ARGONNE  CHICAGO
New Opportunities
Demand New Technology
7
“Resource sharing & coordinated
problem solving in dynamic, multiinstitutional virtual organizations”
“When the network is as fast as the computer's internal
links, the machine disintegrates across the net into a set
of special purpose appliances”
[email protected]
(George Gilder)
ARGONNE  CHICAGO
8
Grid Communities & Technologies

Yesterday
– Small, static communities, primarily in science
– Focus on sharing of computing resources
– Globus Toolkit as technology base

Today
– Larger communities in science; early industry
– Focused on sharing of data and computing
– Open Grid Services Architecture emerging

Tomorrow
– Large, dynamic, diverse communities that share
a wide variety of services, resources, data
– New issues: Trust, distributed RM, knowledge
[email protected]
ARGONNE  CHICAGO
9
NSF TeraGrid

NCSA, SDSC, Argonne, Caltech

Unprecedented capability
– 13.6 trillion flop/s
– 600 terabytes of data
– 40 gigabits per second
– Accessible to thousands
of scientists working on
advanced research

www.teragrid.org
[email protected]
ARGONNE  CHICAGO
10
[email protected]
ARGONNE  CHICAGO
Data Grids for High
Energy Physics




11
Enable international
community of 1000s to
access & analyze petabytes
of data
Harness computing &
storage worldwide
Virtual data concepts:
manage programs,
data, workflow
Distributed system
management
[email protected]
ARGONNE  CHICAGO
NEESgrid Earthquake Engineering
Collaboratory
12
U.Nevada Reno
www.neesgrid.org
[email protected]
ARGONNE  CHICAGO
Grid Computing
13
Grid Computing
By M. Mitchell Waldrop
May 2002
Hook enough computers together and what do you get? A new kind of
utility that offers supercomputer processing on tap.
Is Internet history about to repeat itself?
[email protected]
ARGONNE  CHICAGO
Industrial Perspective on Grids:
A Wide Range of Applications
14
Grid Services Market Opportunity 2005
Unique by Industry with Common Characteristics
Manufacturing
Financial
Services
Energy
Derivatives
Analysis
Seismic
Analysis
Statistical
Analysis
Reservoir
Analysis
Portfolio
Risk
Analysis
Mechanical/
Electronic
Design
LS /
Bioinformatics
Other
Process
Simulation
Cancer
Research
Entertainment
Finite
Element
Analysis
Drug
Discovery
Digital
Rendering
Protein
Folding
Massive
Multi-Player
Games
Failure
Analysis
Protein
Sequencing
Streaming
Media
Web
Applications
Weather
Analysis
Code
Breaking/
Simulation
Academic
“Gridified” Infrastructure
Sources: IDC, 2000 and Bear Stearns- Internet 3.0 - 5/01 Analysis by SAI
[email protected]
ARGONNE  CHICAGO
15
Overview

Grid: why and what

Evolution of Grid technology
– Open Grid Services Architecture

Future directions
– Towards lightweight VOs: dynamic trust
relationships
– Towards global knowledge communities:
virtual data and dynamic workspaces
[email protected]
ARGONNE  CHICAGO
16
Open Grid Services Architecture

Service-oriented architecture
– Key to virtualization, discovery,
composition, local-remote transparency

Leverage industry standards
– Internet, Web services

Distributed service management
– A “component model for Web services”

A framework for the definition of
composable, interoperable services
“The Physiology of the Grid: An Open Grid Services Architecture for
Distributed Systems Integration”, Foster, Kesselman,ARGONNE
Nick, Tuecke,
2002
[email protected]
 CHICAGO
17
Web Services



XML-based distributed computing technology
Web service = a server process that exposes
typed ports to the network
Described by the Web Services Description
Language, an XML document that contains
– Type of message(s) the service understands &
types of responses & exceptions it returns
– “Methods” bound together as “port types”
– Port types bound to protocols as “ports”

A WSDL document completely defines a
service and how to access it
[email protected]
ARGONNE  CHICAGO
18
OGSA Structure

A standard substrate: the Grid service
– Standard interfaces and behaviors that
address key distributed system issues
– A refactoring and extension of the Globus
Toolkit protocol suite

… supports standard service specifications
– Resource management, databases,
workflow, security, diagnostics, etc., etc.
– Target of current & planned GGF efforts

… and arbitrary application-specific
services based on these & other definitions
[email protected]
ARGONNE  CHICAGO
19
Open Grid Services Infrastructure
Client
Introspection:
• What port types?
• What policy?
• What state?
Grid Service
Handle
handle
resolution
Grid Service
Reference
Lifetime management
• Explicit destruction
• Soft-state lifetime
GridService
(required)
Data
access
Service
data
element
Service
data
element
Other standard interfaces:
factory,
notification,
collections
Service
data
element
Implementation
Hosting environment/runtime
(“C”, J2EE, .NET, …)
[email protected]
ARGONNE  CHICAGO
20
Open Grid Services Infrastructure
GWD-R (draft-ggf-ogsi- gridservice-23)
Open Grid Services Infrastructure (OGSI)
http://www.ggf.org/ogsi-wg
Editors:
S. Tuecke, ANL
K. Czajkowski, USC/ISI
I. Foster, ANL
J. Frey, IBM
S. Graham, IBM
C. Kesselman, USC/ISI
D. Snelling, Fujitsu Labs
P. Vanderbilt, NASA
February 17, 2003
Open Grid Services Infrastructure (OGSI)
[email protected]
ARGONNE  CHICAGO
Example:
Reliable File Transfer Service
Client
Client
21
Client
Request and manage file transfer operations
File
Notf’n Policy
Grid
Service Transfer Source
Fault
Monitor
Perf.
Monitor
Query &/or
subscribe
to service data
Pending
Performance
Policy
Faults
interfaces
service
data
elements
Internal
State
Data transfer operations
[email protected]
ARGONNE  CHICAGO
Open Grid Service Architecture:
Next Steps
22
 Technical specifications
– Open Grid Services Infrastructure is complete
– Security, data access, Java binding, common
resource models, etc., etc., in the pipeline
 Implementations and compliant products
– Here: OGSA-based Globus Toolkit v3, …
– Announced: IBM, Avaki, Platform, Sun, NEC,
HP, Oracle, UD, Entropia, Insors, …, …
 Rich set of service defns & implementations
[email protected]
ARGONNE  CHICAGO
Globus Toolkit v3 (GT3)
Open Source OGSA Technology

Implements OGSI interfaces

Supports primary GT2 interfaces
23
– High degree of backward compatibility

Multiple platforms & hosting environments
– J2EE, Java, C, .NET, Python

New services
– SLA negotiation, service registry, community
authorization, data management, …

Rapidly growing adoption and contributions:
“Linux for the Grid”
[email protected]
ARGONNE  CHICAGO
24
Overview

Grid: why and what

Evolution of Grid technology
– Open Grid Services Architecture

Future directions
– Towards lightweight VOs: dynamic trust
relationships
– Towards global knowledge communities:
virtual data and dynamic workspaces
[email protected]
ARGONNE  CHICAGO
25
Future Directions

Grids are about computers, certainly
– “On-demand” access to computing, etc.
– Challenging future issues here: e.g., scale
[email protected]
ARGONNE  CHICAGO
26
CMS Event Simulation Production

Production Run on the Integration Testbed
– Simulate 1.5 million full CMS events for physics
studies: ~500 sec per event on 850 MHz processor
– 2 months continuous running across 5 testbed sites
– Managed by a single person at the US-CMS Tier 1
[email protected]
ARGONNE  CHICAGO
27
CMS Event Simulation Production

Production Run on the Integration Testbed
– Simulate 1.5 million full CMS events for physics
studies: ~500 sec per event on 850 MHz processor
– 2 months continuous running across 5 testbed sites
– Managed by a single person at the US-CMS Tier 1
[email protected]
ARGONNE  CHICAGO
28
Future Directions

Grids are about computers, certainly
– “On-demand” access to computing, etc.
– Challenging future issues here: e.g., scale

But they are ultimately about people, their
activities, and their interactions
– New interaction modalities supported by ondemand formation of lightweight VOs
– New technologies needed: e.g., trust,
security, data and knowledge integration

Convergence of interest between
“Compute” and “Collaboration” Grids?
[email protected]
ARGONNE  CHICAGO
Global Knowledge Communities
[email protected]
29
ARGONNE  CHICAGO
30
Example Issue: Trust and Security

Effective VO operation depends critically on
– Trust: can I rely on you?
– Protection mechanisms to govern actions


Suffers from VO-organization policy mismatch
Goal: collaborations no longer defined by slow
centralized mechanisms but can
– form spontaneously;
– be managed in a distributed manner; and
– be protected by an infrastructure that
maintains and enforces trust relationships
[email protected]
ARGONNE  CHICAGO
31
Grid Security Services
Requestor's
Domain
Trust
Service
Attribute
Service
Audit/
Secure-Logging
Service
Service Provider's
Domain
Authorization
Service
Authorization
Service
Privacy
Service
Trust
Service
Attribute
Service
Audit/
Secure-Logging
Service
Privacy
Service
Credential
Validation
Service
Credential
Validation
Service
Bridge/
Translation
Service
Requestor
Application
WS-Stub
Secure Conversation
WS-Stub
Credential
Validation
Service
Service
Provider
Application
Credential
Validation
Service
Authorization
Service
Authorization
Service
Attribute
Service
Attribute
Service
Trust
Service
Trust
Service
VO
Domain
[email protected]
ARGONNE  CHICAGO
Understanding and Enhancing
VO Trust and Security
Usability
analysis
Community
Social network analysis
Other analyses
trust ( Tr, Te, As, L ) <- Cs;
recommend ( Rr, Re, As, L ) <- Cs;
Trust
Monitoring for
reputation,
compliance,
intrusion
detection, etc.
Establishment,
enhancement,
maintenance,
verification
Workflow analysis
Risk analysis
allowed (S, O, A, C)
Policy
Factoring wrt environment
Mechanism
[email protected]
32
Feasibility
analysis wrt
cost, legality,
etc.
VOTA, PKI, VPN, etc.
ARGONNE  CHICAGO
Virtual Data
for Collaborative Science

33
Much collaboration is concerned with the
development & use of knowledge, whether
– Programs for data analysis and generation
– Computations involving those programs
– Metadata concerning data, programs,
computations—and their interrelationships

In a distributed, heterogeneous, fractal (?)
environment with widely varying
– Data and analysis program formats
– Degrees of formality and scale
– Scientific goals and sharing policies
[email protected]
ARGONNE  CHICAGO
Sloan Digital Sky Survey
Production System
[email protected]
34
ARGONNE  CHICAGO
Virtual Data Concept

35
Capture and manage information about
relationships among
– Data (of widely varying representations)
– Programs (& their execution needs)
– Computations (& execution environments)

Apply this information to, e.g.
– Discovery: Data and program discovery
– Workflow: Structured paradigm for organizing,
locating, specifying, & requesting data
– Explanation: provenance
– Planning and scheduling
– Other uses we haven’t thought of
[email protected]
ARGONNE  CHICAGO
36
“I’ve come across some
interesting data, but I need
to understand the nature of
the corrections applied
when it was constructed
before I can trust it for my
purposes.”
Motivations
Data
created-by
Transformation
execution-of
“I want to search an astronomical
database for galaxies with certain
characteristics. If a program that
performs this analysis exists, I
won’t have to write one from
scratch.”
[email protected]
“I’ve detected a calibration
error in an instrument and
want to know which derived
data to recompute.”
consumed-by/
generated-by
Derivation
“I want to apply an
astronomical analysis
program to millions of
objects. If the results
already exist, I’ll save
weeks
of computation.”
ARGONNE
 CHICAGO
Example:
Sloan Galaxy Cluster Analysis
37
DAG
Sloan Data
100000
Galaxy cluster
size distribution
Number of Clusters
10000
1000
100
10
1
[email protected]
1
10
Number of Galaxies
100
Jim Annis, Steve Kent, Vijay
Sehkri, Fermilab; Michael
ARGONNE
 CHICAGO
Milligan,
Yong Zhao,
Chicago
38
Integrating Provenance Data
Group Index
Personal
VDS
DV
Personal
Index
Personal
Index
DS
TR
Collaborationlevel
index
DV
DV
TR
DS
TR
TR
Collaboration
VDS
DV
Group VDS
DS
Personal
Index
DV
DV
Personal
VDS
Collaboration-wide
index
[email protected]
ARGONNE  CHICAGO
39
Summary

Yesterday
– Small, static communities, primarily in science
– Focus on sharing of computing resources
– Globus Toolkit as technology base

Today
– Larger communities in science; early industry
– Focused on sharing of data and computing
– Open Grid Services Architecture emerging

Tomorrow
– Large, dynamic, diverse communities that share
a wide variety of services, resources, data
– New issues: Trust, distributed RM, knowledge
[email protected]
ARGONNE  CHICAGO
40
For More Information

The Globus Project™
– www.globus.org

Technical articles
– www.mcs.anl.gov/~foster

Open Grid Services Arch.
– www.globus.org/ogsa

Chimera
– www.griphyn.org/chimera

Global Grid Forum
– www.gridforum.org
[email protected]
ARGONNE  CHICAGO