Transcript IBM and GRID Computing

Grid Computing
and Scaling Up the Internet
Ian Foster
Argonne National Laboratory
University of Chicago
Globus Project
Brian E Carpenter
IBM and 6NET
IPv6 Forum, San Diego, June 27, 2003
2
Why Grids and IPv6?
• Grid computing represents a fundamental
shift in how we approach distributed
computing, like the fundamental shift in
information access introduced by the Web
• IPv6 represents a major step function in the
Internet’s ability to scale, like the
introduction of IPv4 twenty years ago
• Inevitably there is synergy between these
two game changers
• Let’s share a common goal of reaching 10
billion Internet nodes
[email protected]
ARGONNE  CHICAGO
3
Topics
•
•
•
•
Brief introduction to Grid computing
Why Grids and IPv6 need each other
Practical aspects of enabling GT3 for IPv6
Future directions and summary
[email protected]
ARGONNE  CHICAGO
4
The Grid Is …
a) A collaboration & resource sharing
infrastructure with origins in the sciences
b) A distributed service integration and
management technology
c) A disruptive technology that enables a
virtualized, collaborative, distributed world
d) An open source technology & community
e) A marketing slogan
f) All of the above
[email protected]
ARGONNE  CHICAGO
Quality, economies of scale
The (Power) Grid:
On-Demand Access to Electricity
5
Time
[email protected]
ARGONNE  CHICAGO
6
By Analogy, A Computing Grid
• Decouple production and consumption
–
–
–
–
Enable on-demand access
Achieve economies of scale
Enhance consumer flexibility
Enable new devices
• On a variety of scales
–
–
–
–
Department
Campus
Enterprise
Internet
[email protected]
ARGONNE  CHICAGO
7
Not Exactly a New Idea …
• “The time-sharing computer system can
unite a group of investigators …. one can
conceive of such a facility as an …
intellectual public utility.”
– Fernando Corbato and Robert Fano , 1966
• “We will perhaps see the spread of
‘computer utilities’, which, like present
electric and telephone utilities, will service
individual homes and offices across the
country.”
– Len Kleinrock, 1967
[email protected]
ARGONNE  CHICAGO
But Things are Different Now …
[email protected]
8
ARGONNE  CHICAGO
But Wait A Minute—Computing
isn’t Really Like Electricity!
9
• I import electricity but must export data
• “Computing” is not interchangeable but
highly heterogeneous
– Computers, data, sensors, services, …
• So the story is more complicated
• But more significantly, the sum can be
greater than the parts
– Real opportunity: Construct new capabilities
dynamically from distributed services
– Virtualization & distributed service mgmt
[email protected]
ARGONNE  CHICAGO
Virtualization &
Distributed Service Management
Less capable, integrated
Less connected
User service locus
Device Continuum
Distributed service
management
Dynamic, secure
service discovery
& composition
[email protected]
10
Larger, more integrated
More connected
Dynamically provisioned
Resource &
service
aggregation
Delivery of virtualized
services with QoS
guarantees
ARGONNE  CHICAGO
Why the Grid?
Origins: Revolution in Science
11
• Pre-Internet
– Theorize &/or experiment, alone
or in small teams; publish paper
• Post-Internet
– Construct and mine large databases of
observational or simulation data
– Develop simulations & analyses
– Access specialized devices remotely
– Exchange information within
distributed multidisciplinary teams
[email protected]
ARGONNE  CHICAGO
12
Example
Science Grids
Edinburgh
Glasgow
DL
Belfast
Newcastle
Manchester
Cambridge
Oxford
Cardiff
RAL
Hinxton
London
Soton
Tier0/1 facility
Tier2 facility
Tier3 facility
10 Gbps link
2.5 Gbps link
622 Mbps link
Other link
[email protected]
ARGONNE  CHICAGO
The Grid/eScience World:
Status
13
• Dozens of major Grid projects in scientific &
technical computing/research & education
– Deployment, application, technology
– www.mcs.anl.gov/~foster/grid-projects
• Globus Toolkit™ broadly adopted as de facto
standard for major protocols & services
• Global Grid Forum a significant force for
community building and standardization
– GGF9: Seattle, June 2003, 800 people
– www.ggf.org; 200+ organizations; Boeing,
Merck, Ford, J&J, IBM, Platform, …
[email protected]
ARGONNE  CHICAGO
Why the Grid?
(2) Revolution in Business
14
• Pre-Internet
– Central data processing facility
• Post-Internet
– Enterprise computing is highly
distributed, heterogeneous,
loosely coupled,
inter-enterprise (B2B)
– Business processes increasingly
computing- & data-rich
– Outsourcing becomes feasible =>
on-demand service providers of
various sorts
[email protected]
ARGONNE  CHICAGO
Common
eScience/eBusiness Vision
15
• Link dynamically acquired resources
– From collaborators, customers, eUtilities, …
(members of evolving “virtual organization”)
• Into a “virtual computing system”
– Dynamic, multi-faceted system spanning
institutions and industries
– Loose coupling of heterogeneous systems
– Configured on demand to meet instantaneous
needs, for:
• Multi-faceted QoS for demanding workloads
– Security, performance, reliability, …
[email protected]
ARGONNE  CHICAGO
16
Increased functionality,
standardization
The Emergence of
Open Grid Standards
Managed shared
virtual systems
Computer science research
Open Grid
Services Arch
Web services, etc.
Internet
standards
Custom
solutions
1990
[email protected]
Real standards
Multiple implementations
Globus Toolkit
Defacto standard
Single implementation
1995
2000
2005
2010
ARGONNE  CHICAGO
17
Open Grid Services Architecture
• Service-oriented architecture
– Key to virtualization, discovery, composition,
local-remote transparency
• Leverage industry standards
– Internet, Web services
• Distributed service management
– A “component model for Web services”
• A framework for the definition of
composable, interoperable services
“The Physiology of the Grid: An Open Grid Services Architecture for
Distributed Systems Integration”, Foster, Kesselman,ARGONNE
Nick, Tuecke,
2002
[email protected]
 CHICAGO
18
Open Grid Services Infrastructure
Client
Introspection:
• What port types?
• What policy?
• What state?
Grid Service
Handle
handle
resolution
Grid Service
Reference
Lifetime management
• Explicit destruction
• Soft-state lifetime
GridService
(required)
Data
access
Service
data
element
Service
data
element
Other standard interfaces:
factory,
notification,
collections
Service
data
element
Implementation
Hosting environment/runtime
(“C”, J2EE, .NET, …)
[email protected]
ARGONNE  CHICAGO
19
Open Grid Services Infrastructure
Client
Introspection:
Lifetime management
GWD-R• (draft-ggf-ogsigridservice-29)
Editors:
What port types?
• Explicit destruction
• What
policy?
Open Grid
Services
Infrastructure
(OGSI)
S. Tuecke, ANL
• Soft-state lifetime
• What state?
http://www.ggf.org/ogsi-wg
K. Czajkowski, USC/ISI
Data
GridService
I. Foster, ANL
Other
standard interfaces:
(required)
access
J. Frey, IBM
factory,
Grid Service
S. Graham, IBM
notification,
Handle
C. Kesselman, USC/ISI
collections
Service
Service
Service
handle
resolution
Grid Service
Reference
data
element
data
element
T.data
McGuire, IBM
element
T. Sandholm, ANL
D. Snelling, Fujitsu Labs
ImplementationP. Vanderbilt, NASA
April 5, 2003
Open Grid Services
(OGSI) Version 1.0
Hosting Infrastructure
environment/runtime
(“C”, J2EE, .NET, …)
[email protected]
ARGONNE  CHICAGO
Resource
allocation
OGSA Interactions
Create
Service
Authentication
& Authorization
are applied to
all requests
Service
instances
Service
factory
Grid Service
Handle
Service data
Keep-alives
Notifications
Service
invocation
Regist
er
Service
Interactions standardized using WSDL and SOAP
Service
requestor
(e.g. user
application)
Service
discovery
Service
registry
21
Open Grid Services Architecture
OGSA services: registry,
authorization, monitoring, data
access, management, etc., etc.
Open Grid Services Infrastructure
Web Services
Host. Env.
Other
OGSA schemas
schemas
More specialized &
domain-specific
services
& Protocol Bindings
HostingEnvironment
Environment
Hosting
Transport
Protocol
GWD-R (draft-ggf-ogsa-platform-3)
Open Grid Services Architecture Platform
http://www.ggf.org/ogsa-wg
[email protected]



Data access and
integration
Security
Agreement and SLA
negotiation

Manageability

…
Editors:
I. Foster, Argonne & U.Chicago
D. Gannon,
Indiana U.
ARGONNE
 CHICAGO
Globus Toolkit v3 (GT3)
Open Source OGSA Technology
22
• Implements OGSI interfaces
• Supports primary GT2 interfaces
– High degree of backward compatibility
• Multiple platforms & hosting environments
– J2EE, Java, C, .NET, Python
• New services
– SLA negotiation, service registry, community
authorization, data management, …
• Rapidly growing adoption and contributions:
“Linux for the Grid”
[email protected]
ARGONNE  CHICAGO
23
Topics
•
•
•
•
•
General Introduction
Brief introduction to Grid computing
Why Grids and IPv6 need each other
Practical aspects of enabling GT3 for IPv6
Future directions and summary
[email protected]
ARGONNE  CHICAGO
24
Grid Deployment Options
A function of business need, technology and
organizational flexibility
Inter-Grids
Extra-Grids
Intra-Grids
Cactus
(SF)
Express
Project
Grid
Grid
NTG
Grid
VPN
NAS/SAN
NAS/SAN
NAS/SAN
Fin.
Services
[email protected]
MFG
ARGONNE  CHICAGO
25
Physical
Organisation
Virtual Organisation
Physical
Organisation
Virtual Organization
View of Deployment
Virtual Organisation
Physical
Organisation
[email protected]
Virtual
Organisation
Physical
Organisation
ARGONNE  CHICAGO
Global Knowledge Communities
[email protected]
26
ARGONNE  CHICAGO
Virtual Organizations Look Like
Dynamic Mergers & Acquisitions
27
• The effect of a Grid VO on networks is like a
temporary partial merger of the organizations
• Merging two networks is painful today
– “Private” IPv4 address space becomes
ambiguous
– Worst case: forced to renumber both networks
• Temporary partial mergers of an arbitrary
number of IPv4 networks is unthinkable
• IPv4-based Grids are forced to rely on HTTP
proxying between organizations: inefficient,
and cannot exploit network-level security
[email protected]
ARGONNE  CHICAGO
28
Overlapping Virtual Organizations
VO
VO
VO
• Any system can be in any number of VOs
with any number of other systems
– Needs uniform address space to avoid proxies
& allow IPSEC
– Addressing ambiguities unacceptable
– Security boundaries ≠ organization boundaries
– Not achievable at massive scale with IPv4
[email protected]
ARGONNE  CHICAGO
Critical Advantages of IPv6
for OGSA
29
• Potential for massive scaling
• Uniform global address space eliminates the
problem of ambiguous “private” addresses
and network address translation
– Wasteful proxies can be avoided
– Network level security can be used
• Autoconfiguration is a big plus for
infrastructure configuration
[email protected]
ARGONNE  CHICAGO
There’s No Such Thing as an
IPv6 Killer App, But ...
30
• It would be nice to find the killer app that
only works on IPv6
• OGSA won’t be that, but there is a good
chance that it will be the first major
middleware suite to be IPv6-capable out of
the box almost from Day One
• The IPv6 community should make the most
of it
[email protected]
ARGONNE  CHICAGO
31
Topics
•
•
•
•
Brief introduction to Grid computing
Why Grids and IPv6 need each other
Practical aspects of enabling GT3 for IPv6
Future directions and summary
[email protected]
ARGONNE  CHICAGO
32
Testing, Testing
• 6NET is a three-year EU-funded project to
demonstrate that continued growth of the
Internet can be met using IPv6.
• Includes a work package for IPv6 Middleware
and User Application Trials (led by IBM)
• Globus is the subject of a trial (lead site UCL)
– Target is Globus Toolkit 3, i.e. OGSA
– GT3 (OGSA) alpha code is now available and
being tested on IPv6/Linux at UCL
– Credits: Sheng Jiang, Piers O’Hanlon, Peter
Kirstein
[email protected]
ARGONNE  CHICAGO
33
Status (Evolving Daily)
• Background: GT3 runs mainly over Java; Java2
Development Kit 1.4 supports IPv6, although
GT3 alpha ships with JDK 1.3
• In principle, all GT3 Java components will “just
work” with IPv6 via switch to JDK 1.4
– In practice, it is not so simple
– Jim Bound of HP is also looking into this
• UCL has begun testing with JDK 1.4.
– GT3 Master Host Environment listens on both
IPv6 and IPv4. Simple test from IPv6 completes,
but some IPv4 packets are observed
• Exact environment is GT3 alpha code with Java
SDK 1.4.1. on Redhat Linux 7.3 and 8.0
[email protected]
ARGONNE  CHICAGO
34
Status (update May 6th)
• Tested the postgresql IPv6 patch; found one
bug there: after removing IPv6 items from
the configuration file, the IPv6 address was
still enabled
• Working to deploy GT3 core on Apache
Tomcat Java servlet container (IPv6
enabled)
• Waiting for IBM Websphere to do the same.
• Starting to port the OGSA stand-alone web
container to be IPv6-enabled
– Sheng Jiang
[email protected]
ARGONNE  CHICAGO
35
GridFTP (evolving daily)
• GridFTP (striped FTP) as shipped with GT3
alpha is C code that does not support IPv6
sockets.
• Globus is developing a new generic I/O
module called XIO that does support IPv6
sockets
• GridFTP is being rewritten, still in C, to
exploit XIO
• Any other components of GT3 that remain
in C can also support IPv6 via XIO
[email protected]
ARGONNE  CHICAGO
36
Further Plans (Evolving Daily)
• Plan is to make more extensive tests with
about 10 nodes
– Issues with IPv6 will be reported into the
Globus bug-tracking system
– Good relations established between 6NET and
Globus teams
• Also need to consider what is required to
operate GT3 in the cases of
– IPv6 only
– IPv6 and IPv4 coexistence
• Final goal is a realistic systematic trial
between 6NET sites
[email protected]
ARGONNE  CHICAGO
37
Topics
•
•
•
•
•
General Introduction
Brief introduction to Grid computing
Why Grids and IPv6 need each other
Practical aspects of enabling GT3 for IPv6
Future directions and summary
[email protected]
ARGONNE  CHICAGO
38
Grid Past, Present, Future
• Past
– Origins and broad adoption in eScience,
fueled by open source Globus Toolkit
• Present
– Rapidly growing commercial adoption
– Open Grid Services Architecture (OGSA)
• Future
– Key enabler of new applications & industries
based on resource virtualization and
distributed service integration
[email protected]
ARGONNE  CHICAGO
39
Future Directions
• GT3 will become fully functional for IPv6
• Global Grid Forum must chase down any
IPv4 dependencies in its standards
• Grid computing will become a key enabler of
new applications based on resource
virtualization and loosely coupled distributed
service integration
• IPv6 will enable Grid Virtual Organizations to
span existing network boundaries smoothly
and securely
[email protected]
ARGONNE  CHICAGO
40
Summary
• Grid computing is the new model for sharing
networked IT resources efficiently &
securely.
– Transforming the Internet into a computing
platform for e-business on demand
• The key toolkit is the Globus Toolkit open
source package based on the Open Grid
Services Architecture
• A marriage between OGSA and IPv6 is the
key to massive scaling in a fully connected
but secure network environment.
[email protected]
ARGONNE  CHICAGO
41
Pointers
•
•
•
•
www.globus.org
www.ggf.org
www.ipv6forum.org
www.6net.org
[email protected]
[email protected]
[email protected]
ARGONNE  CHICAGO