Transcript Intel GPE

GPE4UNICORE
Grid Programming
Environment for
UNICORE
Ralf Ratering,
Hans–Christian Hoppe
Intel Software&Solutions
Group
Software
&Solutions
group
Copyright © 2007 Intel Corporation
What is
GPE good
for?
 GPE turns a collection of distributed
computers/data centers into a Grid
What does
GPE provide?
?
Copyright © 2007 Intel Corporation
 A full Grid Software stack
including graphical user interfaces
and programming libraries
How about
other Grid
Software?
Why does
Intel develop
GPE?
 GPE services run in the Globus
Toolkit 4 container
 GPE client framework works
with UNICORE6 and ChinaGrid
 GPE is an Open Source
technology demonstrator for
customers and ISVs
 Influence standards
 Drive new technologies into
market
GPE Current Status
• GPE included in UNICORE6 alpha6
release at www.unicore.eu/download/unicore6
–
–
–
–
Application Client and GridBeans
Expert Client
BPEL workflow service
UNICORE-specific source code available
from CVS
• Main GPE development under
http://gpe4gtk.sourceforge.net
– Complete GPE sources available under BSD license
– Server implementation uses Globus Toolkit 4
container
Copyright © 2007 Intel Corporation
GPE Ecosystem
GPE4GTK as standalone solution
– Intel software testing
– Proof of concepts at customers
– Research and Education
GPE user interfaces and GridBeans
– UNICORE 6
– China Education Grid (Ministry of Education)
– South Korea (K-Grid)
Support for virtual machine technology
– Cooperation with Globus team
– Cooperation with CERN
– Strong interest in industry
Copyright © 2007 Intel Corporation
Why do we engage in UNICORE?
• Major Grid platform in European HPC
community
• UNICORE is great for “HPC on demand”
• UNICORE6 implementation will have
great performance and scalability
• Join forces in standards bodies
Copyright © 2007 Intel Corporation
GPE Components
GridBeans
GridBeans
GridBeans
Application
Client
Portal
Client
Expert
Client
Admin
Client
File
Manager
GPE Client API
Service
Registry
BPEL
Workflow
Service
Resource
Broker
OS
Repository
Service
Service
Registry
BPEL
Workflow
Service
Resource
Broker
Atomic Services
Atomic Services
Globus Toolkit 4 Container
UNICORE6 container
TSI
TSI
Copyright © 2007 Intel Corporation
Add target
system
to Grid
Use target
system
Manage
jobs
Target
System
Factory
Target
System
Service
Job
Management
Service
Storage
Management
Service
File
Transfer
Service
Target
System
Resource
Job
Resource
Storage
Resource
File
Transfer
Resource
Properties
Resources
Services
Operations
Atomic Services: Standard Grid
Interfaces using WSRF
Hardware and
software properties,
workload, etc.
Copyright © 2007 Intel Corporation
Job status,
description,
owner, etc.
Access
storages
Available
file space,
mount point, etc.
Manage
file
transfers
Transferred bytes,
used port numbers,
etc.
How do GridBeans work?
Expert Client
Apps Client
Applications
Application
GridBean
Portal Client
App1
GB
App2
GB
App3
GB
Application
GridBean
download
Services
GridBean
Service
• GridBeans encapsulate application logic
• GridBeans provide Swing interfaces for Java clients
and JSR-168 portlets for web portals
• GridBeans create JSDL descriptions from user input
Copyright © 2007 Intel Corporation
GPE: Running a Workflow
send job description
including workflow
Applications
Implements
Target System
Service Interface
Services
Other web
services
BPEL Workflow
Service
orchestrate
Atomic Services
Utility
...
Data Center 1
Copyright © 2007 Intel Corporation
Resource
Broker
Submit JSDL to
target system service
Retrieve the job EPR from the
SubmitResponse through
jobManagement partnerlink
A GPEBPEL
Workflow
Wait until
the job
status is
READY or
FAILED
Start the job: Start request is
stored in startJobRequest variable
Wait until job
status is
SUCCESSFUL
or FAILED
Copyright © 2007 Intel Corporation
<invoke inputVariable="submitJobRequest"
operation="Submit"
outputVariable="submitJobResponse"
partnerLink="targetSystem"
portType="tss:TargetSystem" />
JSDL
<assign>
<copy>
<from part="SubmitResponse"
query="/tss:SubmitResponse/tss:JobReference"
variable="submitJobResponse" />
<to partnerLink="jobManagement" />
</copy>
</assign>
<while condition="bpws:getVariableData('getJob1StateResponse',
'GetResourcePropertyResponse')/child::text()[1]!='READY'
and bpws:getVariableData('getJob1StateResponse',
'GetResourcePropertyResponse')/child::text()[1]!='FAILED'">
<invoke inputVariable="getJobStateRequest"
operation="GetResourceProperty"
outputVariable="getJobStateResponse"
partnerLink="jobManagement“
portType="u-rp:ResourcePropertiesPortType" />
</while>
<invoke inputVariable="startJobRequest"
operation="Start"
outputVariable="startJobResponse"
partnerLink="jobManagement"
portType="jms:JobManagement" />
<while condition="bpws:getVariableData('getJobStateResponse',
'GetResourcePropertyResponse')/child::text()[1]!=
'SUCCESSFUL' and
bpws:getVariableData('getJobStateResponse',
'GetResourcePropertyResponse')/child::text()[1]!='FAILED'">
<invoke inputVariable="getJobStateRequest"
operation="GetResourceProperty"
outputVariable="getJobStateResponse"
partnerLink="jobManagement"
portType="u-rp:ResourcePropertiesPortType" />
</while>
Implementing and Influencing
Standards with GPE
• Open Grid Forum (OGF)
– JSDL (Job Submission Description Language)
• GPE-BPEL extension
– OGSA (Open Grid Services Architecture)
• BES (Basic Execution Services)
• DMI (Data Movement Interface)
– ByteIO (File Transfer and Streaming)
• OASIS
–
–
–
–
WSRF (Web Services Resource Framework)
WSN (Web Services Notification)
WSS (Web Services Security)
WSBPEL (WS-Business Process Execution Language)
• W3C
– WS-Addressing, SOAP, WSDL, XML, etc.
Copyright © 2007 Intel Corporation
Expert Client
DEMO
Copyright © 2007 Intel Corporation
Outlook: Using Virtual Machine
Technology on the Grid
• Reliability: Partitions remain unaffected if other partitions
crash
• Security: Protect sensitive user data in VM instance
• Flexibility: VMs can checkpoint and migrate during run-time
– I want to move to a faster system, when it becomes available
– My job has high priority, please free the fastest machine for me
• Configurability: Dynamically deployed OS images and
applications on user request
– I need two cores on a fast CPU with 1Gb of memory and RedHat Linux 9
• Management: Maintenance of systems, software and users in
centralized services
Copyright © 2007 Intel Corporation
GPE Virtualization
• Globus virtual workspaces integrated in GPE
– Collaboration with ANL team led by Kate Keahey
– Virtual target system factory invokes Globus
workspace service to create XEN virtual machines
– OS Images are fetched from GPE OS Image
Repository Service
– Available since GPE 1.3
• Create show cases with CERN in OpenLab
collaboration
– Use virtual machines for gLite testing
– Implement prototypes for virtual machines hosting
gLite computing elements
• Create show cases with industrial customers
Copyright © 2007 Intel Corporation
GPE Plans for 2007
• Further exploit Intel technology
– Use virtualization, management and security capabilities
on new Intel platforms
– Leverage virtualization technology in the Grid
• Move towards Enterprise SOA
– Run proof of concepts at Intel customers and partners
– Identify concrete business needs
• Harden HPC-centric implementation for
production environments
– Scalability, Robustness, Performance
• Implement prototypes of upcoming OGF
specifications
– participate in “interop feasts”
Copyright © 2007 Intel Corporation
Backup
Copyright © 2007 Intel Corporation
History: From UNICORE to GPE
•
Pallas:
– German HPC software company
implemented UNICORE client software
– Commercial version UNICOREpro in
2003
– Other products included Vampir and
VampirTrace (now Intel Trace Collector
and Trace Analyzer)
•
After Pallas’ acquisition by Intel in
2003, the new Open Source project
GPE started
– Idea: Support Globus and other Grid
Middleware implementations, too
– Keep the good concepts from UNICORE
• Graphical User Interfaces
• Resource Abstraction
• Client Plug-in Concept
• Ease of use
Copyright © 2007 Intel Corporation
Running a Simple Job
Applications
Services
GridBean
creates JSDL
description
from user
interface
Application
GridBean
send job
description
Target
System
Service
TSI
Utility
Data Center
Copyright © 2007 Intel Corporation
transfer
input/output
control
job
Job
Mgmt.
Service
TSI
File
Transfer
Service
TSI