Transcript Reusable Grid Portal Components Mary Thomas OGCE Consortium Outline • • The Portal Challenge The Solution: – The Open Grid Computing Environments Consortium (OGCE) • • • The OGCE Portal Architecture Application Portal Examples NMI OGCE.

Reusable Grid Portal
Components
Mary Thomas
OGCE
Consortium
Outline
•
•
The Portal Challenge
The Solution:
– The Open Grid Computing Environments
Consortium (OGCE)
•
•
•
The OGCE Portal Architecture
Application Portal Examples
NMI OGCE Plans
OGCE
Consortium
The Portal Challenge
OGCE
Consortium
Three-Tiered Architecture
Portal User Interface
Grid and
Web
Protocols
JDBC,
Local, or
Remote
Connection
Portal
Client
Stub
Database
Service
Database
Portal
Client
Stub
Grid Resource
Broker Service
HPC
or
Compute Cluster
Portal
Client
Stub
Information
and
Data Services
Grid Information
Services, SRB
Three-tiered architecture is accepted standard for
accessing Grid and other services
OGCE
Consortium
Portal are Gateways to Grid
• Early Grid Portals were great success:
– Applications oriented,
– Provided scientists with some useful tools
– Encapsulated some complexities.
• But they were:
– All “stovepipe” solutions.
• Very little reusable code and infrastructure
• Nothing could be shared between portals
– Many ignored emerging architecture of the Grid
• A framework based on distributed ubiquitous services
OGCE
Consortium
Impact of Distributed Grid and Web
Services
Launch, configure
And control
Application Interface
Workflow service
Grid Portals
App Instance
App Instance
App Instance
Open Grid Service Architecture Layer
Registries and
Name binding
Reservations
And Scheduling
Data Management
Service
Security
Policy
Administration
& Monitoring
Logging
Event/Mesg
Service
Accounting
Service
Grid Orchestration
Open Grid Service Infrastructure (web service component model)
Resource layer
1000s of PCs ->massive supercomputers
Online instruments
OGCE
Consortium
(slide by Dennis
Gannon)
Advantage of Portlet Architecture
• Portlet/component approach very compatible with
Web/Grid services model
• Each Grid service can be associated with a unique
portlet
– Very easy to add new services
– Many different Groups can contribute portlets which can be plugged
into a portal.
• Currently Indiana, Argonne, Michigan, NCSA,Texas + GridSphere
portal group following this model. Most likely others as well.
– Jetspeed is Apache standard. Basis for IBM WebSphere portal.
Also Oracle, Sun and others.
• Now a Java Standard (waiting for JSR 168)
• User can select and configure portlets he/she wishes
to use
– Selection becomes part of persistent context
OGCE
Consortium
The Next Generation Grid Portal
• Building on Standard Technologies
– Portlet Design (JSR-168) IBM, Oracle, Sun, BEA, Apache
– Grid standards: Java CoG, Web/Grid Services
– Web server: JetSpeed (open source)
• User configurable, Service Oriented
– Philosophy: The Portal is a gateway to distributed Grid and Web
Services
• With common API, portlets can be exchanged, interoperate
Event and
logging
Services
Portal Server
MyProxy
Server
Metadata
Directory
Service(s)
Application
Factory
Services
Messaging
and group
collaboration
Directory
& index
Services
OGCE
Consortium
User Requirements
Have Not Changed
• Users & Clients are:
– Portal clients/users
– Portal developers
• Security
– Single sign-on, authentication to all grid resources
• Certificate based / uses MyProxy
– VO management
• File Management
– Grid FTP
– File staging
– Metadata management
• Remote Job Management
– Job submission, scheduling, and tracking. Log and output mgmt.
– Workflow management tools
OGCE
Consortium
User Requirements
• Information Services
–
–
–
–
Grid resource and status information directories
Data and metadata catalogs
Personal and VO metadata and service index/directories
Database Interfaces
• Application Interfaces
– Configure and launch workflow from domain specific interface
• Access to Collaboration Tools
– VO collaboration. Synchronous and asynchronous
– Access Grid venues
• Developers need common API
OGCE
Consortium
The Solution:
The Open Grid Computing
Environment (OGCE) Consortium
OGCE
Consortium
Open Grid Computing Environments
(OGCE) Consortium
• Motivation:
– Emerging technologies impacting portal developers:
• Grid/Web services; ubiquitous database use, XMl
– Lessons learned from GCE-RG and other projects
(GridPort/HotPage, Alliance Portal, DOE Components,
NEESGrid, Gateway, GRaDS, others)
– Multiple projects repeating similar tasks, work,
development;
• OGCE established in Fall 2003 to foster
collaborations and sharable components with
portal developers worldwide.
OGCE
Consortium
OGCE
• Funded by:
– NSF NMI: software releases will be part of NMI
and synchronized with GRIDS releases
– Also: NCSA Alliance, NPACI, DOE, NASA,
others
• Release Schedule:
– First Release: 11-15-03
– Next Release: April, 2004
• Website: http://www.ogce.org
OGCE
Consortium
OGCE Tasks/Goals
• Development of interoperable Grid portal
architecture
– Define standards for common API’s between portlet
service and remote services
• Establishment of a Grid Portal Collaboratory
– Repository of portlet and portal service components
• An online forum for developers of Grid Portals
• The building of reusable portal components:
– Can be integrated in a common portal container system
• Contribute to standards organizations:
– NSF NMI program  software releases
– GGF if appropriate or applicable
OGCE
Consortium
The OGCE Portal Architecture
OGCE
Consortium
OGCE Initial Architecture
Proxy
Portlets
Java
COG
API
HTTP
Java
CoG
Kit
Remote
Interfaces
Portal
Local
Portlets
Grid
Protocols
Grid Services
GRAM,
MDS-LDAD
MyProxy
Grid Services
CoG
Stubs
Other Services
SOAP
Teamlets
Service
API
CHEF
Services
Jetspeed
Internal
Services
Initial architecture aggregates multiple services into a single
portal using portlet containers
OGCE
Consortium
OGCE Initial Arch:
Features & Capabilities
• OGCE approach encompasses multiple portal
development systems:
– Preserving development investments
• Central portal system (Jetspeed) acts as a central
portal container:
– Portal responsible for managing the lifecycle of pluggable portal
components, or portlets.
• Grid/computational portlets must provide services on
the behalf of the user:
– Not just a collection of links/static pages
• User view (from the left) is of a single portal composed
of portlets.
OGCE
Consortium
Supports Aggregation of
Diverse Tasks
• Portlets provide a uniform management interface to
the portal (on the left) connect to heterogeneous set of
APIs on the right.
– First horizontal thread
• Portlets interact with web interfaces written in other
languages such as Perl, Python, PHP, ColdFusion, etc.
– Architecture for bridged support is depicted in the second horizontal
thread
• Examples:
– Locally running Myproxy Grid portlet uses Java CoG kit to interact
with remote Grid services
– Job Scheduler portlet uses GridPort Toolkit to submit job via
Community Scheduling Frameworks (CSF) service
– Teamlets (sharable portlets) share collaboration and Grid services,
such as calendars, message boards, file services.
OGCE
Consortium
Portal
Grid
Service
Stubs
Portlets
and
Teamlets
Service
API
Grid
Protocols
Grid
Services
Java
CoG Kit
OGCE - Revised Architecture
Local
Portal
Services
Remote
Content
Services
HTTP
Remote
Content
Servers
Jetspeed
Internal Services
• Revised portal architecture provides a unified interface for portal services.
• Supports aggregation of services in backend
OGCE
Consortium
Features
• All services will employ a single (generic)
service API that allows them to be deployed
and accessed in a uniform fashion.
• Developer then chooses (during portal
configuration) the implementations of
particular services.
• Services may be local, may access remote
sites via Grid protocols, or may access remote
content, such as web interfaces written in Perl.
• OGCE Twist:
– Entire portal will auto-install and include all required
software components
OGCE
Consortium
Key Technologies????
• Core Portal Framework:
– CHEF/JetSpeed (U. Mich)
• Core Grid Technologies:
– JavaCoG (provides Globus interface)
– GridPort Toolkit (aggregates JavaCoG, and
other grid services)
OGCE
Consortium
Portal/Application Gallery
OGCE
Consortium
TeraGrid Portal
• Access Point for TeraGrid
Services
• Version 0: Collecting Initial
Services
– Public Information about
Resources
– Private Information for the
developers.
• Version 1: User centered portal
– HotPage/GridPort style access to
user accounts, credentials, job
submission & management.
• Version 2: Portals for Science
Collaborations (Q3 2004)
– Shared spaces, whiteboards, AG
access, group authorization,
shared application services
OGCE
Consortium
Portal
Grid
Service
Stubs
Portlets
and
Teamlets
Service
API
Local
Portal
Services
Remote
Content
Services
GridPort Toolkit
TeraGrid Integrated Architecture
HTTP
Grid
Services
Java
CoG Kit
Web
Services
Remote
Content
Servers
Jetspeed
Internal Services
Diagram demonstrates how existing software projects (such as GridPort) can
be adapted to support NMI Portals software system
OGCE
Consortium
Alliance scientific portal
Create
job
description
Gather
data files,
resources
Retrieve customized
run description template
from data management
services
Generate
and store
derived
products
Store and
record
results
Execute
application
Broker Service
Need ‘x’
put to ‘l’
by time y:00
Monitoring results captured in event
services persistent channel. myMEAD
records address of service and channel
Need to
store
‘y’ and ‘z’
monitoring
Teragrid
data service
myMEAD
personal
information
space
Provides context
to resolve ‘x’, ‘y’, ’z’
GridFTP
Delayed invocation
of GridFTP
MetaCatalog
Service(MCS)
NEESgrid MetaCatalog service
Teragrid Data Servicesingle point of contact
to server-side data
servers
Storage Resource
Broker (SRB)
Globus Toolkit 3.0
Teragrid
OGCE
Consortium
DOE Fusion Grid Portal
• Portal Development on
the DOE SciDAC Grid
– Initially, Fusion Grid
• Multi-site collaborative
project:
– TACC, Indiana, SDSC,
General Atomics
• Developing interoperable
portal and Web services
• Portal architecture based
on NMI Portal software
OGCE
Consortium
LEAD Portal
OGCE
Consortium
CHEF - NEESGrid
• NEES Tools
–
–
–
–
–
Data Viewers (Live/Stored)
Telepresense
Video as Data
Electronic Notebook
Data/Metadata Repository
• Grid Technologies
–
–
–
–
Grid sign on using myproxy
Grid computational portal
GridFTP
..Many more
OGCE
Consortium
OGCE Plans
OGCE
Consortium
OGCE Portal System
OGSA
Globus
COG
CHEF
Toolkit
NMI
Dist
System
Integration
Gridport
Alliance
Portal
Testing
Workflow
Slide courtesy of Chuck Severance
OGCE
Consortium
NMI Features List (April 04 Release)
• Content Management:
– Resources: Users and groups can upload and share documents
and URLs.
– News: Several news feeds are available, including XMLHack and
BBC News.
– Grid Context: Store arbitrary web objects (movies, web pages,
audio files) in a customizable, annotated directory tree.
• Collaborative:
– Chat: Engage other members of your group in online discussion.
– Discussion: Participate in discussions with other members of your
group
– Calendar Program:schedule appointments and attach associated
documents and URLs. Calendars can be used to schedule both
user and group activities.
– Anabas Impromptu: system for real time shared display, audio, and
chats.
OGCE
Consortium
NMI Features List (April 04 Release)
•
GT 3.0/Interactive:
– Proxy Manager and Job Submit: Obtain and manage one or more Grid credentials
that can be used to access Grid resources through your browser.
– GridFTP: Use Grid credential to browse remote directories and to upload and
download files between your desktop and a remote host.
– Application Manager
– Application Factory Manager
•
Remote Job Management
– GridAnt for workflow
– Job submission portal
– Community Scheduling Framework (CSF): submit jobs to scheduler
•
Information Services:
– LDAP Browser:
– Navigate the LDAP server of your choice.
– GridPort Information Repository Portlets (GPIR): Portlets are available for browsing
Machine Summaries, System Status, and Total Grid Capabilities.
•
Application Portlets:
– Specialized portlets for projects
OGCE
Consortium
Key Feature –
Download a Portal & Install
•
Prerequisite Software
•
– Java 2 Standard Edition version
1.4.x or higher.
– Apache Ant version 1.5.x or higher.
•
•
– Move into the portal/chef-1.0.7
directory.
– Edit file "build.properites"
– Type "ant deploy" to build in and
install the portal.
– Start the Tomcat web server.
– Point your browser at http://<your
host name>:10081/ogce and create
an account.
Preparing the Installation
– Optionallly, create a "portal" account
on your web server.
– Download the portal bundle
– Put the download bundle into the
home directory of "portal" account.
– Untar or unzip the portal bundle.
– Globus: installation process will
create the directories .globus and
.globus/certificates in the
Software automatically installed
includes:
Install the Portal
•
OS/Machines Tested:
–
–
–
–
MAC OSX
PC/Windows XP
Linux
Cygwin
– Globus/Java CoG
– CHEF
OGCE
Consortium
Current Status
• SC03 Demonstration portal and software download
available
– Beta1
• Updated Release available now
– Bug fixes
– Now supporting GT3.x with J2SDK 1.4.2
• Working on NMI April 2004 release
– Including new features to support science portal applications
• Open GCE Runtime Engine (OGRE) workflow services and portlets
• Sample template portlets for simple science applications
– Based on aggregation of existing projects
– OGSA/OGSI/GT3.x compatible
OGCE
Consortium
Future Directions
• Moving towards new portlet standard
– JSR 168
– Should allow compatibility with
• GridSphere
• Academic education and business portals Oncourse, OneStart, CHEF
(of course), and uPortal
• Major commercial products offerings from Sun, IBM, Oracle, etc.
• Incorporate new features:
– Grid Services (CSF, OGSA-DAI portlets, Job Sequencer)
– Access Grid, Polycom and other AV Portlets
• Collaborations and Contributions
– DOE SciDAC projects (such as CMCS)
• Expand online repository
– Develop process for third party contributions
OGCE
Consortium
Future Directions
• Moving towards new portlet standard
– JSR 168
– Should allow compatibility with
• GridSphere
• Academic education and business portals Oncourse, OneStart, CHEF
(of course), and uPortal
• Major commercial products offerings from Sun, IBM, Oracle, etc.
• Incorporate new features:
– Grid Services (CSF, OGSA-DAI portlets, Job Sequencer)
– Access Grid, Polycom and other AV Portlets
• Collaborations and Contributions
– DOE SciDAC projects (such as CMCS)
• Expand online repository
– Develop process for third party contributions
OGCE
Consortium
Participants
the globus alliance
University of Texas
at Austin
Indiana University
JavaCoG
University of Michigan
National Center for
Supercomputing Applications
Argonne National Labs
OGCE
Consortium
References
• OGCE Website
– http://www.ogce.org
• NSF Middleware Initiative
– http://www.nsf-middleware.org
OGCE
Consortium