Transcript Co-ordination & Harmonisation of Advanced e-INfrastructures CHAIN Worldwide Interoperability Test Roberto Barbera – Univ.

Co-ordination & Harmonisation of Advanced e-INfrastructures
CHAIN Worldwide Interoperability Test
Roberto Barbera – Univ. of Catania and INFN
Diego Scardaci - INFN
CHAIN Workshop – Taipei, 27 Feb. 2012
Research Infrastructures – Grant Agreement n. 260011
Outline

Science Gateway Architecture

World-wide interoperability test with Science
Gateways

Summary and conclusions
2
Access:
Science Gateway model
Embedded Applications
App. 2
....... App. N
Science
Gateway
App. 1
Administrator
Power User
Basic User
Standard-based
middleware-independent
Grid Engine
Users from
different
organisations
having different
roles and
privileges
3
AuthN/AuthZ Infrastructure


Portal Engine
 Liferay Community Edition 6.0.6
Authentication



Shibboleth-2.4.3-2.1/log4shib-1.0.3-2.3/simpleSAMLphp
Implementations of SAML for AuthN
(Security data exchange among different Security Domains)
OpenLDAP

Takes care of user roles and privileges (AuthZ)
Science Gateway
Authentication



Single Sign-On
Users can only do what the SG
allows them to do
All transactions are tracked
Authorization
4
Standards

The framework for Science Gateways proposed and fostered by
CHAIN is fully web-based and adopts official worldwide standards
and protocols, through their most common implementations:





Web interface: JSR 168 and JSR 286 standards (also known as "portlet 1.0"
and "portlet 2.0" standards)
Authentication: OASIS Security Assertion Markup Language (SAML)
standard and its Shibboleth and SimpleSAMLphp implementations;
Authorisation: Lightweight Direct Access Protocol, and its OpenLDAP
implementation
Digital certificate management: Cryptographic Token Interface Standard
(PKCS#11) standard and its Cryptoki implementation
Application interface: Open Grid Forum (OGF) Simple API for Grid
Applications (SAGA) standard and its JSAGA implementation
5
Interoperability
(http://en.wikipedia.org/wiki/Interoperability)


Interoperability is a property referring to the ability of diverse
systems and organizations to work together (inter-operate). The
term is often used in a technical systems engineering sense, or
alternatively in a broad sense, taking into account social, political,
and organizational factors that impact system to system
performance
According to ISO/IEC 2382-01 (Information Technology Vocabulary,
Fundamental Terms), interoperability is defined as follows: "The
capability to communicate, execute programs, or transfer data among
various functional units in a manner that requires the user to have little
or no knowledge of the unique characteristics of those units"
6
CHAIN interoperability test:
objectives

To demonstrate that:

e-Infrastructures can be made interoperable to each other
using standards (with the meaning of interoperability given
above)

VRC-specific applications can be submitted from anywhere and
run everywhere
7
CHAIN interoperability test:
requirements
1.
User interface is only web based, for simplicity
2.
Users must be transparently authenticated & authorised on all
e-Infrastructures without any additional human/machine
intervention
3.
There must be the smallest possible interaction with both site
managers and e-Infrastructure operators
4.
No modification of the middleware should be requested to
developers
8
Catania Grid Engine
Liferay Portlets
Science
GW 1
Science
GW 2
Science
GW 3
Grid Engine
eToken
Server
Science GW Interface
Data Engine
Job Engine
User
Track. &
Monit.
User
Tracking
DB
SAGA/JSAGA API
Grid MWs
9DONE
DONE
DONE
By mid April
In progress
9
Job Engine





The Job Engine is made of a set of libraries to develop
applications able to submit and manage jobs on a grid
infrastructure
It is compliant with the OGF SAGA standard;
It is optimized to be used in a web portal running an
application server (e.g., Glassfish, Tomcat, etc.) based on
J2EE
It can be used also in stand-alone mode
JSAGA is the SAGA implementation adopted
10
Job
Submission
Worker Threads for
Status Checking
MONITORING
MODULE
Job
Check Status/
Get Output
USER
TRACKING
DB
WT
WT
WT
WT
WT
WT
WT
WT
WT
WT
Job Queue
Worker Threads for Job Submission
GRID INFRASTRUCTURE(S)
Job Engine -Architecture
Job Engine - Features

The Job Engine has been designed with the following features in mind:
Feature
Description
Status
Middleware
Independent
Capacity to submit job to resources running different
middleware
DONE
Easiness
Create code to run applications on the grid in a very
short time
DONE
Scalability
Manage a huge number of parallel job submissions
DONE
fully exploiting the HW of the machine where the Job
Engine is installed
Performance
Have a good response time
Accounting &
Auditing
Register every grid operation performed by the users DONE
Fault Tolerance
Hide middleware failure to end users
ALMOST
DONE
Workflow
Providing a way to easily create and run workflows
IN PROGRESS
DONE
12
Job Engine – Scalability
Job submission time (h)


Submission time
scales linearly with
number of jobs;
Actual time depends
on HW capabilities
and thread-pools
configuration
Time to submit 10000 jobs (h)
40000 Jobs
submitted
at the same time!
13
Job Engine –
Accounting & Auditing



A powerful accounting & auditing system is included in the Job
Engine
It is fully compliant with EGI VO Portal Policy and EGI Grid
Security Traceability and Logging Policy
The following values are stored in the DB for each job submitted:








User ID
Job Submission timestamp
Job Done timestamp
Application name
Job ID
Robot certificate ID
VO name
Execution site (name, lat, long)
Overall usage (Dec. 2011 → today)
350
300
250
200
150
100
50
0
Dec
Jan
Feb
14
CHAIN interoperability test:
e-Infrastructures involved so far
gLite-based e-Infrastructures/Projects
EUAsiaGrid
EUChinaGRID
EU-IndiaGrid
EUMEDGRID
GISELA
IGI (Italy)
SAGrid (South Africa)
15
Step 1 – Grid authorisation

User VO(s) are authorized at all sites of the various
e-Infrastructures


This would break requirement 3 and could require quite some
time to be done
The certificates of the Science Gateway(s) to be used for
the test are registered in the gLite VOMS servers and
Unicore XUUDB
16
Step 2 – Resource discovery

Create a (super)-topBDII that gathers all the topBDII’s of
the various e-Infrastructures



This would break requirement 3 and require a grid service
to be managed by CHAIN
Information can become outdated
Insert the list of the topBDIIs/WMSs/TargetSystems of the
various e-Infrastructures/middleware in the configuration of
the portlets of the test applications and let the Grid Engine
choose at job submission resource managers and services to
be used
17
Multi-infra/multi-mw Job Engine
gLite
Infrastr. Info
(BDII,VO, etc.)
gLite
Infrastr.
Info
Unicore
Infrastr.
Info
EUMEDGRID
e-Infrastructure
Submit
User
Multi-infrastructure/
Multi-middleware
Science Gateway
GISELA
e-Infrastructure
Juelich
Supercomputing Centre
18
MyJobsMap (1/4)
19
MyJobsMap (2/4)
20
MyJobsMap (3/4)
21
MyJobsMap (4/4)
Both sequential and MPI-enabled
jobs successfully executed
22
Summary and conclusions

The adoption of standards (JSR 286, SAML, SAGA, etc.) in the
Science Gateway proposed and fostered by CHAIN represents a
first step towards a global interoperable architecture

The model is used to demonstrate CHAIN global harmonisation
of e-Infrastructures world-wide

The model nicely implements HTC/HPC interoperability at the
user/application level
23
Co-ordination & Harmonisation of Advanced e-INfrastructures
Thank you