Transcript Lezione 001 6 Aprile 2009 - Istituto Nazionale di Fisica

Università degli Studi di Bari – Corso di Laurea Specialistica in Informatica
“Tecnologia dei Servizi “Grid e cloud computing”
A.A. 2009/2010
Giorgio Pietro Maggi [email protected], http://www.ba.infn.it/~maggi
Lezione 7b - 9 Dicembre 2009
Il materiale didattico usato in questo corso è stato mutuato da quello
utilizzato da Paolo Veronesi per il corso di Griglie Computazionali
per la Laurea Specialistica in Informatica tenuto nell’anno
accademico 2008/09 presso l’Università degli Studi di Ferrara.
Paolo Veronesi
[email protected], [email protected]
http://www.cnaf.infn.it/~pveronesi/unife/
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
0
Defining the Grid
A
Grid is the combination
of networked resources and the
corresponding middleware, which
provides services for the user.
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
1
The EGEE Project

Aim of EGEE:
“to establish a seamless European Grid infrastructure for the support of the
European Research Area (ERA)”

EGEE
 1 April 2004 – 31 March 2007
 71 partners in 27 countries, federated in regional Grids

EGEE-II
 1 April 2007 – 30 April 2008
 Expanded consortium

EGEE-III
 1 May 2008 – 30 April 2010
 Transition to sustainable model
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
2
Enabling Grid for E-sciencE project
Flagship Grid infrastructure project cofunded by the European Commission
starting from April 2004
Entering now in the 3° phase
Archeology
Astronomy
Astrophysics
Civil Protection
Comp. Chemistry
Earth Sciences
Finance
Fusion
Geophysics
High Energy Physics
Life Sciences
Multimedia
Material Sciences
…
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
>250 sites
48 countries
>50,000 CPUs
>20 PetaBytes
>10,000 users
>150 VOs
>150,000 jobs/day
3
Defining the Grid
A
Grid is the combination
of networked resources and the
corresponding middleware, which provides
services for the user.
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
4
EGEE Infrastructure
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
5
EGEE Infrastructures

Production service



Pre-production service




Scaling up the infrastructure with resource centres around the globe
Stable, well-supported infrastructure, running only well-tested and reliable
middleware
Run in parallel with the production service (restricted nr of sites)
First deployment of new versions of the gLite middleware
Test-bed for applications and other external functionality
T-Infrastructure (Training&Education)


Complete suite of Grid elements
and application (Testbed, CA,
VO, monitoring, support, …)
Everyone can register and use
GILDA for training and testing
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
20 sites on 3 continents
6
EGEE Operations Process

Geographically distributed responsibility for
operations:


There is no “central” operation
Regional Operation Centers




Tools are developed/hosted at different sites:
 GOC DB (RAL), SAM (CERN), GStat (Taipei),
CIC Portal (Lyon)
Highlights:
Grid operator
on duty operation
Distributed
 10 teams working in weekly rotation
Evolving and maturing procedures
 Crucial in improving site stability and management
Procedures being in introduced into and shared with the
Operations
coordination
related
infrastructure projects




Responsible or resource centers in their region
Weekly operations meetings
Regular ROC managers meetings
Series of EGEE Operations Workshops
Procedures described in Operations Manual




Introducing new sites
Site downtime scheduling
Suspending a site
Escalation procedures; etc.
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
7
Improved reliability through multi-level monitoring
Doubled size and usage without impact on operations
Central
probes
(SAM)
Local
probes
Network
monitoring
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
8
EGEE operations
Operations Coord.
Centre (OCC)
- management, oversight of
all operational and support
activities
Regional Operations
Centres (ROC)
- providing the core of the
support infrastructure, each
supporting a number of resource
centres within its region
Resource Centres (RC)
- providing resources
(computing, storage, network…)
- At FZK, coordination and
management of user support,
single point of contact for users
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
9
Monitoring Visualization
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
10
10
The EGEE support infrastructure
•RC A
•RC A
RC A
•RC B
•RC B
RC B
•RC C
•RC C
RC C
CIC
Portal
•ROCBC
•ROC
ROC
N
•ROCBC
•ROC
ROC
N
COD
Network
Support
Network
Support
OtherGrids
Grids
Other
Other Grids
VO Support
VO Support
VO Support
C
B
A
VO TPM C
VO TPM B
VO TPM A
GGUS
Central
System
TPM
OtherGrids
Grids
Other
Other Grids
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
Deployment
Middleware
Deployment
support
support
support
Middleware
Middleware
Middleware
support
support
support
Middleware
Middleware
Middleware
support
support
support
11
Defining the Grid
A
Grid is the combination
of networked resources and the
corresponding middleware, which provides
services for the user.
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
12
gLite Middleware Distribution

Combines components from different providers





Focus on providing a deployable MW
distribution for EGEE production service


Middleware services + configuration tools
Follows a service oriented approach


Condor and Globus (via VDT)
LCG
EGEE
Others
Usage of webservices where useful and possible performance-wise
Complemented by application-level servcies
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
13
Production Grid Middleware
Key factors in EGEE Grid Middleware Development:
1.
Strict software process
Use industry standard software engineering methods

Software configuration management, version control, defect tracking,
automatic build system, …
2.
Conservative approach in what software to use
Avoid “cutting-edge” software

Deployment on over 200 sites cannot assume a homogenous
environment – middleware needs to work with many underlying
software flavors
Avoid evolving standards

Evolving standards change quickly (and sometime significantly cf.
OGSI vs. WSRF) – impossible to keep pace on > 200 sites
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
14
gLite Process
Development
Directives
External
Software
Error Fixing
Software
Integration
Certification
Pre-Production
Deployment
Packages
Problem
Production
Infrastructure
Fail
Integration
Tests
Testbed
Deployment
Fail
Pass
Functional
Tests
Installation Guide,
Release Notes, etc
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
Pass
Pre-Production
Deployment
Pass
Fail
Scalability
Tests
15
gLite Software Process

Technical Coordination Group (TCG)
 gathers & prioritizes user requirements
from HEP, biomed, (industry), sites
gLite development is client-driven!
Software from EGEE-JRA1 and other projects
 JRA1 preview testbed (currently being set up)




early exposure to users of “uncertified” components
SA3 Integration Team
 Ensures components are deployable and work
 Deployment Modules implemented high-level gLite node types

(WMS, CE, R-GMA Server, VOMS Server, FTS, etc)
Build system now spun off into the ETICS project (Jan 2006)
SA3 Certification Team
 Merge of the JRA1 testing and SA1 certification teams
 Dedicated testbed; test release candidates and patches
 Develop test suites
SA1 Pre-Production System
 Scale tests by users



Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
16
Building Software for the Grid
Applications
Environmental
Sciences
Life &
Pharmaceutical
Sciences
Geo Sciences
Middleware
APST
Globus GT4
Condor
Courtesy IBM
Platform
Infrastructure
Unix
Windows
JVM
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
TCP/IP
MPI
.Net Runtime
VPN
SSH
17
Building Software for the Grid
Applications
Environmental
Sciences
Life &
Pharmaceutical
Sciences
Geo Sciences
Upper Middleware & Tools
Middleware
APST
Globus GT4
Lower Middleware
Platform
Infrastructure
Unix
Windows
JVM
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
TCP/IP
MPI
.Net Runtime
Condor
Bonds
Courtesy IBM,
VPN
SSH
18
Defining the Grid
A
Grid is the combination
of networked resources and the
corresponding middleware, which provides
services for the user.
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
19
EGEE Applications

>270 VOs from several
scientific domains










Astronomy & Astrophysics
Civil Protection
Computational Chemistry
Comp. Fluid Dynamics
Computer Science/Tools
Condensed Matter Physics
Earth Sciences
Fusion
High Energy Physics
Life Sciences
Further applications
under evaluation
Applications have moved from
testing to routine and daily usage

~80-95% efficiency
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
20
The Future of Grids

Increasing the number of infrastructure users by
increasing awareness



Increasing the number of applications by improving
application support and middleware functionality


Improved usability through high level grid middleware extensions
Increasing the grid infrastructure



Dissemination and outreach
Training and education
Incubating related projects
Ensuring interoperability between projects
Protecting user investments

Towards a sustainable grid infrastructure
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
21
Grid Interoperability

Incubator for new Grid efforts world-wide


Leading role in building world-wide
Grids through interoperation efforts



Bilateral: EGEE/OSG, EGEE/NDGF,
EGEE/NAREGI, EGEE/Unicore/DEISA
Multilateral: Grid Interoperability Now
(GIN)
GI N
Experiences and
requirements fed back into
standardization process (OGF)


Infrastructure and application efforts
Many EGEE members are area directors,
WG chairs, WG members
Contacts with industry strengthened

Industry Days, Industry Task Force,
Business Associates Programme
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
22
EGEE working with related infrastructure projects
GI N
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
23
Evolution
National
European
e-Infrastructure
Global
Testbeds
Routine Usage
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
Utility Service
24




Need to prepare permanent, common Grid infrastructure
Ensure the long-term sustainability of the European e-Infrastructure
independent of short project funding cycles
Coordinate the integration and interaction between National Grid
Infrastructures (NGIs)
Operate the production Grid infrastructure on a European level for a wide
range of scientific disciplines
Must be no gap in the
support of the production
grid
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
25
Summary
Grids represent a powerful new tool for science
Today we have a window of opportunity to move grids from research
prototypes to permanent production systems (as networks did a few
years ago)
EGEE offers …
 … a mechanism for linking together people, resources and data of
many scientific community
 … a basic set of middleware for gridfying applications with
documentation, training and support
 … regular forums for linking with grid experts, other communities and
industry
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
26
gLite Middleware overview
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
27
Grid Middleware

When using a PC or
workstation you







Login with a username and
password
(“Authentication”)
Use rights given to you
(“Authorisation”)
Run jobs
Manage files: create them,
read/write, list directories
Components are linked by a
bus
Operating system
One admin domain
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
• When using a Grid you
– Login with digital
credentials
(“Authentication”)
– Use rights given you
(“Authorisation”)
– Run jobs
– Manage files: create
them, read/write, list
directories
• Services are linked by the
Internet
• Middleware
• Many admin domains
28
EGEE Project and gLite
•
Enabling Grids for E-sciencE (EGEE) is the largest multi-disciplinary grid
infrastructure in the world
–
–
–
–
•
Brings together more than 120 European organisations
Consists of 250 sites in 48 countries and more than 68,000 CPUs
Is available to some 8,000 users 24 hours a day, 7 days a week
Processes more than 150,000 jobs per day from different scientific domains
gLite is the middleware powering the EGEE infrastructure and many other
related projects
–
–
–
Is an integrated set of components designed to enable resource sharing among
different institutions
Pulls together contributions from many other projects, including LCG and VDT
Enable users with a large set of services
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
29
The “global” grid
e-Infrastructures adopting gLite
e-Infrastructures interoperable or in progress to be made interoperable with gLite
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
30
Additional Infrastructures: GILDA
•
EGEE provides a training infrastructure: GILDA (Grid INFN
Laboratory for Dissemination Activities)
–
–
–
Runs the entire gLite stack protocols
Used to demonstrate EGEE grid technology project
Supports beginner and expert training courses on gLite
•
Adopted by several Grid projects worldwide
•
Own Certification Authority
•
Available 365 days for everyone !
•
Used in the ISSGC schools series
•
Since 2007 other middleware than gLite are tested on GILDA
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
31
The GILDA t-Infrastructure (https://gilda.ct.infn.it)
•20 sites in 3 continents
•> 11000 certificates issued, >20%
renewed at least once
•> 250 courses, training events,
official university curricula
•> 2,000,000 hits on the web site from
>100 different countries
•> 4.5 TB of training material
downloaded from the web site
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
32
gLite in the Grid “ecosystem”
Condor
Globus
MyProxy
...
EDG
OSG,
…
...
VDT
DataTAG
LCG
CrossGrid
...
SRM
GridCC
NextGrid
EGEE
interactive
USA
Future grids
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
…
DEISA
EU
Used in
33
The Middleware structure
•
Applications have access both to
Higher-level Grid Services and to
Foundation Grid Middleware
•
Higher-Level Grid Services are
supposed to help the users building
their computing infrastructure but should
not be mandatory
•
Foundation Grid Middleware are
actually developed in EGEE
–
–
–
Must be complete and robust
Should allow interoperation with other major
grid infrastructures
Should not assume the use of Higher-Level
Grid Services
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
34
gLite infrastructure
Workload Management System (WMS)
Data Management
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
35
Typical Job workflow
glite-job-submit myjob.jdl
Myjob.jdl
Information
Executable = “gridTest”;
Input Sandbox
StdError
=
“stderr.log”;
Replica
StdOutput = “stdout.log”; Service
InputSandbox = {“/home/joda/test/gridTest”};
Catalog
JDL
OutputSandbox = {“stderr.log”, “stdout.log”};
User Interface
JDL
Author.
Service
InputData = “lfn:testbed0-00019”;
DataAccessProtocol = “gridftp”;
Requirements = other.Architecture==“INTEL” && \
other.OpSys==“LINUX”;
Rank = “other.GlueHostBenchmarkSF00”;
Job Submit
Event
Output Sandbox
Input Sandbox
Resource
Broker
Storage
Element
Job
Logging &
Book-keeping
Job Submission
Service
Output Sandbox
Job Status
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
GSI data
acc/transf
Computing
Element
36
Security System
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
37
gLite Security
•
Authentication based on X.509 PKI infrastructure
–
–
–
•
Certificate Authorities (CA) issue (long lived) certificates
identifying individuals (much like a passport)
Trust between CAs and sites is established (offline)
In order to reduce vulnerability, Grid user identification
is done by (short lived) proxies of their certificates
Proxies can
–
–
–
–
Be delegated to a service such that it can act on the
user’s behalf
Include additional attributes (like VO information via the
VO Membership Service VOMS)
Be stored in an external proxy store (MyProxy)
Be renewed (in case they are about to expire)
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
38
Which CA are trusted in LCG/EGEE?
http://www.eugridpma.org/
“The EUGridPMA is the international organization to coordinate the trust fabric for e-Science grid
authentication in Europe. It collaborates with the regional peers APGridPMA for the Asia-Pacific
and The Americas Grid PMA in the International Grid Trust Federation. The charter document
defines the group's objective, scope and operation. It is the basis for the guidelines documents on
the accreditation procedure, the Authentication profile for X.509 secured "classic" certification
authorities and other IGTF recognised Profiles. “
In LCG/EGEE CA are installed on machine trough rpms.
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
39
Conventional grid security
Bob
Cert request
Certification Authority (CA)
Bob´s Grid certificate
grid-proxy-init
- Single sign-on
- Delegation through proxy certificate
Grid resources (B)
User Interface
(UI)
Grid resources (A)
Sysadmin A :
- Create user “grid1“
- Map Bob´s certificate to “grid01“
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
Sysadmin B :
- Create user “user001“
- Map Bob´s certificate to “user001“
- Manual user “mapping“
- No info about VOs
40
gLite: VOMS

Virtual Organization Membership Service (VOMS)

EGEE/gLite enhancement for VO management
Provides information on user's relationship with Virtual Organization (VO)
Membership
Group membership
Roles of user
Multiple VO
User can register to multiple VOs and create an aggregate proxy
Access ressources in every registered VO
Backward compatibility
Extra VO related information in users proxy certificate
Users proxy can still be used with non VOMS-aware services
7 Maggio 2009 – Paolo Veronesi
Griglie Computazionali - Lezione 007
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
41
41
gLite: VOMS - Web interface

Requires a valid certificate from a
recognized CA imported on the
browser

VO user can
Query membership details
Register himself in the VO
Needs a valid certificate
Track his requests

VO manager can
Handle requests from users
Administer the VO

Everybody can
Get information about the VO
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
42
gLite – Enhanced security in gLite
Bob
Cert request
Certification Authority (CA)
Bob´s Grid certificate
VO membership request
User Interface
(UI)
VO
Service
VO
Grid resources (A)
VO
Database
VO
Manager
voms-proxy-init
Automatic mapping
for Bob
VO
Account
Pool
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
Automatic mapping
for Bob
Grid resources (B)
VO
Account
Pool
43
LCAS & LCMAPS
•
At resources level, authorization info is extracted
from the proxy and processed by LCAS and
LCMAPS
•
Local Centre Authorization Service (LCAS)
–
–
•
Checks if the user is authorized
Checks if the user is banned at the site
Local Credential Mapping Service (LCMAPS)
–
–
Map remote credentials to local credentials (eg. different
UNIX uid/gid)
Map also VOMS group and roles (full support of FQAN)

enables privileges separations
Tecnologia dei Servizi “Grid e cloud computing” - Lezione 7b
44