Transcript www.nordugrid.org NorduGrid: the light-weight Grid solution LCSC 2003 Linköping, October 23, 2003 Oxana Smirnova Some facts  NorduGrid is: – A Globus-based Grid middleware solution for Linux clusters –

www.nordugrid.org
NorduGrid:
the light-weight Grid solution
LCSC 2003
Linköping, October 23, 2003
Oxana Smirnova
Some facts
 NorduGrid is:
– A Globus-based Grid middleware solution for Linux
clusters
– A large international 24/7 production quality Grid
facility
– A resource routinely used by researchers since
summer 2002
– A freely available software
– A project in development
 NorduGrid is NOT:
–
–
–
–
2003-10-23
Derived from other Grid solutions (e.g. EU DataGrid)
An application-specific tool
A testbed anymore
A finalized solution
[email protected]
www.nordugrid.org
2
Some history

Initiated by several Nordic universities
–

Started in January 2001
–
–

–
Forced by the necessity to execute “ATLAS Data Challenges”
Deployed a light-weight and yet reliable and robust Grid solution in time for
the ATLAS DC tests in May 2002
Will continue for 4-5 years more (and more?..)
–
–
–
2003-10-23
Common Certification Authority and Virtual Organization tools, Globus2
configuration
Common applications (high-energy physics research)
Switched from deployment to R&D in February 2002
–
–

Initial budget: 2 years, 3 new positions
Initial goal: to deploy EU DataGrid middleware to run “ATLAS Data Challenge”
Cooperation with EU DataGrid
–

Copenhagen, Lund, Stockholm, Oslo, Bergen, Helsinki
Form the ”North European Grid Federation” together with the Dutch Grid,
Belgium and Estonia
Will provide middleware for the ”Nordic Data Grid Facility”
…as well as for the Swedish Grid facility SWEGRID, Danish Center for Grid
Computing, Finnish Grid projects etc
[email protected]
www.nordugrid.org
3
The resources

Almost everything the Nordic academics
can provide (ca 1000 CPUs in total):
– 4 dedicated test clusters (3-4 CPUs)
– Some junkyard-class second-hand clusters
(4 to 80 CPUs)
– Few university production-class facilities
(20 to 60 CPUs)
– Two world-class clusters in Sweden, listed
in Top500 (238 and 398 CPUs)

Other resources come and go
–
–
–
–

Canada, Japan – test set-ups
CERN, Dubna – clients
It’s open so far, anybody can join or part
Number of other installations unknown
People:
– the “core” team keeps growing
– local sysadmins are only called up when
users need an upgrade
2003-10-23
[email protected]
www.nordugrid.org
4
A snapshot
2003-10-23
[email protected]
www.nordugrid.org
5
NorduGrid specifics
1.
It is stable by design:
a)
b)
c)
2.
It is light-weight, portable and non-invasive:
a)
b)
c)
d)
e)
f)
3.
2003-10-23
The nervous system: distributed yet stable Information System
(Globus’ MDS 2.2+patches)
The heart(s): Grid Manager, the service to be installed at master
nodes (based on Globus, replaces GRAM)
The brain(s): User Interface, the client/broker that can be installed
anywhere as a standalone module (makes use of Globus)
Resource owners retain full control; Grid Manager is effectively a
yet another user (with many faces though)
Nothing has to be installed on worker nodes
No requirements w.r.t. OS, resource configuration, etc.
Clusters need not be dedicated
Runs on top of existing Globus installation (e.g. VDT)
Works with any Linux flavor, Solaris, Tru64
Strategy: start with something simple that works for users
and add functionality gradually
[email protected]
www.nordugrid.org
6
How does it work

Information system knows everything
–
–
–
–

The server (“Grid manager”) on each gatekeeper does most of
the job
–
–
–
–
–

Pre- and post- stages files
Interacts with LRMS
Keeps track of job status
Cleans up the mess
Sends mails to users
The client (“User Interface”) does the brokering, Grid job
submission, monitoring, termination, retrieval, cleaning etc
–
–
–
–
2003-10-23
Substantially re-worked and patched Globus MDS
Distributed and multi-rooted
Allows for a pseudo-mesh topology
No need for a centralized broker
Interprets user’s job task
Gets the testbed status from the information system
Forwards the task to the best Grid Manager
Does some file uploading, if requested
[email protected]
www.nordugrid.org
7
Information System
 Uses Globus’ MDS 2.2
– Soft-state registration
allows creation of any
dynamic structure
– Multi-rooted tree
– GIIS caching is not used
by the clients
– Several patches and bug
fixes are applied
 A new schema is
developed, to serve
clusters
– Clusters are expected to
be fairly homogeneous
2003-10-23
[email protected]
www.nordugrid.org
8
Front-end and the Grid Manager
 Grid Manager replaces Globus’ GRAM, still using Globus
ToolkitTM 2 libraries
 All transfers are made via GridFTP
 Added a possibility to pre- and post-stage files, optionally
using Replica Catalog information
 Caching of pre-staged files is enabled
 Runtime environment support
2003-10-23
[email protected]
www.nordugrid.org
9
Summary of Grid services on the
front-end machine
 GridFTP server
– Plugin for job submission via a virtual
directory
– Conventional file access with Grid access
control
 LDAP server for information services
 Grid Manager
2003-10-23
[email protected]
www.nordugrid.org
10
The User Interface

Provides a set of utilities to be invoked from the command line:
ngsub
ngstat
ngcat
ngget
ngkill
ngclean
ngrenew
ngsync
ngcopy
ngremove

Contains a broker that polls MDS and decides to which queue at which cluster a
job should be submitted
–
–
–
–
–
2003-10-23
to submit a task
to obtain the status of jobs and clusters
to display the stdout or stderr of a running job
to retrieve the result from a finished job
to cancel a job request
to delete a job from a remote cluster
to renew user’s proxy
to synchronize the local job info with the MDS
to transfer files to, from and between clusters
to remove files
The user must be authorized to use the cluster and the queue
The cluster’s and queue’s characteristics must match the requirements specified in the
xRSL string (max CPU time, required free disk space, installed software etc)
If the job requires a file that is registered in a Replica Catalog, the brokering gives priority
to clusters where a copy of the file is already present
From all queues that fulfills the criteria one is chosen randomly, with a weight proportional
to the number of free CPUs available for the user in each queue
If there are no available CPUs in any of the queues, the job is submitted to the queue with
the lowest number of queued job per processor
[email protected]
www.nordugrid.org
11
Job Description: extended
Globus RSL
(&(executable="recon.gen.v5.NG")
(arguments="dc1.002000.lumi02.01101.hlt.pythia_jet_17.zebra" "dc1.002000.lumi02.recon.007.01101.hlt.pythia_jet_17.eg7.602.ntuple"
"eg7.602.job" “999")
(stdout="dc1.002000.lumi02.recon.007.01101.hlt.pythia_jet_17.eg7.602.log")
(stdlog="gridlog.txt")(join="yes")
(
|(&(|(cluster="farm.hep.lu.se")(cluster="lscf.nbi.dk")(*cluster="seth.hpc2n.umu.se"*)(cluster="login-3.monolith.nsc.liu.se"))
(inputfiles= ("dc1.002000.lumi02.01101.hlt.pythia_jet_17.zebra"
"rc://grid.uio.no/lc=dc1.lumi02.002000,rc=NorduGrid,dc=nordugrid,dc=org/zebra/dc1.002000.lumi02.01101.hlt.pythia_jet_17.zebra")
("recon.gen.v5.NG" "http://www.nordugrid.org/applications/dc1/recon/recon.gen.v5.NG.db")
("eg7.602.job" "http://www.nordugrid.org/applications/dc1/recon/eg7.602.job.db")
("noisedb.tgz" "http://www.nordugrid.org/applications/dc1/recon/noisedb.tgz"))
)
(inputfiles= ("dc1.002000.lumi02.01101.hlt.pythia_jet_17.zebra"
"rc://grid.uio.no/lc=dc1.lumi02.002000,rc=NorduGrid,dc=nordugrid,dc=org/zebra/dc1.002000.lumi02.01101.hlt.pythia_jet_17.zebra")
("recon.gen.v5.NG" "http://www.nordugrid.org/applications/dc1/recon/recon.gen.v5.NG")
("eg7.602.job" "http://www.nordugrid.org/applications/dc1/recon/eg7.602.job"))
)
(outputFiles= ("dc1.002000.lumi02.recon.007.01101.hlt.pythia_jet_17.eg7.602.log"
"rc://grid.uio.no/lc=dc1.lumi02.recon.002000,rc=NorduGrid,dc=nordugrid,dc=org/log/dc1.002000.lumi02.recon.007.01101.hlt.pythia_jet_17.e
g7.602.log")
("histo.hbook"
"rc://grid.uio.no/lc=dc1.lumi02.recon.002000,rc=NorduGrid,dc=nordugrid,dc=org/histo/dc1.002000.lumi02.recon.007.01101.hlt.pythia_jet_17.
eg7.602.histo")
("dc1.002000.lumi02.recon.007.01101.hlt.pythia_jet_17.eg7.602.ntuple"
"rc://grid.uio.no/lc=dc1.lumi02.recon.002000,rc=NorduGrid,dc=nordugrid,dc=org/ntuple/dc1.002000.lumi02.recon.007.01101.hlt.pythia_jet_1
7.eg7.602.ntuple"))
(jobname="dc1.002000.lumi02.recon.007.01101.hlt.pythia_jet_17.eg7.602")
(runTimeEnvironment="ATLAS-6.0.2")
(CpuTime=1440)(Disk=3000)(ftpThreads=10))
2003-10-23
[email protected]
www.nordugrid.org
12
Task flow
Cluster A
RSL
Gatekeeper
GridFTP
B!
Front-end
Grid
Manager
2003-10-23
[email protected]
www.nordugrid.org
Cluster B
13
Performance

The main load: “ATLAS Data Challenge 1” (DC1)
–
–
–
–
April 5th 2002: first job submitted
May 10th 2002: first pre-DC1-validation-job
End-May 2002: now clear that the system is mature enough to do and
manage real production.
DC1, phase1 (detector simulation):
•
•
•
–
DC1, phase2 (pile-up of data):
•
•
–
–
2003-10-23
2150 jobs, 5-6 hours of processing 1 GB of input each
Other applications:
–
–

Piling up the events above with a background signal
1300 jobs, ca. 4 hours each
DC1, phase3 (reconstruction of signal)
•

Total number of jobs: 1300, ca. 24 hours of processing 2 GB of input each
Total output size: 762 GB
All files uploaded to Storage Elements and registered in the Replica Catalog.
Calculations for string fragmentation models (Quantum Chromodynamics)
Quantum lattice models calculations (sustained load of 150+ long jobs at any
given moment for several days)
Particle physics analysis and modeling
At peak production, up to 500 jobs were managed by the NorduGrid at
the same time
[email protected]
www.nordugrid.org
14
What is needed for installation


A cluster or even a single machine
For a server:
– Any Linux flavor (binary RPMs exist for RedHat and Mandrake, ev. for
Debian)
– A local resource management system, e.g., PBS
– Globus installation (NorduGrid has an own distribution in a single
RPM)
– Host certificate (and user certificates)
– Some open ports (depends on the cluster size)
– One day to go through all the configuration details

The owner always retains a full control
– Installing NorduGrid does not give automatic access to the resources
– And other way around
– But with a bit of negotiations, one can get access to very considerable
resources on a very good network

2003-10-23
Current stable release is 0.3.28; daily CVS snapshots are
available
[email protected]
www.nordugrid.org
15
Summary
 NorduGrid pre-release (currently 0.3.28) works reliably
 Release 1.0 is slowly but surely on its way; many fixes are
still needed
 We welcome developers: much functionality is still
missing, such as:
–
–
–
–
–
–
Bookkeeping, accounting
Group- and role-based authorization
Scalable resource discovery and monitoring service
Interactive tasks
Integrated, scalable and reliable data management
Interfaces to other resource management systems
 We welcome new users and resources
– Nordic Data Grid Facility will provide support
2003-10-23
[email protected]
www.nordugrid.org
16