Transcript Status AliWeek0900

DataGrid Applications
Federico Carminati
WP6 WorkShop
December 11, 2000
The distributed
computing model
Assumptions
Raw data will be kept at CERN – (backup if affordable)
Tier 1 will have ~10% of raw data
Reconstruction will be done at CERN (2 passes)
ESD (aka DST) and TAG will be shipped to Tier1-2
Simulation will all be done in the Tier1-2, data will be shipped to
CERN (in which form? Raw, ESD, AOD)
The Tier1-2 will process ESD to produce AOD (formerly aka ntuple) as many time as necessary, producing their own TAG
Users will access the ESD/AOD/TAG remotely
11 December
2000
DataGrid WP6 Workshop
2
The distributed
computing model
Basic principle
Every physicist should have in principle equal access to the
data and to the resources
The system will be extremely complex
Number of components in each site
Number of sites
Different tasks performed in parallel: simulation,
reconstruction, scheduled and unscheduled analysis
11 December
2000
DataGrid WP6 Workshop
3
WP 8 philosophy
Define a common upper middle-layer of GRID
services common to the 4 experiments
Common API for common tasks
File replica
Job submission and monitoring
….
Collaborate to define a common set of
requirements and milestones, also with WP 9/10
Share the same testbeds and facilities for data
challenges
Introduce a user view in the project
11 December
2000
DataGrid WP6 Workshop
4
WP8 plans and
requirements
Testbed Release 0 (1Q 2001)
A working, standard, installation of Globus, at CERN and at
other labs.
Standard recipes for file transfer and job submission (at
CERN, this installation will probably be interfaced to LSF).
A contact point for GLOBUS and later DataGRID software in
each lab incase of problems
A clear policy for "experiment-wide" authorisation to allow
testing across national boundaries.
This will only work if enough support is provided to users
WP8 activities in general require to have a substantial
part of the GRID services available since as soon as
possible
11 December
2000
DataGrid WP6 Workshop
5
WP8 plans and
requirements
Testbed release 1 (3Q 2001)
Distributed user autentication and resource allocation or preallocation, as dynamic allocation can come later.
Distributed data dictionary (location of files on different servers).
Basic network configuration, monitoring and diagnostic tools.
Basic monitoring and diagnostic of a cluster of PC's.
Distributed scheduling for the jobs that are submitted in a
coordinated way (this does not include the "chaotic job activity"
coming from isolated users which should be addressed in the
subsequent release).
Access to the basic information about job status and errors.
Guidelines for configurating farms of PC's with fast disk access.
11 December
2000
DataGrid WP6 Workshop
6
WP8 plans and
requirements
Testbed release 2 (3Q 2002)
Replica management and network optimised trasfer of data
from different file systems.
Tools for configurating farms of PC's with fast disk access,
monitoring their status and for automathized s/w installation
and management.
A prototype of scheduling and load balancing for chaotic
analysis jobs
Basic functionality for dynamic resource allocation
Basic functionality for job partitioning
11 December
2000
DataGrid WP6 Workshop
7
WP8 plans and
requirements
Testbed release 3 (3Q 2003)
Scheduling and balancing of chaotic analysis
Tools to ensure "robustness" and error recovery of the
system
11 December
2000
DataGrid WP6 Workshop
8
WP8 Wkshop Nov 16
Conclusions
WP8 non-experiment-specific personpower
Needed to implement WP8 common policies
CERN person identified and hired (I.Augustin)
Should come from the funded contribution of each partner
(CNRS, CERN, PPARC, NIKHEF and INFN) as a share of
the 60 funded person-months
A message sent to the PMB for each partner to identify this
personpower
11 December
2000
DataGrid WP6 Workshop
9
WP8 Wkshop Nov 16
Conclusions
Two kinds of Test-bed participation.
Formal participation defined by WP6
Informal commitment to install Datagrid provided tools, and participate
in tests.
Kick-start and Update kits
Experiments to provide WP8 with installation and upgrade kits
WP8 personpower will install them in the WP6 test-beds locations
Kits have to coexist on the same machines without interference
Locations participating but not in WP6 will provide their own
personpower for the installation
This activity will be coordinated by the CERN WP8 person (I.Augustin)
11 December
2000
DataGrid WP6 Workshop
10
WP8 Wkshop Nov 16
Conclusions
Collection of requirements
Present WP8 requirements judged vague by other WP's (rightly!)
Other WP's should have asked WP8/9/10 questions – they didn’t
Meeting of December 1st of the ATF was rather inconclusive
New strategy: WP8-10 to produce a three tiered document
Short term use cases
Long term use cases
General requirements
WP8-10 also to produce pilot applications
December 15 ATF should consolidate user requirements
DataGRID Workshop on January 15
US experts invited to discuss user requirements and first proposition of
architecture
11 December
2000
DataGrid WP6 Workshop
11
WP8 Wkshop Nov 16
Conclusions
Set up a technical WP8 technical WG
one application software expert from each experiment and
from ESA (WP9) and biology (WP10)
the WP8 experiment-neutral personpower in the different
partners (5 people)
Chaired by the CERN WP8 person acting as WP8
architect)
11 December
2000
DataGrid WP6 Workshop
12
WP8 Wkshop Nov 16
Conclusions
Main tasks
Help WP8 architect to collect the requirements from the
experiment and ESA and Biology for the ATF
Liaise with the Middleware WP's (1-5) regarding the services
required by the applications, and the definition of the appropriate
interfaces.
Discuss WP8 architectural questions
Do WP8-10 require a common 'upper middleware' layer over
middleware services?
Do they want to interface to generic middleware services directly?
Definition of the 'sample' applications
CMS and LHCb have existing running distributed applications
ALICE is following and ATLAS will see whether it can add its
Define requirements for the Test-bed and Networking
11 December
2000
DataGrid WP6 Workshop
13
WP8 Wkshop Nov 16
Conclusions
Main tasks (cont)
Provide technical liaison with the Test-bed and Networking
workpackages (e.g. attend their meetings)
Special priority points are, for instance
1.
2.
3.
Provision of Standard, Supported, Documented Globus installation kit
Definition of Test-bed sites and contact people
Definition of Certification system to be employed by the project
Compact sub-group
Will meet, virtually or in person, in its own right, preparing our
point of view for the global Test-bed and Architecture meetings
Experiment TB reps, unless the same person, would normally not
attend, unless required for a forthcoming TB meeting
The frequency of meetings decided by WP8 architect and
coordinator.
11 December
2000
DataGrid WP6 Workshop
14
Conclusions
Activity of WP8 well started
Main concerns are
Architecture design seems to start with difficulty
We do not have an architect yet
Coordination with WP9-10 not yet very effective
Need better communication with WP1-5
Need to start interacting more effectively with WP6-7
11 December
2000
DataGrid WP6 Workshop
15