Transcript Computing and Grid used by BABAR, Tevatron and LHC

(European) Networking infrastructure needs for
Particle Physics
Matthias Kasemann
DESY and CERN
Serenate Users Workshop
January 17-19, 2003
HEP institutes (from PDG, 2002)
Country
Country
Country
Country
Country
Argentina
6
Colombia
4
India
17
Pakistan
1
Taiwan
5
Armenia
1
Costa Rica
1
Indonesia
2
Peru
2
Ukraine
5
Australia
8
Croatia
2
Iran
3
Poland
7
UK
24
Austria
6
Cuba
1
Ireland
3
Portugal
5
Uruguay
1
Azerbeijan
2
Czech Rep.
4
Israel
6
Romania
3
USA
166
Belarus
1
Denmark
3
Italy
33
Russia
15
Uzbekistan
2
Belgium
8
Egypt
2
Japan
37
Slovakia
4
Venezuela
2
Brazil
10
Finland
4
Korea
15
Slovenia
2
Vietnam
3
Bulgaria
2
France
30
Mexico
8
South Africa
2
Yugoslavia
3
Canada
22
Georgia
3
Mongolia
1
Spain
11
Chile
5
Germany
39
Netherlands
6
Sweden
6
China
22
Hungary
3
Norway
3
Switzerland
10
602 institutes, collaborating at ~10
January 17, 2003
HEP networking needs
M. Kasemann
accelerator centers
around
the world2/20
From Physics to Raw Data:
what happens in a detector
e+
f
Z0
e-
250Kb – 1 Mb
2037 2446 1733 1699
4003 3611 952 1328
2132 1870 2093 3271
4732 1102 2491 3216
2421 1211 2319 2133
3451 1942 1121 3429
3742 1288 2343 7142
_
f
Raw data
Detector
Fragmentation, Interaction with
detector material response
(Bytes)
Decay
Multiple scattering,Noise, pile-up,
cross-talk,
interactions
Read-out
inefficiency, addresses,
ambiguity,
Particle production and decays observed in
ADC, TDC
resolution,
detectors are Quantum Mechanical processes.
values,
response
Hundreds or thousands of different productionBit patterns
function,
and decay-channels possible, all with different
alignment,
probabilities.
temperature
In the end all we measure are probabilities!!
Theoretical
Model of
Particle
interaction
January 17, 2003
HEP networking needs
M. Kasemann
3/20
From Raw Data to Physics:
what
happens
during
analysis
250Kb – 1 Mb
100 Kb
25 Kb
5 Kb
e+
2037 2446 1733 1699
4003 3611 952 1328
2132 1870 2093 3271
4732 1102 2491 3216
2421 1211 2319 2133
3451 1942 1121 3429
3742 1288 2343 7142
Raw data
Convert to
physics
quantities
500 b
f
Z0
eDetector
response
apply
calibration,
alignment,
Interaction with Fragmentation, Basic physics
detector material Decay
Pattern,
Physics
Results
recognition,
analysis
Particle
identification
Analysis
Reconstruction
Simulation (Monte-Carlo)
January 17, 2003
_
f
HEP networking needs
M. Kasemann
4/20
Particle Physics Computing Challenges
Geographical dispersion:
Complexity:
Scale:
of people and resources
the detector and the data
Petabytes per year of data per experiment
Example: CMS Experiment
1750 Physicists
150 Institutes
32 Countries
Major challenges associated with:
Communication and collaboration at a distance
Distributed computing resources
Remote software development and physics analysis
January 17, 2003
HEP networking needs
M. Kasemann
5/20
LHC Data reduction and recording:
ATLAS, CMS, ALICE, LHCb
On-line System
•
•
•
•
protons
Multi-level trigger
Filter out background
Reduce data volume
24 x 7 operation
antiprotons
CMS
January 17, 2003
HEP networking needs
M. Kasemann
6/20
Goal for LHC analysis
Example:
ATLAS guiding principles
(true for all LHC experiments):

Every physicist in ATLAS must have the best possible access to
the data necessary for the analysis, irrespective of his/her
location.

The access to the data should be transparent and efficient.

We should profit from resources (money, manpower and
hardware) available in the different countries.

We should benefit from the outcome of the Grid projects.
January 17, 2003
HEP networking needs
M. Kasemann
7/20
Computing for the LHC
experiments
A new Project has been setup at CERN:
the LHC Grid Computing Project (LCG)
The first phase of the project: 2002-2005
preparing the prototype computing environment, including
 support for applications – libraries, tools, frameworks,
common developments, …..
 global grid computing service
Shared funding by Regional Centers, CERN, Contributions
Grid software developments by national and regional Grid projects
Phase 2: 2005-2007
construction and operation of the initial LHC Computing Service
January 17, 2003
HEP networking needs
M. Kasemann
8/20
Distributed Analysis must work
Summary of Computing Capacity Required for
all LHC Experiments in 2008
-------------- CERN -------------Tier 0
Tier 1
Total
Processing (K SI2000)
Disk (PetaBytes)
Magnetic tape (PetaBytes)
12,000
1.1
12.3
8,000
1.0
1.2
20,000
2.1
13.5
Other
Tier 1
Total
Tier 1
CERN as
% of Tier 1
Total
Tier 0 + 1
CERN as
% of total
Tier 0 + 1
49,000
8.7
20.3
57,000
9.7
21.6
14%
10%
6%
69,000
10.8
33.9
29%
20%
40%
CERN will provide the data reconstruction & recording service (Tier 0)
-- but only a small part of the analysis capacity
current planning for capacity at CERN + principal Regional Centers


2002: 650 KSI2000  <1% of capacity required n 2008
2005: 6,600 KSI2000  < 10% of 2008 capacity
January 17, 2003
HEP networking needs
M. Kasemann
9/20
Centers taking part in LCG-1
Centers that have declared resources –
December 2002
Tier 0
CERN
Tier 1 Centers
Brookhaven National
Lab
CNAF Bologna
Fermilab
FZK Karlsruhe
IN2P3 Lyon
Rutherford Appleton
Lab (UK)
University of Tokyo
CERN
January 17, 2003
Other Centers
Academica Sinica (Taipei)
Barcelona
Caltech
GSI Darmstadt
Italian Tier 2s(Torino, Milano,
Legnaro)
Manno (Switzerland)
Moscow State University
NIKHEF Amsterdam
Ohio Supercomputing Centre
Sweden (NorduGrid)
Tata Institute (India)
Triumf (Canada)
UCSD
UK Tier 2s
University of Florida–
Gainesville
University of Prague
HEP networking needs……
M. Kasemann
10/20
planned LHC computing resources, Q4 2003 (status: 12/2002)
1200
Support
FTE's
CPU [KSi2k]
Tape [TB]
Disk [TB]
Support [FTE]
1000
70
60
Si95 ~ 9 Si2k
800
50
40
600
30
400
20
200
10
0
0
SA
M. Kasemann
U
R
HEP networking needs
K
U
an
iw
Ta
a
si
us
R
nd
la
Po
n
pa
Ja
N
ch
ze
ER
ly
Ita
nd
la
ol
H
y
an
m
er
G
ce
an
.
Fr
ep
C
C
January 17, 2003
11/20
Target for the end of the decade
LHC data analysis using
“global collaborative environments integrating largescale, globally distributed computational systems and
complex data collections linking tens of thousands of
computers and hundreds of terabytes of storage”
The researchers concentrating on science, unaware of the
details and complexity of the environment they are
exploiting
Success will be when the scientist does not mention the
Grid
January 17, 2003
HEP networking needs
M. Kasemann
12/20
Data Grid for LHC experiments
CERN/Outside Resource Ratio ~1:2
Tier0/( Tier1)/( Tier2)
~1:1:1
Experiment
~PBytes/sec
Online System
Bunch crossing per 25 nsecs.
100 triggers per second
Event is ~1 MByte in size
France Center
~100 MBytes/sec
Tier 0 +1
2.5 Gbits/sec
CERN Computer
Center > 20 TIPS
Italy Center
UK Center
USA Center
Tier 1
0.15 -2.5 Gbits/sec
Tier 2
Tier 3
~622 Mbits/sec
InstituteInstitute Institute
~0.25TIPS
Physics data cache
Workstations,
other portals
January 17, 2003
Tier2 Center
Tier2 Center
Tier2 Center
Tier2 Center
Tier2 Center
Institute
100 - 1000
Mbits/sec
Tier 4
Physicists work on analysis “channels”.
Each institute has ~10 physicists
working on one or more channels
HEP networking needs
M. Kasemann
13/20
Building a Grid for LHC
Collaborating
Computer
Centers
January 17, 2003
HEP networking needs
M. Kasemann
14/20
Building a Grid for LHC
The “virtual” LHC Computing Center
Collaborating
Computer
Centers
Alice VO
CMS VO
January 17, 2003
HEP networking needs
M. Kasemann
15/20
HENP Major Links:
Bandwidth Roadmap (Scenario) in Gbps
Year
Production
Experimental
Remarks
2001
0.155
0.622-2.5
SONET/SDH
2002
0.622
2.5
SONET/SDH
DWDM; GigE Integ.
2003
2.5
10
DWDM; 1 + 10 GigE
Integration
2005
10
2-4 X 10
 Switch;
 Provisioning
2007
2-4 X 10
~10 X 10;
40 Gbps
1st Gen.  Grids
2009
~10 X 10
or 1-2 X 40
~5 X 40 or
~20-50 X 10
40 Gbps 
Switching
2011
~5 X 40 or
~20 X 10
~25 X 40 or
~100 X 10
2nd Gen  Grids
Terabit Networks
~Terabit
~MultiTerabit
~Fill One Fiber
2013
January 17, 2003
HEP networking
needs
Kasemann
From: ICFA
SCIC, H.
Newman,M.
Feb,
2002
16/20
The “virtual” LHC Computing
Center
The aim is to build
 a general computing service  for a very large user population  of independently-minded scientists  using a large number of independently managed sites!
This is NOT a collection of sites providing pre-defined services
 it is the user’s job that defines the service
 it is current research interests that define the workload
 it is the workload that defines the data distribution
DEMAND - Unpredictable & Chaotic
But the SERVICE had better be Available & Reliable
January 17, 2003
HEP networking needs
M. Kasemann
17/20
HENP Lambda Grids:
Fibers for Physics
Analysis Problem:


Extract “Small” Data Subsets of 1 to 100 Terabytes from 1 to 1000
Petabyte Data Stores
Survivability of the HENP Global Grid System, with hundreds of such
transactions per day (by ~ 2007) requires that each transaction be
completed in a relatively short time.
Example: Take 800 secs to complete the transaction. Then

Transaction Size (TB)
1
10
100
Net Throughput (Gbps)
10
100
1000
(Capacity of Fiber Today)
Summary: Providing Switching of 10 Gbps wavelengths within
~3 years; and Terabit Switching within 5-7 years would enable
“Petascale Grids with Terabyte transactions”, as required to fully
realize the discovery potential of major HENP programs, as well as
other data-intensive fields.
January 17, 2003
HEP networking needs
M. Kasemann
18/20
A major concern: the Digital Divide
In an era of global collaborations, it is a particular concern that
Digital Divide problems will delay and in some cases prevent
physicists in the economically less favored regions of the world from
participating effectively as equals in their experiments.
One can decompose the Digital Divide Problems into three
components:



the Long Range (wide area) Connection,
the Last Mile Connection and
the Institution Internal Network.
If one of these components presents a bandwidth bottleneck, it will
not be possible to effectively exploit the new generation of Grid
architectures that are being designed and implemented to empower
physicists in all world regions and to work with the data of the
experiments like LHC.
Pointed out by:
The International Committee for Future Accelerators – Standing
Committee on Inter-Regional Connectivity (ICFA SCIC)
subcommittee on the Digital Divide
January 17, 2003
HEP networking needs
M. Kasemann
19/20
HEP Computing & Networking needs:
Summary
LHC computing based on models emerging and experiences gained in
current HEP experiments
Experiments need substantial resources to perform computing and analysis
(10 - 20% of detector costs)
Central laboratories not able to provide all required resources

It is essential to fund and operate this in a distributed way (using Grid ideas and
technology)
LHC experiments will need a substantial amount of computing and starts
the Exabyte Era for event storage.

To work in this computing world, the GRID is a mandatory tool. And in
consequence, networks at compatible speeds connecting institutes.
Bandwidths that allow working at long distance with distributed data are
essential.
Infrastructure bottlenecks can not be tolerated or the whole system will not
work well.
January 17, 2003
HEP networking needs
M. Kasemann
20/20