Transcript HEPFARMA

Tier 2 Prague
Institute of Physics AS CR
Status and Outlook
J. Chudoba, M. Elias, L. Fiala, J. Horky,
T. Kouba, J. Kundrat, M. Lokajicek,
J. Svec, P. Tylka
13. September 2013
NEC2013 Varna
M. Lokajicek
1
Outline
•
•
•
•
•
Institute of Physics AS CR (FZU)
Computing Cluster
Networking
LHCONE
Looking for new resources
– CESNET National Storage Facility
– IT4I supercomputing project
• Outlook
11 September 2013
2
Institute of Physics AS CR (FZU)
• Institute of Physics of the Academy
of the Czech Republic
• 2 locations in Prague, 1 in Olomouc
– In 2012: 786 employees (281 researchers + 78 doctoral
students)
– 6 Divisions
•
•
•
•
•
•
Division of Elementary Particle Physics
Division of Condensed Matter Physics
Division of Solid State Physics
Division of Optics
Division of High Power Systems
ELI Beamlines Project Division
• Department of Networking and Computing Techniques (SAVT)
11 September 2013
3
FZU - SAVT
• Institute’s networking and computing service
department
– Several server rooms
– Computing clusters
Golias – Particle physics, Tier2
•
•
•
•
Few nodes from already before EDG
WLCG iMoU 4 July 2003 (interim)
New server room 1 November 2004
WLCG MoU from 28 April 2008
– Dorje – solid state, condensed matter
Luna, Thsun, smaller group clusters
11 September 2013
4
Main server room
• Main server room (in FZU, Na Slovance)
– 62 m2, ~20 racks, 350 kVA motor generator,
200 + 2 x 100 kVA UPS,
108 kW air cooling, 176 kW water cooling
– continuous changes
– hosts computing servers and central services
11 September 2013
5
11 September 2013
6
Cluster Golias
• Upgraded every year
several (9) sub-clusters of the identical
HW
• 3800 cores, 30 700 HS06
• 2 PB disk space
• Tapes used only for local backups (125
LTO4, max 500 cassettes)
• Serving:
ATLAS, ALICE, D0 (NOvA), Auger, STAR, …
• WLCG Tier2
Golias@FZU + xrootd servers@REZ (NPI)
11 September 2013
7
Utilization
• Very high average utilization
–
–
–
–
Several different projects, different tools for production
D0 – production submitted locally by 1 user
ATLAS – panda, ganga, local users; DPM
ALICE – VO box; xrootd
3.5 k
D0
ATLAS
ALICE
11 September 2013
8
„RAW“ Capacities
HEPSPEC2006
%
TB disk
%
2009
10 340
2010
19 064
100
427
100
2011
23 484
100
1 714
100
2012
29 660
100
2521
100
D0
9 993
34
35
1
ATLAS
12 127
41
1880 (+16 MFF)
74
ALICE
7 540
25
606(+100 Řež)
24
2013
29 660
100
2521
100
D0
9 993
34
35
1
ATLAS
12 127
41
1880 (+16 MFF)
74
ALICE
7540
25
606(+140 Řež)
24
11 September 2013
186
9
2012 D0, ATLAS and ALICE usage
•
•
•
•
D0
290 M tasks
90 MHEPSPEC06 hours
13% contribution to D0
2012 Data transfers inside farm - month means
to and from working nodes in TB
3000.0
2500.0
2000.0
1500.0
incomming
1000.0
outgoing
500.0
•
•
•
•
•
2012-11
2012-10
2012-09
2012-08
2012-07
2012-06
2012-05
2012-04
2012-03
2012-02
2012-01
0.0
ATLAS
2,2 M tasks
90 MHEPSEPC06 hours,
1,9 PB disk space
Data transfer 1,2 PB to farm
0,9 PB from farm
•
•
•
•
•
ALICE
2 M simulation tasks
60 MHEPSEC06 hours
Data transfer 4,7 PB to farm and 0,5 PB from
farm
5% our contribution to ALICE tasks processing
2% contribution to ATLAS
•
140TB disk space in INF (Tier3)
11 September 2013
10
11 September 2013
11
Network - CESNET, z. s. p. o.
•
FZU Tier2 Network connections
– 10 Gbps LHCONE (GEANT), 18 July 2013
– 10 Gbps KIT from 1st Sept 2013
– 1 Gbps FNAL, BNL, Taipei
– 10 Gbps to commodity network
– 1-10 Gbps to Tier3 collaborating
institutes
http://netreport.cesnet.cz/netreport/hep-cesnet-experimental-facility2/
11 September 2013
12
LHCONE - Network transition
• Link to KIT saturated at 1
Gbps E2E line
• LHCONE from 18 July
2013 over 10 Gbps
infrastructure
• Relieves also the
commodity network
10 Gbps
11 September 2013
13
Atlas tests
• Testing upload speed of
files > 1 GB to all Tier1
centra
30
• After LHCONE connection
only 2 sites with < 5MB/s
• Prague Tier2 ready for
validation as T2D
60
11 September 2013
14
LHCONE – trying to understand monitoring
• L
Prague – DESY
Very asymmetric throughput
LHCONE line cut
DESY – Prague
LHCONE optical line cut
At 4:00
One way latency improved
11 September 2013
15
International contribution of Prague center to
ATLAS + ALICE centra T2 LCG
6
5
4
3
jobs %
2
cpu %
1
0
2005
2006
2007
2008
• http://accounting.egi.eu/
• Grid + local tasks
• Long term slide down until we
received regular financing in 2008
11 September 2013
2009
2010
2011
2012
• Original 3% target is not
achievable with current financial
resources
• Necessary to look for other
resources
16
Remote storage
•
•
CESNET - Czech NREN + other services
New project: National storage facility
•
•
Three distributed HSM based storage
sites
Designed for research and science
community
–
–
–
100TB for both ATLAS and Auger
experiments offered
Implemented as remote Storage
Element with dCache
disk <-> tape migration
FZU Tier-2 in Prague
FZU<->Pilsen - 10Gbit link with ~3.5ms latency
CESNET storage site in Pilsen
11 September 2013
17
Remote storage
Influence of distributing a Tier-2 data storage on physics analysis
CPU
Bytes transferred % Efficiency
remote/local Method
TTreeCache events/s ( %)
local
rfio
ON
100%
117%
98,9%
local
rfio
OFF
74%
100%
72,7%
remote
dCap
ON
75%
101%
73,5%
remote
dCap
OFF
46%
100%
46,9%
• TTreeCache in ROOT helps a lot – both for local and for remote
transfers
• TTreeCached remote jobs faster than local ones without the cache
11 September 2013
18
Outlook
• In 2015 after LHC start up
– Higher data production
– Flat financing not sufficient
– Computing can become an item of
M&O A (Maintenance &Operations
cat. A)
• Search for new financial
resources or new unpaid
capacities necessary
– CESNET
• Crucial free delivery of network
infrastructure
• Unpaid External storage, how long?
– IT4I, Czech supercomputing project
search for computing capacities
(free cycles), relying on other
project to find the way how to use
them
11 September 2013
19
• 16th International
workshop on Advanced
Computing and
Analysis Techniques in
physics (ACAT)
• http://www.particle.cz/
acat2014
11 September 2013
•
Topics
• Computing Technology
for Physics Research
• Data Analysis Algorithms and Tools
• Computations in
Theoretical Physics:
Techniques and
Methods
20
Backup
11 September 2013
21