Transcript Slide 1

WLCG
Service Challenge 4:
Preparation, Planning and
Outstanding Issues at INFN
[email protected]
Workshop sul Calcolo e Reti dell'INFN
Jun 7, 2006
1
Outline
1.
2.
3.
4.
5.
6.
WLCG
Service Challenge 4: overall schedule
SC4 service planning at T1 and T2 sites
Results of pre-SC4 testing at INFN
Accounting results
VO Box
Outstanding issues
2
SC Milestones 2006
January
February
March
April
May
June
July
August
September
October
November
December
WLCG
SC3 disk repeat – Nominal rate (200 MB/s) capped at 150MB/s/
CHEP Workshop; T1-T1 Use Cases,
SC3 disk - tape repeat (50MB/s, 5 drives)
Detailed plan for SC4 service agreed (M/W + DM service
enhancements).
gLite 3.0 release beta testing
SC4 disk – disk (200 MB/s) and disk – tape (75 MB/s) throughput tests.
gLite 3.0 release available for distribution
Installation, configuration and testing of gLite 3.0 release at sites
Start of SC4 production tests by experiments of ‘T1 Use Cases’
T2 Workshop: identification of key Use Cases and Milestones for T2s
Tape throughput tests at full nominal rates!
T2 Milestones – debugging of tape results if needed
LHCC review – rerun of tape tests if required
WLCG Service Officially opened. Capacity continues to build up
1st WLCG ‘conference’. All sites have network / tape h/w
in production(?)
Final service / middleware review leading to early 2007
upgrades for LHC data taking??
3
INFN sites in SC4
WLCG
• ALICE
–
–
–
–
Catania
Torino
(Bari TBD)
(Legnaro TBD)
• ATLAS
–
–
–
–
Frascati (in)
Napoli
Milano (sj)
Roma
• CMS
–
–
–
–
Bari (in)
Legnaro
Pisa (sj)
Roma
• LHCb
– CNAF
4
COMPONENT
Tier-1
Tier-2
Comments
ATLAS, LHCb
-
ALICE: server instance only needed either at T0 or T1
WLCG
CMS: server at T0
-
-
ALICE: server instance only needed either at T0 or T1
CMS/LHCb: server at T0
BDII/GLUE
ALICE
ALICE
FTS server
ALL
-
Local File
Catalogue
ALICE, ATLAS, LHCb
(CMS: backup solution)
ALICE
Lcg-utils
ALL
ALL
ATLAS, LHCb
ATLAS, LHCb
ALICE,
(ATLAS),
(LHCb)
ALICE
ATLAS: at T0, but some instances in a few T1 sites for
redundancy purposes, would be useful
CMS: server at T0 can suffice
LHCb: at T0, both LCG and glite versions (another
instance, in some T1s may be useful for redundancy)
ALL
ALL
ALICE: CE gLite not necessarily required in every T1
and T2
CMS: CE gLite only at T1
LHCb: CE classic only
ALICE, ATLAS
ALICE, ATLAS
SRM 1.1
ALL
ALL
VOBOX
ALICE, ATLAS
CMS, LHCb
ALICE, CMS
ALICE, ATLAS, CMS
ALICE, ATLAS, CMS
SC4 gLite 3.0 Service Planning
VOMS server
Myproxy server
GFAL
Resource Broker
Glite 3.0
CE: classic
and gLite 3.0
RGMA/GridICE
APEL/DGAS
CMS/LHCb: server at CERN
LHCb: in Lyon, RAL e CNAF, a local read-only replica
of the central master is required (to be synchronized
every 30’)
CMS: using SRM 2.1 (dCache) in US
ALICE: VOBOX + gLite UI
5
SC4 Preparation
WLCG
• Activities:
– Mar: Installation and testing of gLite 3.0 in PPS 
passed, with the exception of the FTS service (Oracle
backend), under finalization
– Apr, May: testing of CERN to CNAF disk-disk
throughput target rate (200 MB/s) with Castor2 
ongoing
– May: testing of CERN to CNAF disk-tape throughput
target rate (75 MB/s) with Castor2  passed
– May 22: start of deployment of gLite 3.0.0 RC5
(released on May 4) at CNAF (all WN, CE gLite, CE
LCG still waiting for upgrade, some UIs, FTS)
6
Disk – Tape Throughput (CERN – CNAF)
WLCG
• Tue Apr 19 to Thu Apr 27: estimated overall rate to tape 76.94 MB/s
• Target: 75 MB/s
• Estimated Daily Rate to Tape (MB/s):
Wed
Thu
Fri
Sat
Sun
Mon
Tue
Wed
Thu
19
20
21
22
23
24
25
26
27
5.32
103.80
148.00
25.46
11.60
62.50
62.50
91.34
182.00
MB/s
MB/s
MB/s
MB/s
MB/s
MB/s
MB/s
MB/s
MB/s
• 6 tape drives
• long down times of the WAN transfer sessions from CERN to CNAF
due to LSF Castor2 issues already experienced during the disk-disk
throughput phase in April
7
Disk-disk throughput (CERN – CNAF) 1/3
•
•
•
•
•
•
WLCG
Apr: LSF issues caused instability and seriously limited the avg throughout
May 2: Castor2 upgrade to version 2.0.4-0;
May 3: local write tests and remote transfers (few concurrent file transfers). Results
show good local Castor-2-to-LSF interaction. Power problem in the Tier-1 premises in
the afternoon;
May 4: transfers re-activated by gradually increasing the number of concurrent transfers
by steps of 10 files. Throughput increases linearly (10 files -> 100 MB/s, 40 files -> 200
MB/s)
Problems with the name server DB (this service is shared by Castor1 and Castor2). Tests
with 50 concurrent files, 2 parallel streams: 1800 Mb/s (200 MB/s)
Stable run at 200 MB/s for approximately 1 and a half day (until May 6, 8 p.m.)
8
Disk-disk throughput (CERN – CNAF) 2/3
WLCG
• Average
always around
170 MB/s 
bottleneck
exists, still to
be understood
• Daily statistics
(May 26)
9
Disk-disk throughput (CERN – CNAF) 3/3
WLCG
• RTT: 11.1 ms, default tx/rx socket buffer: 2 MB (tuning of
TCP kernel parameters - net.core.rmem_max,
net.core.wmem_max, net.core.rmem_default and
net.core.wmem_default - on the 4 file servers does not
improve the overall performance)
• From 4 to 5 file servers: no improvement, back to 4
• Aggregate local write performance: more than 350 MB/s
• LSF: increase of the number of slots per file server (i.e. the
number of transfer jobs that a given file server can
receive), no improvement
• Next: direct gridftp sessions to the Castor file servers
10
SC4 Accounting
WLCG
• Accounting metrics for SC4 monthly reports:
– CPU usage in KSI2K-days (via DGAS)
• kSI2K days = raw_cpu_hours * GlueHostBenchmarkSI00 /
(1000*24)
can be replaced by a weighted average of the
computing power of Worker Nodes in the farm
• GlueHostBenchmarkSI00
–
–
–
–
Wall-clock time in KSI2K-days (via DGAS)
Disk space used in TB
Disk space allocated in TB
Tape space used in TB
• Validation of raw data gathered, by comparison
via different tools
11
CPU Grid Accounting Results (T2)
Jan-Apr 2006
WLCG
T2 sites: Bari, Catania, Frascati, Legnaro, Napoli, Pisa Roma1-CMS, Roma1-Atlas, Torino
12
Disk Space Accounting Results (T2)
Jan-Apr 2006
WLCG
13
Planning (1/3)
WLCG
• June 01-15:
– Ongoing deployment of gLite 3.0.0 RC5 at T1
• FTS production server now working, deployment of catch-all
channels
– Start of gLite 3.0.0 RC5 deployment in other INFN sites
• INFN Grid 3.0 to be released soon
• SRM still not deployed in every SC4 site
• June 01: start of service phase (see following slides)
• July:
– Testing of concurent data transfer T1  all SC4 INFN sites
– FTS: Testing of VO channel shares
– Upgrade of the existing 1 GE production link at CNAF to 10 GE
14
Planning (2/3)
WLCG
• ATLAS transfer tests (GOAL: stability of DM
infrastructure):
–
–
–
–
–
–
–
From June 19, for 3 weeks
T0  CNAF (59 MB/s)  T2’s (max 20 MB/s, full set of AOD)
raw data to tape at T1
Network reliability still to be evaluated for some T2s
Read performance of Castor2?
Usage of LFC catalogues at T1 sites
DQ2 to submit, manage and monitor the T1 export:
• DQ2 is based on the concept of dataset subscriptions: a site is
subscribed by the T0 management system @ CERN to a dataset that
has been reprocessed
• The DQ2 site service running at the site's VObox will then pick up
subscriptions, submit and manage the corresponding FTS requests
• T2s subscribing to datasets stored on their associated Tier1also using
DQ2
15
Planning (3/3)
WLCG
• CMS: load transfer tests (T1 and T2’s) from May 26:
– T1  T2: bursty transfers driven by analysis
• 20 MB/sec aggregate Phedex (FTS) traffic to/from temporary disk at
each T1
• Currently still based on srmcp  ftscp
• SC3 functionality rerun using gLite infrastructure, trying to rise to
25000 jobs/day aggregated over all sites by end of June
– Bari, Legnaro and Pisa already running jobs
– T2  T1: continuous Monte Carlo transfers
•
•
•
•
aggregating to 1TB/day to CERN
Total of 25M events/month
Test T2  T1 transfers at 20 MB/sec per Tier 2
Last 2 weeks of July: 150 MB/sec simulated raw data to tape at T1s, 25
MB/s to CNAF
16
VO Box
WLCG
• “The term VO Box designates a system provided by the
Site for use by one or more designated Virtual
Organisations (VOs) for interactive access and running
persistent processes.” (from VO-Box Security
Recommendations and Questionnaire, CERN-LCGEDMS-639856)
• installed and maintained by the Site Resource
Administrators, as the VO Box is part of the trusted
network fabric of the Site
• Interactive access to the VO box, and the capability to run
services on the system, MUST be limited to a
• Interactive access and capability to run services limited to
specific, named list of individuals within the VO. These
individuals MUST have user-level access only.
• Well defined network connectivity requirements (ports in
use)
• Vo Box subject to security service challenges.
17
VO Box Task Force: Some Conclusions
(from GDB Meeting slides)
WLCG
• Classification of VO services into two types
– Class 1 : sites not worried
– Class 2 : sites want to see them go away ASAP
• VO boxes with purely class 1 services:
– Needed by all VOs, high-level distributed VO services
– Not really a problem for sites (given solution of a few
policy and security aspects)
• Class 2 services:
– Potential threat to operational stability and security
– Try to avoid new ones as much as possible
– Class 2 services may need to be put somewhere else
(otherwise entire VO box is class 2)
18
VO Box Core Services
WLCG
•
•
•
•
Gsissh or other authenticated way of logging in
Grid client tools (UI)
Proxy renewal service
Shared file system (for ALICE packman, class 2,
need replacement ASAP)
• Candidates for moving into LCG stack:
– Xrootd
• Need some integration / cooperation with SRM products
– Package manager
• Incorporate AliEn packman or develop some alternative
– MonaLisa
• Monitoring
– Many other Longer Term Issues
19
Outstanding issues
•
Accounting (monthly reports):
–
–
–
–
–
–
•
•
CPU usage in KSI2K-days  DGAS
Wall-clock time in KSI2K-days
 DGAS
Disk space used in TB
Disk space allocated in TB
Tape space used in TB
Validation of raw data gathered, by comparison via different tools
Monitoring of data transfer: GridView or other tools?
Routing in LHC Optical Private Network? Backup connection to
FZK becoming urgent
Implementation of a LHC OPN monitoring infrastructure still in its
infancy
SE Reliability when in unattended mode
Several mw components still require in-depth testing
SE kernel configuration:
•
•
•
•
–
•
WLCG
TCP send/receive socket buffer autotuning only for kernels ≥ 2.4.27
and ≥ 2.6.7
What VO Box?
20