Transcript Summary of the HEPiX Spring 2014 Meeting Arne Wiebalck Ben Jones Vincent Brillault CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it CERN ITTF June 06, 2014

Summary of the
HEPiX Spring 2014 Meeting
Arne Wiebalck
Ben Jones
Vincent Brillault
CERN IT Department
CH-1211 Genève 23
Switzerland
www.cern.ch/it
CERN ITTF
June 06, 2014
HEPiX – www.hepix.org
• Global organization of service managers and support
staff providing computing facilities for HEP community
• Participating sites include BNL, CERN, DESY,
FNAL, IN2P3, INFN, NIKHEF, RAL, TRIUMF …
• Meetings are held twice per year
– Spring: Europe, Autumn: U.S./Asia
• Reports on status and recent work, work in progress &
future plans
– Usually no showing-off, honest exchange of experiences
Wiebalck, Jones, Brillault: Summary of the HEPiX Spring 2014 Meeting - 2
Outline
• HEPiX News & 2014 Spring Meeting
• Site reports
• Storage & File Systems
• End User Services
Arne
• Basic IT Services
• Computing & Batch Systems
Ben
• IT Facilities
• Clouds & Virtualisation
• Networking & Security
Vincent
Wiebalck, Jones, Brillault: Summary of the HEPiX Spring 2014 Meeting - 3
HEPiX News
• New (American) Co-Chair
– Sandy Philpott reaches end of her mandate
– Call for nominations to be sent out in June
– Election will take place at autumn HEPiX
• New HEPiX website
– First version planned to go live end of June
– Migration of content in the coming months
Wiebalck, Jones, Brillault: Summary of the HEPiX Spring 2014 Meeting - 4
Next HEPiX Meetings
• Autumn 2014
– University of Nebraska, Lincoln (NE), U.S.
– Oct 13 – Oct 17, 2014
• Spring 2015
– Oxford, U.K.
– Mar 23 – Mar 27, 2015
• Autumn 2015
– DESY Zeuthen (?)
Wiebalck, Jones, Brillault: Summary of the HEPiX Spring 2014 Meeting - 5
HEPiX Spring 2014
• May 19 – May 23 at LAPP, Annecy-Le-Vieux
– Very well organized, pretty rich program
– Network access: eduroam (as in Bologna, Ann Arbor)
• 105 registered participants
– Europe: 77 (France 26), U.S./Canada: 10, Asia: 6 (CERN: 15)
– Many first timers
– 12 participants from 6 companies
• 71 presentations
– 26 hours of presentations
– Many offline discussions
• Sponsors: WD, DDN, and Univa
Wiebalck, Jones, Brillault: Summary of the HEPiX Spring 2014 Meeting - 6
Updates from the WGs (1)
• IPv6
– No formal report, but some IPv6 related talks in networking track
– Sites are encouraged to attend the pre-WLCG in June
• Configuration management
– Report by Ben
– Explained objectives: share information & experiences, not replace
Puppet community
– Yaim is a concern (Puppet wrappers)
• Benchmarking
– New SPEC CPU suite expected by the end of the year
– Identifying experiment code to compare benchmarks candidates
with will start now
Wiebalck, Jones, Brillault: Summary of the HEPiX Spring 2014 Meeting - 7
Updates from the WGs (2)
• Energy efficiency
– No enthusiastic answers from community
– Stopped for now
• Batch systems
– No formal report
– Data of the survey and findings from pre-GDB will be on the website
• Bit preservation
– Report from German
– Focus long-term costing evaluations
Wiebalck, Jones, Brillault: Summary of the HEPiX Spring 2014 Meeting - 8
Site reports (1)
• Configuration Management
– Many sites have moved to Puppet or are moving
– Few sites left that use Quattor (RAL, INFN) or other systems
• Batch system reviews ongoing
– Discussions about “suitability”
– HTCondor takes the lead (non-proprietary solutions)
• Cloud storage
– Ceph still a hot topic, evaluated at several sites:
RAL (see later), BNL, ASGC, CNAF, AGLT2, …
– “Dropbox”-like services enter production
KIT (bwSync based on PowerFolder for 450’000 users),
DESY (see later)
Wiebalck, Jones, Brillault: Summary of the HEPiX Spring 2014 Meeting - 9
Site reports (2)
• Hardware
– 4GB RAM per core
– 10GbE
– Share procurement experiences?
• Windows
– XP: still run at many sites, but usually confined in VLANs or
blocked by firewall, i.e. without internet access or mail
– Active migration to Win 7 and 8
(Win 8 already default at various sites)
• Monitoring
– Logstash, ElasticSearch, and Kibana become more wide-spread
Wiebalck, Jones, Brillault: Summary of the HEPiX Spring 2014 Meeting - 10
Site reports (3)
• “Cooling problem” at INFN
– Problem with cooling system
– Took out their CC for 1 week
(Tier1 back after 36 hours)
– Happened Sunday at 1am …
Wiebalck, Jones, Brillault: Summary of the HEPiX Spring 2014 Meeting - 11
Site reports (4)
• FNAL
– PDU plug incident
– Alarm was triggered, but firemen
did not find the problem due to the
rapid air exchange
– Thermographic imaging for an
early in-situ
Wiebalck, Jones, Brillault: Summary of the HEPiX Spring 2014 Meeting - 12
Storage
• Ceph @ RAL
–
–
–
–
Cloud backend (OpenNebula): ~1PB, currently being installed
CASTOR replacement: ~1.8PB from batch via CephFS
Special ATLAS Panda queue for ARC CE with CephFS
Works well so far
• Cloud Storage @ DESY
– Motivation: data locality
– Requested features: big, fast, reliable, ACLs, support on
all clients, web access, …
– DESY’s approach: ownCloud on dCache
– No quotas, “unlimited” space (~5’000 users)
– Billing?
Wiebalck, Jones, Brillault: Summary of the HEPiX Spring 2014 Meeting - 13
End User Services (1)
• Session on “Future OS for HEP”
– Alan Silverman on the creation of Scientific Linux
–
–
–
–
–
–
10 years ago: Several Unix flavors and “Linux?”
In 2003 RedHat went commercial, the binary distribution needed a license
Various labs negotiated good deals with RedHat
FNAL and CERN tried to do so too, but it didn’t work out
In Spring 2004 FNAL and CERN started to rebuild from source: Scientific Linux
Since then, SL has been a major community driving force in HEP
– Karanbir Singh on CentOS
–
–
–
–
Started for similar reasons and around the same time as SL
Stable platform to solve every day's problems (unlike Fedora)
Compatible with RHEL
In January 2014 CentOS and RedHat “joined forces”:
RH became main sponsor, pays main developers (open source unit),
owns CentOS trademarks
– Special Interest Groups
Wiebalck, Jones, Brillault: Summary of the HEPiX Spring 2014 Meeting - 14
End User Services (2)
– FNAL and CERN presented their points of view on what should be “SL?7”
–
–
–
–
–
–
–
Should we continue to build a separate distribution?
Should we adopt CentOS and become part of a bigger community?
Is CentOS already ready for this?
Does it need to be ready?
Do we lose the “community factor” of SL?
What if RedHat changes plans with CentOS?
Aren’t we already in a situation where we don’t have a single OS?
– Summary and conclusion
–
–
–
–
–
–
–
CentOS still too much in flux to take firm decision now
HEP community should try to influence CentOS discussion
Request for a “Scientific SIG”
FNAL and CERN will observe the situation and stay in close contact
Both teams will take their own decision on how to build “SL7” (not final)
The situation will be re-assessed at next HEPiX
Both teams acknowledge the preference for a common solution
Wiebalck, Jones, Brillault: Summary of the HEPiX Spring 2014 Meeting - 15
Basic IT Services
• Ten talks
–
–
–
–
Quattor & Puppet
Data transformations with Lavoisier
Cluster consolidation at NERSC
Agile Infrastructure updates
• Quattor: positive and not so positive updates
• RAL described positive developments in Quattor community
–
–
–
–
–
Adopting Aquilion to replace SCDB
Development moved to GitHub improving collaboration
Release manager & regular updates
YUM!
ncm-puppet
Wiebalck, Jones, Brillault: Summary of the HEPiX Spring 2014 Meeting - 16
Basic IT Services (2)
• IRFU – migration from Quattor to Puppet
– Highlighted lack of documentation / debugging tools available in
Quattor
– Chose Puppet due to large community, developers etc, but also due
to CERN
– Migration took 2 years, but now fully puppet, EMI-3, SLC6
• Secrets
– DESY presented results of initial POC, which is slightly complicated
due to key management
• Lavoisier data transformation
– tool to describe data transformations between different formats
• Cluster Consolidation at NERSC
– Improving supercomputer cluster by managing heterogeneous nodes
using OS, xCat & cfengine
Wiebalck, Jones, Brillault: Summary of the HEPiX Spring 2014 Meeting - 17
Basic IT services (3)
• AI updates
– All three tracks reported progress
– Addition of customer-orientated talk by service manger (Jérôme for
Batch)
• Takeaways:
– More sites migrating to puppet
– Those sites still using Quattor enjoying some improvements with
more nimble development & adoption of Aquilon
– YUM for all
– NERSC cluster consolidation shows that choice of configuration
management less important than just doing it properly
Wiebalck, Jones, Brillault: Summary of the HEPiX Spring 2014 Meeting - 18
Computing & batch systems
• Ten talks covering:
–
–
–
–
batch systems, with specific talks on Condor and Univa GE
multicore scheduling, linux control groups
CPU / system evaluations and comparisons
benchmarking
• Condor
– Three talks from very different stages of Condor adoption
• CERN reported on Condor scalability testing with a view to
adoption
• RAL described experiences after running Condor for 1 year
– Were using Torque/MAUI but installation had become brittle
– Rejected LSF/Univa GE in favour of OSS solution
– Scalability issues with SLURM > 6K jobslots, worse with plugins
Wiebalck, Jones, Brillault: Summary of the HEPiX Spring 2014 Meeting - 19
Computing & batch (2)
• RAL experience with Condor very smooth
–
–
–
–
no issues scaling to 14K cores
very configurable
using with ARC CE, will deco CREAM
lots of other UK sites moving to Condor based on experience
• Nebraska presented much longer experience with Condor
– “Swiss army knife” of high throughput computing, batch just one
facet
– lots of useful best practices, for monitoring, security, accounting,
extensibility
• Univa Grid Engine used by IN2P3 as migration from Oracle
– originally tested Sun GE before Oracle acquisition
– Oracle support was very disappointing
– migration to Univa felt like a version update rather than new product
Wiebalck, Jones, Brillault: Summary of the HEPiX Spring 2014 Meeting - 20
Computing & batch (3)
• Univa themselves presented on linux cgroups
– interesting alternative to virtualization for compartmentalization
– safer job suspension & job reaping, CPU isolation, memory limits
• Multi-core a hot topic
– batch system review from GDB highlighted increasing importance to
WLCG
– desirable to use same resources for multi-core & single-core
– CC-IN2P3 reported that mixing not currently possible with Univa GE
– Experiment perspective of multicore provided by LHCb, some
difficulties in understanding where the VO or the site is responsible
for resolving issues
• Benchmarking – HS06 coming to end of useful life
– WG looking at change, HS16 working title of new spec
– Some discussion as to whether should be free
Wiebalck, Jones, Brillault: Summary of the HEPiX Spring 2014 Meeting - 21
Computing & batch (4)
• Ivy Bridge v Opteron
–
–
–
–
Ivy Bridge same per-core in HS06 as Opteron
per system, Intel boxes have clear advantage
Opteron scale better for multi-job throughput, possibly due to SMT
Opteron purchase price lower per HS06
• Avaton (Atom system-on-chip)
– micro server & cloud storage market
– very power efficient
– CNAF tested half pizza box machine & HP “Moonshot” (4.3U 42
server carts)
– Factor three less per core on HS06 compared with modern Xeon
– Factor three advantage in HS06 per watt
– Would mean many more nodes for equivalent workloads
Wiebalck, Jones, Brillault: Summary of the HEPiX Spring 2014 Meeting - 22
Networking & security (1)
• IPv6:
– FZU (Prague):
• Dual stack DPM (head node, disk nodes, WN): WAN IPv6 not really used
• Some migration difficulties: helpful insights for other deployments
• Partial monitoring using Nagios. Example: default route (dynamic)
– QMUL (UK): Dual stack ok, but:
• Small MTU issues: due to ICMPv6 blocked
• IPv6 using software routing: slow!
• One-way IPv6 wrong route issue fixed
– UK:
• Deployment not uniform
• Lot of sites involved in various tests
– Pre-GDB on IPv6 next week
Wiebalck, Jones, Brillault: Summary of the HEPiX Spring 2014 Meeting - 23
Networking & security (2)
• PerfSONAR:
– Deployment task force ended in April
• Only 8 sites missing, 64 using outdated version
• Still too much data missing (configuration/firewall issues)
– New task force
• Fix deployment issues
• Improve metrics and their usages
• Measuring WLCG data streams at batch job level
– Monitoring network traffic in userland by process/job
– Overhead: 5% CPU on one core
– First results:
Wiebalck, Jones, Brillault: Summary of the HEPiX Spring 2014 Meeting - 24
Networking & security (3)
• New Windows security at CEA and IRFU:
–
–
–
–
–
Computer not compliant: no internet, dedicated VLAN
Standard PC: Chrome & IE, banned software removed
Management: no internet, no mail except webmail, no remote access
Management+: no mail at all, special admin accounts
Management++ (AD root): special room
• Emergency suspension list:
– Efficient suspension/unsuspension mechanism for the Grid
– Hierarchical infrastructure: CERN – NGIs – Sites
– Monitoring: only NGIs by EGI
• Security Update:
– Crypto-currency mining
– SSL/x509 nightmares
– Windigo
Wiebalck, Jones, Brillault: Summary of the HEPiX Spring 2014 Meeting - 25
IT Facilities (1)
• Open compute @ CERN (ITTF 2014/03/14)
Interesting result from tests but procurement limitations
• New Data-center in Orsay: 1.5MW in 220m²
–
–
–
–
Long delays: budget issues (project reported)
High energy efficiency required: long term investment
French electrical regulation (16A, 30mA differential): custom PDU!
Passive rack issues (hot): mixing active/passive for better air flow
• Wigner:
– ¾ of the site for Tier-0 capacity
– First room operated while site under construction!
– Minor issues so far: specifications not detailed enough
Wiebalck, Jones, Brillault: Summary of the HEPiX Spring 2014 Meeting - 26
IT Facilities (2)
• INFN cooling issue
– Incident:
• One chiller burning
• Shared power supply for control logic: security kills 5/6 chiller
• Overheating CC -> Emergency shutdown: power cut !
– Damages:
• Bios configuration (dead batteries), IPMI configuration
• Broken: 30% IPMI, 1% PCI cards (mostly network cards)
– Designing new emergency shutdown procedure: testing ?
• Business continuity at DESY:
–
–
–
–
Ongoing effort for ISO 27001: should be conform, but needs writing
CC climat & power managed by 2 other departments
Focus on optimizing the incident reaction over finding single point of failures
Incident: one power line cut, other line overheated, saved by batteries
Wiebalck, Jones, Brillault: Summary of the HEPiX Spring 2014 Meeting - 27
Virtualization (1)
•
Big Data transfers
–
–
•
FermiCloud On-demand Services
–
–
–
–
–
•
LHC only one of the big data source, not the biggest
TCP/IP tests planed with various configurations & infrastructures
Address concurrent peak utilization
Code deployment: CVMFS
Using GlideinWMS, pushing to different clous
Various issues, including limit to 100 VM due to Squid@FNAL
EWS price: $125 for 1088 jobs (1k CPU-hours), network included
Experiences with Vacuum model
–
–
–
Idea: Host creates VM by itself, Pilot job will handle the rest
Flavor of new VM: targeted shared VS exchanges between hosts
VM VS Batch Efficiency: 99.37% (Sigma: 0.57%)
Wiebalck, Jones, Brillault: Summary of the HEPiX Spring 2014 Meeting - 28
Virtualization (2)
•
Virtualization @ RAL Tier-1
–
–
–
–
•
Most service virtualized using Hyper-V
Live migration issues not clearly understood
Cloud prototype (for federated clouds)
Using HTCondor power management for dynamic WN
Helix Nebula update:
–
Successful test deployments over multiple suppliers
•
•
–
Using ATLAS PanDA framework + SlipStream
High CPU jobs with low I/O
Issues:
•
•
Supplier heterogeneity (e.g. VM deployment: 5-25 mins)
Cost high or Undefined
Wiebalck, Jones, Brillault: Summary of the HEPiX Spring 2014 Meeting - 29