Transcript Document 7683305

PDSF Site Report
Shane Canon
PDSF/NERSC/LBNL
US LHC Computing Meting
Brookhaven National Laboratory
Brookhaven, March 28, 2001
Who are we
• Lawrence Berkeley National Laboratory
• National Energy Research Scientific
Computing Center
• Energy Sciences Network
• Parallel Distributed Systems Facility
2 box cars full of hardware
from SSC - only name left.
5/25/2016
Shane Canon
PDSF/NERSC/LBNL
2
PDSF Basics
Our mission is to provide the most effective distributed computer
cluster possible that is suitable for experimental HENP applications
PDSF is a Linux cluster built from commodity hardware and open source
software
- 152 compute nodes, 281 processors, 7.5 TB of storage
- Roughly 7500 SI95 of aggregate CPU power
- Architecture tuned for “embarrassingly parallel” applications
- Cluster uptime for year 2000 was > 98%
- Recently measured cluster utilization for batch jobs was 78% (01/2001) and
89%(02/2001).
- Overall cluster has had zero downtime due to security issues
- PDSF and NERSC have a record of solid security balanced with unobtrusive
practices
5/25/2016
Shane Canon
PDSF/NERSC/LBNL
3
PDSF CLUSTER
Sun
E450
14 Data Vaults (~7.5TB)
Batch Nodes (150)
LSF 4.1
~250 CPU’s total
400MHz, 450MHz,
650MHz, 266MHz
10 GB
Scratch/node
High Bandwidth Nodes
(10 2-processor)
5/25/2016
Sun
Veritas
Vol Mgr
1.2TB
Ultra
60
(AFS)
100Mb network
2 CPU/node
some
GigE
HPSS
333 MHz/PII
6 GB
Scratch/node
CISCO
5500
300 GB
Scratch/node
Shane Canon
PDSF/NERSC/LBNL
4
Sun’s in the Cluster
•
pdsfsu00 - Quad 250MHz Sun
E450 running Solaris 2.6
– Alternative platform for
software development
– Reliable file server
•
pdsfsu05 - Ultra 60 running
Solaris 2.7
– AFS client , access to 96 cells
/afs/cern.ch
/afs/rhic
– knfs server providing AFS for
all the linux nodes and
pdsfsu00
5/25/2016
Shane Canon
PDSF/NERSC/LBNL
5
Mass Storage
PDSF has access to HPSS (High Performance Storage System)
maintained and supported by NERSC (there is a separate allocation
procedure)
Optimized for large files and fast transfers (30 MB/s sustained file
transfer rate - tested on 1 GB files over several hours)
Built on multiple disk farms and tape libraries ~800TB total
Multiple user interface utilities available (to all users that belong to a
project with active allocation).
•HSI - utility built at SDSC (San Diego) and supported by a NERSC
group.
•PFTP - parallel ftp
•FTP
Automatic authentication under HIS and PFTP
5/25/2016
Shane Canon
PDSF/NERSC/LBNL
6
PDSF Modus Operandi
PDSF uses a common resource pool for all projects
- Multiple projects have access to the computing resources, s/w available supports
all experiments
- Actual level of access is determined by the batch scheduler, using fair share rules
- Each project’s investment goes into purchasing hardware and support
infrastructure for the entire cluster
NERSC provides:
-
purchasing expertise and support
staff to run the cluster (2 FTE)
network support
mass storage
housing for computers
heat, electricity
administrative support
5/25/2016
Shane Canon
PDSF/NERSC/LBNL
7
Our Batch System
Load Sharing Facility (4.1) batch system from
Platform Computing
The following queues are available:
long (default queue)
medium
short
normal_su
For < 5CPU days linux jobs
For jobs < 24 hr long on a linux machine
For jobs < 1 hr long on a linux machine
For all sun jobs
short_su
For jobs < 1 hr long on the sun
“Fair Share” allocation based on group’s financial contribution.
“Round Robin” for users with jobs of the same priority
Home grown accounting and reporting (mySQL ans cgi scripts).
LSF Analyzer will be used his coming year (trial run).
5/25/2016
Shane Canon
PDSF/NERSC/LBNL
8
Our Users - HENP
398 users (03/24/2001)
Experiments:
•STAR (RHIC, BNL)
•AMANDA (neutrinos, south pole)
•ATLAS (LHC, CERN)
•E871 (Fermilab)
•SNO (neutrino, mine)
•E895 (AGS, BNL)
PDSF is a secondary computing facility for these experiments
Importance of AFS/network
Experiments gets assistance in moving their software off AFS
Support for HENP Software
Focus on MC simulations and post-DST analysis
5/25/2016
Shane Canon
PDSF/NERSC/LBNL
9
Our past (November)…….
New network...
Oakland
facility
5/25/2016
New hardware
Shane Canon
PDSF/NERSC/LBNL
10
New CPU units
5/25/2016
Shane Canon
PDSF/NERSC/LBNL
11
Our Future in Numbers
(defined by STAR and ATLAS requirements)
FY
$$ CPU
Storage
Other
2000 435.0 6.4k
10
New LSF license
2001 750.0 7.0k
16
Prototype high
bandwidth cluster
2002 500.0 8.0k
11+18
(fast+100Mb)
Scale up high
bandwidth cluster
(SPECInt95) (TB)
25% disk hardware replacement/year starting FY 2001
5/25/2016
Shane Canon
PDSF/NERSC/LBNL
12
Our Staff
• Stephen Chan - PDSF Cluster project
leader
• Carry Whitney - system manager
• Shane Canon - system manager
• Iwona Sakrejda - user support
Come and work with us!
Have to
start training
5/25/2016
them young….
Shane Canon
PDSF/NERSC/LBNL
13
PDSF Current Projects
-Distributed filesystem evaluation
NFS, PVFS, DFS
Evaluate capacity scaling characteristics
-Disk performance analysis
Identify raw performance of disk subsystems
Compare identify bottlenecks in HW, OS, network topology
-FY 2001 HW upgrades
Plans to roughly double raw compute capacity of cluster
Evaluate computing requirements of STAR (and other customers) for memory, disk
Rollout new GigaBit based network topology
-Improvements to monitoring
Use monitoring software with trending to track QOS
More widespread use of SNMP for monitoring
Integrate LSF monitoring into monitoring package
5/25/2016
Shane Canon
PDSF/NERSC/LBNL
14
PDSF Current Projects
(cont’d)
- Creating of beta/testbed environment
Allows us to do more complete testing, and isolate production from tests
Will create environment for customers to verify their codes against proposed updates
-Support for multiple versions of OS
Increase infrastructure support for multiple versions of OS
Necessary if multiple CPU architecture support required (Pentium 4, IA-64)
-Integration with NRSC CSG team
Integrate on call rotation with “big iron” team
Integrate practices from experienced Cray and SP administrators
-Evaluation of LINPack performance
Vehicle for understanding MPI deployment
NERSC senior management curious about scalability of LINPack performance on
commodity clusters
5/25/2016
Shane Canon
PDSF/NERSC/LBNL
15
Our Owners…………...
Experiments’ contributions made at different times brought to
a common denominator by depreciation:
- 3 year flat and then 25%
depreciation/year (hardware life-time)
PRU Contributions in 2000
2%2%
- Moore’s law (50% over 18 months)
4%
NSD
NERSC
ATLAS
E895
E871
23%
69%
On emergency basis experiment can
purchase disk and cpu from the
NERSC share.
Price based on recent purchases with
some overhead (for software and
support).
Money is used to build up the
cluster.
5/25/2016
Shane Canon
PDSF/NERSC/LBNL
16
Our Future
Complete facility, end of FY2002 -(STAR contribution D.Olson)
RHIC Computing Facility
ESnet
Interactive queue
Data vaults
(disk)
High bandwidth
queue
(local data disk)
Low bandwidth queue
5/25/2016
Shane Canon
PDSF/NERSC/LBNL
17
PDSF MO – cont’d
- The use of a common configuration decreases management overhead, lowers
administration complexity, and increases availability of useable computing resources
- Use of commodity Intel hardware makes us vendor neutral, and lowers the cost to
all of our users
- Low cost and easy access to hardware makes it possible for us to update configurations
relatively quickly to support new computing requirements.
- Because the actual physical resources available is always greater than any individual
contributor’s investment, there is usually some excess capacity available for sudden
peaks in usage, and always a buffer to absorb sudden hardware failures
5/25/2016
Shane Canon
PDSF/NERSC/LBNL
18