Transcript Particle Physics Unix Overview

Oxford University
Particle Physics
Unix Overview
Pete Gronbech
Senior Systems Manager and
GridPP Project Manager
13th October 2011
Graduate Lectures
1
Strategy
 Local Cluster Overview
 Connecting to it
 Grid Cluster
 Computer Rooms
 How to get help

13th October 2011
Graduate Lectures
2
Particle Physics Strategy
The Server / Desktop Divide
Servers
Virtual Machine Host
Desktops
General
Purpose Unix
Server
Win XP
PC
Linux File
Servers
Win XP
PC
Linux
Worker
nodes
Win 7
PC
Group
DAQ
Systems
Win 7
PC
Web
Server
NIS
Server
torque
Server
Linux Desktop
Approx 200 Windows XP Desktop PC’s with Exceed, putty or ssh used to access
central Linux systems
13th October 2011
Graduate Lectures
3
Particle Physics Linux

Unix Team (Room 661):




Pete Gronbech - Senior Systems Manager and GridPP Project Manager
Ewan MacMahon – Grid and Local Systems Administrator
Kashif Mohammad – Grid Support
A.N. Other – Local Server and User Support

Aim to provide general purpose Linux based system for code development and
testing and other Linux based applications.

Interactive login servers and batch queues are provided

Systems run Scientific Linux which is a free Red Hat Enterprise based
distribution

Systems are currently mainly running SL5, this is the same version as used on
the Grid and at CERN. Students should use pplxint5 and 6. Legacy applications
only available on SL4 can be run on SL5, but you should actively try to port these
across to SL5.

Worker nodes form a PBS (aka torque) cluster accessed via batch queues.
13th October 2011
Graduate Lectures
4
Current Clusters

Particle Physics Local Batch cluster

Oxfords Tier 2 Grid cluster
13th October 2011
Graduate Lectures
5
PP Linux Batch Farm
Scientific Linux 5
Users log in to the interactive nodes
Pplxint5 & 6, the home directories and all
the data disks (/home area or /data/group )
are shared across the cluster and visible on
the interactive machines and all the batch
system worker nodes.
Various other infrastructure nodes
pplxwn19
pplxwn18
8 * Intel 5345 cores
8 * Intel 5345 cores
Currently acting as NFS – Lustre
gateways for the SL4 nodes
13th October 2011
pplxwn32
pplxwn31
16 * AMD Opteron 6128 cores
pplxwn30
16 * AMD Opteron 6128 cores
pplxwn29
16 * AMD Opteron 6128 cores
pplxwn28
pplxwn27
16 * AMD Opteron 6128 cores
16 * AMD Opteron 6128 cores
pplxwn26
16 * AMD Opteron 6128 cores
pplxwn25
16 * AMD Opteron 6128 cores
pplxwnnn
8 * Intel 5420 cores
pplxwnnn
pplxwnnn
8 * Intel 5420 cores
8 * Intel 5420 cores
pplxwnnn
pplxwnnn
8 * Intel 5420 cores
pplxwn5
8 * Intel 5420 cores
pplxwn4
8 * Intel 5420 cores
pplxint5
pplxint6
Graduate Lectures
16 * AMD Opteron 6128 cores
8 * Intel 5420 cores
Interactive
login nodes
6
PP Linux Batch Farm
Data Storage
NFS
Servers
9TB
CDF Data
pplxfs3
19TB
pplxfs4
Data
Areas
30TB
pplxfs9
Data
Areas
19TB
pplxfs6
Home
areas
13th October 2011
NFS is used to export data to the
smaller experimental groups, where
the partition size is less than the
total size of a server.
The data areas are too big to be backed up. The
servers have dual redundant PSUs, RAID 6 and
are running on uninterruptable powers supplies.
This safeguards against hardware failures, but
does not help if you delete files.
The home areas are backed up to by two different
systems nightly. The OUCS HFS service and a
local back up system. If you delete a file tell us a
soon as you can when you deleted it and it’s full
name.
The home areas are quota’d but if you require
more space ask us. Store your thesis here.
Graduate Lectures
7
Particle Physics Computing
The Lustre file system is used to group multiple file servers together to provide extremely large
continuous file spaces. This is used for the Atlas and LHCb groups.
Lustre MDS
Lustre OSS01
Lustre OSS02
Lustre OSS03
18TB
18TB
44TB
Lustre NFS Gateway
df -h /data/atlas
Filesystem
pplxlustrenfs.physics.ox.ac.uk:/data/atlas
gronbech@pplxint2:~> df -h /data/lhcb
Filesystem
pplxlustrenfs2.physics.ox.ac.uk:/data/lhcb
13th October 2011
Size Used Avail Use% Mounted on
183T 67T 108T 39% /data/atlas
Size Used Avail Use% Mounted on
18T 8.5T 8.6T 50% /data/lhcb
SL4 Nodes
SL5 Node
SL4 Nodes
SL5 Node
SL4 Nodes
Graduate Lectures SL5 Node
SL4 Nodes
8
13th October 2011
Graduate Lectures
9
Strong Passwords etc

Use a strong password not open to
dictionary attack!
– No good
 Uaspnotda!09 – Much better
 fred123

Better to use ssh with a passphrased
key stored on your desktop.
13th October 2011
Graduate Lectures
10
Connecting with PuTTY
Demo
1. Plain ssh terminal connection
2. With key and Pageant
3. ssh with X windows tunnelled to
passive exceed
4. ssh, X windows tunnel, passive
exceed, KDE Session
http://www.physics.ox.ac.uk/it/unix/particle/XTunnel%20via%20ssh.htm
13th October 2011
Graduate Lectures
11
13th October 2011
Graduate Lectures
12
Puttygen to create an ssh key on Windows
Paste this into
~/.ssh/authorized_keys on
pplxint
If you are likely to then
hop to other nodes add :
ForwardAgent yes
to a file called config in
the .ssh dir on pplxint
Save the public and
private parts of the key to
a subdirectory of your h:
drive
13th October 2011
Graduate Lectures
13
Pageant

Run Pageant once after login to load
your (windows ssh key)
13th October 2011
Graduate Lectures
14
SouthGrid Member Institutions

Oxford
RAL PPD
Cambridge
Birmingham
Bristol
Sussex

JET at Culham





13th October 2011
Graduate Lectures
15
2011
Current capacity

Compute Servers

Twin and twin squared nodes
– 880 CPU cores

Storage


Total of ~600TB
The servers have between 12 and 36 disks, the
more recent ones are 2TB capacity each. These
have hardware RAID controllers. We use RAID 6
to provide resilience against double disk failure.
Dual Power supplies and UPS are standard.
13th October 2011
Graduate Lectures
16
Get a Grid Certificate
Must remember to use the same web browser to request and
retrieve the Grid Certificate.
Once you have it in your browser you can export it to the Linux
Cluster to run grid jobs.
Details of these steps and how to request membership of the
SouthGrid VO (if you do not belong to an existing group such as
ATLAS, LHCb) are here:
http://www.gridpp.ac.uk/southgrid/VO/instructions.html
13th October 2011
Graduate Lectures
17
Two New Computer Rooms
provide excellent
infrastructure for the future
The New Computer room built at Begbroke Science Park jointly for the Oxford Super Computer
and the Physics department, provides space for 55 (11KW) computer racks. 22 of which will be
for Physics. Up to a third of these can be used for the Tier 2 centre. This £1.5M project was
funded by SRIF and a contribution of ~£200K from Oxford Physics.
The room was ready in December 2007. Oxford Tier 2 Grid cluster was moved there during
spring 2008. All new Physics High Performance Clusters will be installed here.
13th October 2011
Graduate Lectures
18
Local Oxford DWB Physics
Infrastructure Computer Room
Completely separate from the Begbroke Science park a computer
room with 100KW cooling and >200KW power has been built. ~£150K
Oxford Physics money.
Local Physics department Infrastructure computer room.
Completed September 2007.
This allowed local computer rooms to be refurbished as offices
again and racks that were in unsuitable locations to be re housed.
13th October 2011
Graduate Lectures
19
The end for now…
Ewan will give more details of use of the
clusters next week
 Help Pages



http://www.physics.ox.ac.uk/it/unix/default.htm

http://www.physics.ox.ac.uk/pp/computing/
Email
 [email protected]
Questions….
 Network Topology

13th October 2011
Graduate Lectures
20
Network




Gigabit connection to campus operational since July 2005.
Second gigabit connection installed Sept 2007.
Dual 10 gigabit links installed August 2009
Gigabit firewall installed for Physics. Purchased commercial unit
to minimise manpower required for development and
maintenance. Juniper ISG 1000 running netscreen.

Firewall also supports NAT and VPN services which is allowing
us to consolidate and simplify the network services.

Moving to the firewall NAT has solved a number of problems we
were having previously, including unreliability of
videoconferencing connections.

Physics-wide wireless network. Installed in DWB public rooms,
Martin Wood,AOPP and Theory. New firewall provides routing
and security for this network.
13th October 2011
Graduate Lectures
21
Network Access
Super Janet 4
2* 10Gb/s with Super Janet 5
OUCS
Firewall
Physics Firewall 1Gb/s
1Gb/s
10Gb/s
10Gb/s
1Gb/s
depts
10Gb/s
Backbone
Edge
Router
Physics
Backbone
Router
Campus
Backbone
Router
Backbone
Edge
Router
100Mb/s
depts
100Mb/s
depts
100Mb/s
depts
13th October 2011
Graduate Lectures
22
10Gb/s Linux
Server
Server
switch
10Gb/s
Linux
1Gb/s
Server
1Gb/s Linux
Server
Server
switch
1Gb/s
Win 2k
Server
1Gb/s
Physics Backbone
Physics
Firewall
1Gb/s
1Gb/s
100Mb/s
1Gb/s
Clarendon
Lab
100Mb/s
Particle
Physics
Physics
Backbone
Router
desktop
1Gb/s
1Gb/s
100Mb/s
100Mb/s
desktop
1Gb/s
Astro
Atmos
13th October 2011
Theory
Graduate Lectures
23