Transcript Oxford University Particle Physics Site Report Pete Gronbech Systems Manager 24th May 2004 Hepix Edinburgh - Oxford Site Report.

Oxford University
Particle Physics
Site Report
Pete Gronbech
Systems Manager
24th May 2004
Hepix Edinburgh - Oxford Site Report
1
Servers
Particle Physics Strategy
The Server / Desktop Divide
Desktops
General
Purpose Unix
Server
Win 2K
PC
Group
DAQ
Systems
Win 2K
PC
Mail
Server
Win 2K
PC
Web
Server
Windows
File
Server
Win XP
PC
Linux System
Approx 200 Windows 2000 Desktop PC’s with Exceed used to access central
Linux systems
24th May 2004
Hepix Edinburgh - Oxford Site Report
3
Central Physics Computing Services

E-Mail hubs



Windows Terminal Servers



Running two new servers using Exchange 2003 running on Windows server 2003. Much better Web interface, support for
mobile devices (oma) and for tunnelling through firewalls.
NFS gateway


New web server (Windows 2003) in service.
New web applications for lecture lists, Computer inventory, admissions and finals
Exchange Servers


Use is still increasing 250 users in last three months out of 750 staff/students. Now Win2k and 2003.
Introduced an 8 CPU server (TermservMP) . Much more powerful system but still awaiting updated versions of some
applications which will run properly on OS.
Web / Database



In last year 7.3M messages were relayed , 73% rejected and 5% were viruses.
Anti-virus and anti-spam measures increasingly important in email hubs. Some spam inevitably leaks through and clients
need to deal with this in a more intelligent way.
Runs on windows file server, mounts linux disks and makes them available to windows users via windows DFS. Windows
account name mapped onto linux username. Will replace Samba.
Desktops

Windows XP pro is default OS for new desktops and laptops.
24th May 2004
Hepix Edinburgh - Oxford Site Report
4
Linux / Unix Systems

Central Unix systems are Linux based







Red Hat Linux 7.3 is the standard
Treat Linux as just another Unix and hence
a server OS to be managed centrally.
Wish to avoid badly managed desktop PC’s
running Linux.
Linux based file server (April 2002)
General purpose Linux server (August 2002)
Batch farm installed (Autumn 02, Autumn 03)
Solaris 8 systems used for Electronics CAD
24th May 2004
Hepix Edinburgh - Oxford Site Report
5
CDF
Fermi
7.3.1
General Purpose Systems
7.3.1
7.3.1
7.3.1
7.3.1
7.3.1
7.3.1
Fermi
7.3.1
RH
7.3
RH
7.3
RH
7.3
RH
7.3
1Gb/s
pplx1 morpheus matrix
pplx2
pplxfs1 pplxgen
pplx3
minos DAQ
RH
7.3
RH
7.3
RH
7.3
RH
7.3
RH
7.3
RH
7.3
Autumn
2002
4*Dual 2.4GHz systems
ppminos1 ppminos2
RH
7.3
cresst DAQ
RH
7.3
RH
7.3
RH
7.3
RH
7.3
RH
7.3
7.3.1
7.3.1
7.3.1
7.3.1
7.3.1
LCG2
7.3.1
7.3.1
7.3.1
7.3.1
7.3.1
LCG2
Autumn
2003
Oxford Tier 2 - LCG2
Summer 2004
4*Dual 2.4GHz systems
PBS Batch Farm
Grid Development
ppcresst1 ppcresst2
Atlas DAQ
RH
7.3
RH
7.3
24th May 2004
ppatlas1 atlassbc
RH
7.3
tblcfg
RH
7.3
RH
7.3
se
ce
RH
7.3
RH
7.3
grid tbwn01
Hepix Edinburgh - Oxford Site Report
RH
7.3
pptb01
RH
7.3
pptb02
6
The Linux File Server: pplxfs1
8*146GB SCSI disks
Dual 1GHz PIII, 1GB RAM
24th May 2004
Hepix Edinburgh - Oxford Site Report
7
New Eonstor IDE RAID array added
in April 04. 16* SATA 250GB disks
gives approx 4TB for around £6k.
This unit was supplied by Sweet
Valley, but is also available from
transtec. The controller is from
Infortrend.
This is our second foray into IDE
storage. So far so good.
24th May 2004
Hepix Edinburgh - Oxford Site Report
8
General Purpose Linux Server : pplxgen
pplxgen is a Dual 2.2GHz
Pentium 4 Xeon based
system with 2GB ram. It
is running Red Hat 7.3
It was brought on line at
the end of August 2002.
Provides interactive login
facilities for code
development and test
jobs. Long jobs should be
sent to the batch queues.
Up to 50 users.
Memory to be upgraded
to 4GB next week.
24th May 2004
Hepix Edinburgh - Oxford Site Report
9
PP batch farm running Red Hat
7.3 with Open PBS can be seen
below pplxgen
This service became fully
operational in Feb 2003.
Additional 4 worker nodes were
installed in October 2003. These
are 1U servers and are mounted
at the top of the rack.
Miscellaneous other nodes bring
a total of 21 cpu’s available to
PBS.
24th May 2004
Hepix Edinburgh - Oxford Site Report
10
http://www-pnp.physics.ox.ac.uk/ganglia-webfrontend-2.5.4/
24th May 2004
Hepix Edinburgh - Oxford Site Report
11
CDF Linux Systems
Morpheus
is an IBM x370
8 way SMP 700MHz Xeon
with 8GB RAM and
1TB Fibre Channel disks
Installed August 2001
Purchased as part of a JIF grant
for the CDF group
Runs Fermi Red Hat 7.3.1
Uses CDF software developed at
Fermilab and Oxford to process data
from the CDF experiment.
24th May 2004
Hepix Edinburgh - Oxford Site Report
12
Second round of CDF JIF tender:
Dell Cluster - MATRIX
10 Dual 2.4GHz P4 Xeon servers
running Fermi Linux 7.3.1 and SCALI
cluster software. Installed December
2002
Approx 7.5 TB for SCSI RAID 5 disks
are attached to the master node.
Each shelf holds 14 * 146GB disks.
These are shared via NFS with the
worker nodes.
OpenPBS batch queuing software is
used.
24th May 2004
Hepix Edinburgh - Oxford Site Report
13
Plenty of space in the second rack for
expansion of the cluster.
Additional Disk Shelf with 14*146GB
plus an extra node was installed in
Autumn 2003.
24th May 2004
Hepix Edinburgh - Oxford Site Report
14
Oxford Tier 2 centre for LHC
Two racks each containing 20 Dell dual
2.8GHz Xeon’s with SCSI system disks.
Total of 80 CPU’s.
1.6TB SCSI disk array in each rack.
Systems will be loaded with LCG2 software.
SCSI disks and Broadcom Gigabit Ethernet
causes some problems with installation.
Slow progress being made.
24th May 2004
Hepix Edinburgh - Oxford Site Report
15
Problems of Space, Power and
Cooling.
Second rack currently temporarily
located in theoretical physics
computer room.
A proposal for a new purpose built
computer room on Level 1
(underground) in progress.
False floor, large Air conditioning
units and power for approx 20-30
racks to be provided.
1200W/sq m max air cooling, a rack
full of 1U servers can create 10KW
of heat.
Water cooling??
24th May 2004
Hepix Edinburgh - Oxford Site Report
16
Tape Backup is provided by
a Qualstar TLS4480
tape robot with 80 slots
and Dual Sony AIT3 drives.
Each tape can hold 100GB
of data.
Installed Jan 2002.
Netvault 7.1 Software from BakBone
is used, running on morpheus, for
backup of both cdf and particle
physics systems.
Main userdisks backed up every
weekday night data disks not generally
backed up BUT weekly backups to
OUCS HFS service provide
some security.
24th May 2004
Hepix Edinburgh - Oxford Site Report
18
Network Access
Super Janet 4
2.4Gb/s with Super Janet 4
OUCS
Firewall
100Mb/s
Physics
Firewall
100Mb/s
1Gb/s
1Gb/s
100Mb/s
depts
Backbone
Edge
Router
Physics
Backbone
Switch
1Gb/s
Campus
Backbone
Router
Backbone
Edge
Router
100Mb/s
depts
100Mb/s
depts
100Mb/s
depts
Physics Backbone Upgrade to
Gigabit Autumn 2002
1Gb/s Linux
Server
Server
switch
1Gb/s
Win 2k
Server
1Gb/s
Physics
Firewall
100Mb/s
1Gb/s
100Mb/s
1Gb/s
Clarendon
Lab
100Mb/s
Particle
Physics
Physics
Backbone
Switch
desktop
1Gb/s
1Gb/s
100Mb/s
desktop
1Gb/s
Astro
Atmos
100Mb/s
Theory
Goals for 2004 (Computing)

Continue to improve Network security

Need better tools for OS patch management
 Need users to help with their private laptops
– Use automatic updates (e.g. Windows Update)
– Update Antivirus software regularly



Reduce number of OS’s






Segment the network by levels of trust
All the above without adding an enormous management overhead !
Remove last NT4 machines and exchange 5.5
Digital Unix and VMS very nearly gone.
Standardising on RH 7.3 .
Supporting laptops by having a standard clone and
recommend IBM laptops.
What version of Linux to run ? Currently all 7.3 but what
next?
Looking into Single Sign On for Particle Physics systems
24th May 2004
Hepix Edinburgh - Oxford Site Report
23