An Overview of the Reliability and the Fault Trends of the SRS

Cheryl Hodgkinson Daresbury Laboratory Synchrotron Radiation Department WAO, Trieste, 24th - 28th September 2007

Contents

• Introduction • SRS Operational Statistics • Analysis of the Operational Data – Efficiency – Fault Statistics – Ageing and Obsolete Equipment • Maintaining Reliability – Capital Investment and Redundancy – Risk Analysis – Post Incident Investigation and Modifications – Preventative Maintenance and Scheduling • Conclusions and Acknowledgments WAO, Trieste, 24th - 28th September 2007

Introduction

• The Synchrotron Radiation Source

– 2GeV, 2 nd Generation Light Source – Constructed between 1975 and 1980 – Began scheduled user operations in 1981 – Some SRS systems, mainly injector components were salvaged from a previous accelerator – Operational user facility for 26 years and scheduled for closure December 2008 WAO, Trieste, 24th - 28th September 2007

SRS Reliability Data

2001 -2007

•Dominated by water – vacuum incidents

1984

•04/05 (85%) RF Cavity Absorber •06/07 (78%) TVM 9 •Efficiency in 06/07 was also

– 1992

Window & Quadrupole Vessel

1993 - 2001

Stable period of operation Gradual increase in efficiency to significantly effected by Linac RF and HT systems and Gun Cathode figures in excess issues.

of 90% due to new systems excess of 90%.

WAO, Trieste, 24th - 28th September 2007

SRS Fault Statistics

Vacuum Incidents

WAO, Trieste, 24th - 28th September 2007

Vacuum Incidents

• Usually an isolated single failure • Difficult to predict and prevent

– failed RGA head – Quadrupole absorber – failed flowmeter

• Common Theme?

– SCW Absorber & TVM 9 – Both very different failure mechanisms – Both historical – not needed in the current SRS configuration WAO, Trieste, 24th - 28th September 2007

SRS Fault Statistics

WAO, Trieste, 24th - 28th September 2007

Ageing and Obsolete Equipment

• Early Equipment – Built in-house – Repairs in-situ (time) – Can be repaired • Later Equipment – Plug-in – Replaced with a spare – Return to manufacturer for repair – Support issues WAO, Trieste, 24th - 28th September 2007

Ageing and Obsolete Equipment

• Concern regarding

RESULT

condition of in vacuum water circuits – 17 year old absorber removed from storage ring – Area’s milled away to examine any wall thinning WAO, Trieste, 24th - 28th September 2007

Ageing and Obsolete Equipment

• External water circuits – Elbows – Near feedthroughs – Potential for vacuum incident during repair – Potential to cause more serious failure WAO, Trieste, 24th - 28th September 2007

Ageing Machine – It’s not just equipment!

• It’s about people too!

– Loss of design and build knowledge, as staff retire • Less documentation for older equipment, drawings, maintenance schedules, etc.

• Can’t teach 20 years design, build and operational knowledge.

– Succession planning important – You only know what’s been missed when the fault occurs • Lapses in maintenance schedules • Longer fault rectification WAO, Trieste, 24th - 28th September 2007

Old RF PSU New Commercial Unit

Capital Investment and Redundancy

• Capital investment – Must be bid for against other priority items – Only essential systems and components which would significantly improve efficiency • Spares – Hot • Power Converters • Storage Ring Klystron – Vacuum vessels WAO, Trieste, 24th - 28th September 2007

Risk Analysis

• Benefit difficult to assess – The biggest risk is the one you don’t know about • At top level – Difficult to compare risk and focus resource – How do you compare extremely reliable long lead time items against components that fail frequently, but are a fast repair?

– Does highlight major issues • Linac Klystron WAO, Trieste, 24th - 28th September 2007

Risk Analysis

• Most beneficial at individual technical group level • Direct comparison can be made between equipment and fault hours contributed • Led to successful modernisation in key area’s based on safety and operational risk Location Booster Accelerator Equipment Temperature Interlock Panel Septum Extraction Septum Injection Kickers Extraction Kickers Injection 1 1 1 1 1 Likelihood of Failure Impact 1 1 2 1 2

1 1 2 1 2 09/06 4 2 1 1

Risk Exposure

10/05 09/04 4 1 6 2 2 2 3 01/03

Resources man weeks Cost 0 0 £0K £0K 2 0 2 £0K £0K £0K Comments Replaced Aug 04 and drawings updated Replaced Sept 04, drawing updated. Proven reliability. Spares available, Drawing need updating Replaced Oct 05, drawing updated. Proven reliability. Spares available, Drawing need updating WAO, Trieste, 24th - 28th September 2007

Post Incident Investigation

• Learning from incidents is important • Committee is always formed to recover and also determine actions to prevent reoccurrence.

• Actions from recent faults – Modification to TVM cooling pipes – Revised interlock testing schedule – Test rig to test thermionic valves on-site WAO, Trieste, 24th - 28th September 2007

Preventative Maintenance and Scheduling

WAO, Trieste, 24th - 28th September 2007

Conclusions

• The SRS has provided reliable user operations for over 26 years • Vacuum incidents are isolated incidents, which do not indicate any form of systematic failure which would limit the life of the SRS • Only evidence of any systematic failure due to ageing is the failure of copper elbows on water circuits • The high operating efficiency demanded by the light source community can be maintained on an ageing machine using a variety of techniques.

WAO, Trieste, 24th - 28th September 2007

Acknowledgments

• The author would like to thank

– Previous SRS Operations Managers for compiling these statistics – Current and past Operations Teams for providing the data – Steve Griffiths of EE&PS for providing his Groups Risk Analysis Data And Finally...........

Daresbury Staff past and present for their contributions to SRS Operations WAO, Trieste, 24th - 28th September 2007