Transcript PEP-II Operational Safety and Management Presentation by Roger Erickson B-Factory Operations Review SLAC, April 26, 2006

PEP-II Operational Safety
and Management
Presentation by Roger Erickson
B-Factory Operations Review
SLAC, April 26, 2006
Outline








04/26/06
Accelerator Matrix Management
Communications with Stakeholders
MCC Control Room Staffing
Operator Profile and Training
Accelerator Operations Documentation
Maintenance Management
ROD and Down Time Planning
Conclusions
B-Factory Operations Review
Main Control Center
04/26/06
B-Factory Operations Review
Matrix Management








04/26/06
Accelerator programs are managed through a matrix organization
structure with links to:
Program Coordinators for PEP-II and Linac Pulsed Programs.
Liaison persons representing each experiment.
System Physicists for each major section of the accelerator facility.
Area Managers – one for each major section, who report to the
System Physicists.
Accelerator physicists and engineers responsible for subsystems or
commissioning tasks.
Technical support departments with engineering and technician
support for on-call maintenance.
Accelerator Division Safety Office.
ES&H Departments, including Radiation Physics and OHP.
B-Factory Operations Review
Communications with Stakeholders

8:00 Meeting every day, seven days a week, in MCC conference room.
•
•
•

8:15 Maintenance Meeting, five days each week.
•
•
•

Run by Accelerator Operations Department maintenance managers.
Attended by supervisors of all maintenance groups, Area Managers, and interested
physicists and engineers.
Attendance: typically 45 persons.
Accelerator Plans Meeting, weekly.
•
•
04/26/06
Operations report by EOIC, including run time statistics, progress in carrying out
scheduled program, descriptions of problems, etc.
Reports by representatives of major experimental user groups.
Attended by PPA, Accel Div, and Operations management reps, Program Deputy,
System Physicists, Area Managers, and representatives of user groups and major
technical support departments; typical attendance: 45 persons.
Review of operational progress, problems, and plans.
Attended by Accel Systems Div and Dept Heads, Program Coordinators, Accelerator
Section Heads and ADSO.
B-Factory Operations Review
Control Room Staffing

Program Deputy
•
•
•

Operations Engineer
•
•
•
•
•

Duty rotates among Accelerator Division physicists.
Oversees program for 3 or 4 day period.
Not always present in the control room.
1 EOIC per shift (from a group of 7), on rotating 7-week cycle.
Responsible for safely carrying out the scheduled program.
Exempt position; directs the Accelerator Systems Operators.
Must be present if accelerator is capable of producing beam.
All have experience as accelerator systems operators.
Accelerator Systems Operators
Typically 3 operators per shift on rotating 5-week cycle:
•
Linac and sources.
•
PEP-II rings.
•
Secondary program, PPS control, and backup.
04/26/06
B-Factory Operations Review
Typical Operator Profile
Typical applicant is a recent graduate with a
bachelor’s degree in physics.
•
•
•
13 have bachelor’s degrees.
7 have master’s degrees.
4 have a technical background and extensive SLAC
experience.
Classifications:
•
Accelerator Systems Operator 1
• 1-2 years of experience
•
Accelerator Systems Operator 2
• 2-5 years of experience
•
Accelerator Systems Operator 3
• 5+ years of experience
•
Operations Engineer
• 5+ years of experience
04/26/06
B-Factory Operations Review
Operator Training Program

Workbook and evaluation for each classification:
•
•
•
•
Accelerator Systems Operator 1
Accelerator Systems Operator 2
Accelerator Systems Operator 3
Operations Engineer

ES&H Training (including RWT-II and LRCA)

Accelerator Operations Safety Procedures Training
(including PPS operation, searching, interlock testing for each zone.)

US Particle Accelerator School

Downtime seminars and special projects
04/26/06
B-Factory Operations Review
Accelerator Operations Documentation
Accelerator Division Operations Directives
Administrative Documents
Safety Documents
example: Administrative Guidelines
(11 titles)
Search Procedures
(45 titles)
Accelerator Operations Documents
Training Documents
Linac Turn-on Checklist
(38 titles)
ASO-2 Qualification Workbook
(17 titles)
Maintenance Documents
Documentation Documents
Call-in Lists
(4 titles)
AD Documentation Archive Plan
(5 titles)
Archived Documents and Logs
MCC Operations Log
(83 titles)
04/26/06
B-Factory Operations Review
AD Safety Documents with Site-Wide Applicability










04/26/06
SLAC Guidelines for Operations
Work Authorization for the Linac, PEP-II, FFTB, and Associated
Buildings
Summary of Requirements for Work in SLAC Accelerator Housings
Restart Plan for the Linac, PEP-II, and BABAR
Restart Plan for the FFTB and ESA Accelerator Programs as an
Extension of the Linac, PEP-II, and BABAR Restart Plan
Linear Accelerator Facility Safety Assessment Document
PEP-II Safety Assessment Document
BaBar Safety Assessment Document
Policy for Working with Electrical Hazards in SLAC Accelerator
Tunnels
Lock and Tag Verification Procedures for Accelerator Magnets in
the Linac Complex and PEP-II
B-Factory Operations Review
Accelerator Operations Safety Documents

















MCC Key Safe Checkout Log
CID PPS Log
DRIP/DR PPS Log
Linac PPS Log
Positron PPS Log
BSY PPS Log
ESA PPS Log
FFTB PPS Log
PEP PPS Log
PLS Keys Procedures and Log
MCC Spare Keys Key Safe Checkout
PPS Zone Maps
Logging Requirement for Work on
BCS Devices
VVS Remote Turn-on Procedure
Search Procedures
Electrical Hazard Test Procedures
Entry and Exit Procedures















Safety Inspection Checklists
PPS Interlock Checklists
Beam Containment System
Procedures
High-Voltage Test Facility Safety
Procedures
BSOIC Certification Checklists
Gun Safety Procedures
Laser Safety Procedures
Fire Alarm System Reference Manual
PEP PPS Overview
Safety Configuration Work Control
Incident Response Procedures
Alarm Response Procedures
Warning Response Procedures
Accelerator Operations BAS
Instructions
Accelerator Operations BAS Key and
Lock Procedures
•163 controlled documents in the Accelerator Division (incl. non-safety related).
•MCC Operations Log is now electronic, web-accessible, and archived.
04/26/06
B-Factory Operations Review
Operator Safety Procedures
Operators are trained to carry out certain well-defined tasks,
following formally controlled procedures.
Examples:
•
•
•
•
Interlock Checklists
Search Procedures
Entry and Exit Procedures
Safety Inspection Checklists
Safety procedures:
•
•
•
undergo a formal review and approval process.
are updated as conditions change.
are available on the AD Documentation web site, although the signed
originals are preserved in paper form.
Completed checklists and log books are archived.
04/26/06
B-Factory Operations Review
Maintenance Management

Problems that interrupt the scheduled program trigger immediate repairs
to restore operations.

24-hour on-site staff to repair:
•
•
•
•

On-call staff for:
•
•
•
•
•

04/26/06
Mechanical utilities
Power supplies
Modulators
Cryogenics
Electrical power
Magnets and other mechanical systems
RF systems
Vacuum systems
OHP
Extensive call-in lists identify first responders, alternates, and experts for
all systems.
B-Factory Operations Review
Opportunistic Maintenance

Opportunistic Approach
•

Coordination
•
•
•

•
•
04/26/06
Jobs that are planned and can be launched on short notice.
Provides options for EOIC to take advantage of unscheduled interruptions.
Repair Opportunity Day (ROD)
•

Daily 8:15am meeting.
Coordinated by Operations Maintenance Office.
Area Managers are advocates for their areas.
Standby Maintenance List (SML)
•

Whenever possible, identified maintenance, repair, and installation jobs are preplanned, but deferred until an opportunity arises. The accelerator systems are
not turned off for maintenance on a routine, scheduled basis.
Launched only when required by a driving event.
Carefully planned, and typically involve 100 or more jobs.
ARTEMIS Electronic Database
B-Factory Operations Review
ARTEMIS






04/26/06
ARTEMIS (Accelerator Remedy Trouble Entry and Maintenance
Information System) is an Oracle database with a Remedy interface,
accessible over the internet by everyone involved with SLAC
accelerator maintenance.
Operators report problems as they occur by making entries in
ARTEMIS.
Repair work is launched immediately if the problem is inhibiting the
scheduled accelerator program.
New ARTEMIS entries are reviewed at the daily maintenance meeting.
Open questions are addressed and jobs are assigned.
Jobs may be assigned to the Standby Maintenance List (SML), or to the
Repair Opportunity Day (ROD) list, or marked for the next long down
time.
Jobs are tracked using ARTEMIS until they are finally closed.
B-Factory Operations Review
Daily Maintenance Meeting
04/26/06
B-Factory Operations Review
Review of New Problems
Sample from an actual maintenance meeting.
04/26/06
B-Factory Operations Review
Standby Maintenance List
04/26/06
B-Factory Operations Review
Repair Opportunity Days

A ROD list of pending repair jobs is derived from the
ARTEMIS database and maintained by the Operations
Department. Installation and PM jobs are added as needed.

All ROD jobs are reviewed in the daily maintenance
meetings.

The ROD list can be viewed over the internet.

When a “driving event” occurs, a detailed plan can be
assembled and launched quickly, using the ROD list.

A typical ROD day includes about 100 distinct jobs.
04/26/06
B-Factory Operations Review
Long Down Time Planning

Long Down Times usually occur once each year and typically last two to three months.

Area Managers collect job requests from system physicists, project engineers, maintenance
groups, and the ARTEMIS database.

A comprehensive schedule is prepared, updated, and reviewed at weekly meetings, starting
about four months before the down time.

This schedule lists each job with an assigned shop, and lists the Project or Group Manager,
Responsible Task Person, and Area Manager.

Schedule specifies time and duration of each job (with 1 day time resolution), and number
of persons required.

Schedule specifies authorizations required (WAF, RSWCF, EWP, etc.).

Detailed (shift by shift) schedule of pre-run testing, safety certifications, and turn-on
activities is prepared by the Operations Dept and revised as needed on a real-time basis.
04/26/06
B-Factory Operations Review
Maintenance Statistics

ARTEMIS entries since Jan 1989: 77,328

Total identified problems not yet closed: 456
(as of April 18, 2006)
Many of these involve non-critical problems that were observed, but went away without a
clear explanation; e.g., a flow switch trip or a power supply found to be out of tolerance.

ARTEMIS entries in one year (since April 18, 2005):
•
•
•

Active jobs:
•
•
•
04/26/06
3166 new problems (8.7 per day).
450 problems resulted in lost PEP-II run time.
3296 problems closed (9.0 per day).
SML: 52
Next ROD: 32
Aug-Nov down time: 80
B-Factory Operations Review
Operational Safety Reviews

Linac and PEP-II Re-Start Validation Review (March 2005)

FFTB Re-Start Review (June 2, 2005)

ESA Re-Start Review (July 19, 2005, secondary beams only)

Director’s Integrated Safety Management (ISMS) Review (Sept 2005)

DOE ISMS Review (October 2005)

Injector Test Facility Re-Start Validation Review (October 2005)

ESHAC Review (November 2005)

ESHAC Review (March 2006)
04/26/06
B-Factory Operations Review
New Safety Processes Introduced in FY05

JHAMs (Job Hazard Analysis and Mitigation) individualized and
documented for every employee.

Expanded training requirements for most workers.

AHAs (Area Hazard Analysis) posted for every building.

Arc-flash analysis and warning labels for circuit breakers and
contactors (except Cat 0 light switches, etc.)

All hoisting and rigging equipment, including lifting fixtures, straps,
etc. must be certified.

Fall protection equipment and training required for tall ladders,
raised platforms, rooftop work, etc.
04/26/06
B-Factory Operations Review
Conclusions

PEP-II B-Factory operations are managed by the Accelerator Systems
Division in the PPA Directorate.

Control room operations are carried out by trained operators in
compliance with documented policies and procedures for all safety-related
activities.

Maintenance and installation work is accomplished through a matrix
organization, drawing on engineers, technicians, and ES&H and shop
services from the Operations Directorate.

We have developed organizational structures, software tools, and
documentation that enable us to manage an ambitious accelerator-based
science program safely and effectively.

Enhanced policies, procedures, and training programs introduced in the
past two years ensure that all our work is carried out with high standards
for safety.
04/26/06
B-Factory Operations Review