Human Reliability Assessment

Grace Kennedy [email protected]

16 th /19 th October 2006 06ELD061/06ELP461

Objectives for the Sessions

•

Understand the Human Reliability Assessment process

•

Gain practical experience of a simple task analysis

•

Gain practical experience of error identification

•

Gain practical experience of representation

•

Gain practical experience of quantifying error probabilities in a simple example

Human Error Example 1

• • • •

A KC-135 Aircraft was being pressurised at ground level.

The outflow valves were capped off during a 5 year overhaul and never re-opened.

A civilian depot technician was using a home-made gauge, and no procedure.

The technician's gauge didn't have a max "peg" for the needle which had gone round the gauge more than once.

•

The result….

Human Error Example

Human Error Example 2

Helios Crash 2005

•

Pilot misread instruments AND misinterpreted warning signals

•

Maintenance left pressure control in wrong setting

•

Manufacturer did not respond adequately to previous similar incidents Extract taken from BBC News Site

http://news.bbc.co.uk/1/hi/world/europe/6036507.stm?ls

What can be done about it?

Predicting errors

•

Task analysis and error identification Preventing errors

•

Specifying training requirements

•

Equipment design (e.g. pressure gauge)

•

Detailed procedures (administrative control) Ultimately:

•

Reduce risk

•

Save money

•

Justify design decisions “For every $1 spent in the early stage, approximately $10,000 are saved (if the problem were to be fixed later instead).” – Manprint

What is HRA?

HRA = H uman R eliability A ssessment

Process Techniques Risk Assessment

Understanding

Accident Analyses Psychology Prediction

Human Error

Environment HCI Design

HRA Process Outline

• • • •

Task analysis is used to describe and understand the human interactions with the system The results of the task analysis are used with an error taxonomy (classification scheme) to allow error identification The identified errors are analysed either qualitatively or quantitatively The process is repeated each time a design iteration occurs

Human Reliability Assessment Process

General HRA Process – Kirwan, 1994

Human Reliability Assessment Process

• • • • •

Problem Definition Task Analysis

– Describe what is done – Improve analyst’s knowledge

Error Identification

– Taxonomy – Failure criteria

Representation

– Fault tree/event tree – Risk model

Quantification

– e.g. HEART • • • •

Impact Assessment

– Effect of errors – Risk contribution

Error Reduction

– Re-design tasks – Add engineered features – Procedures / training

Quality Assurance

– Appropriate techniques – Technical checking

Documentation

Definitions of Terms

• • • •

Process:

The overall HRA process

Method:

The steps in a process –

e.g. Task Analysis, Quantification, etc

Technique:

The specific implementation of a method(s) –

e.g. HEART, THERP, etc

Tool:

A software tool to record and guide the use of a technique –

e.g. Fault Tree +

HRA Techniques

• Many HRA techniques available • Working to different levels of detail on different concepts • From expert judgement techniques (e.g. APJ, PC) • Hazard identification techniques (e.g. HAZOPS, THEA) • To quantitative techniques (e.g. HEART, THERP) • To second generation techniques (e.g. CREAM, ATHEANA)

Human Reliability Assessment Process

TASK ANALYSIS

General HRA Process – Kirwan, 1994

Task Analysis

•

Range of techniques to understand what humans are required to do in order to achieve a system goal

–

Collect and organise information

–

Improve the analyst’s understanding

–

Structured approach

–

Support to design and assessment A Guide to Task Analysis, Barry Kirwan & Les Ainsworth (1992), Taylor and Francis ISBN 07484-0058-3

Hierarchical Task Analysis

•

Expresses a job or function in terms of goals, operations and plans

–

Goals

–

Operations

–

Plans Objectives to be achieved Actions required to achieve the goals Conditions under which the actions are carried out

Hierarchical Task Analysis Example

•

Express the task of making a cup of tea using HTA

–

Goals

–

Operations Objectives to be achieved (e.g. Make Tea) Actions required to achieve the goals (e.g. Boil water, Add milk / sugar)

–

Plans Conditions under which the actions are carried out (e.g. boil the water before adding it to the cup) Example provided using TaskArchitect software

Making Tea - one solution (1 of 5)

Plan describes the logic Bar beneath the activity shows no further development Stub beneath the activity shows further development has taken place

Making Tea - one solution (2 of 5)

Making Tea - one solution (3 of 5)

Making Tea - one solution (4 of 5)

Making Tea - one solution (5 of 5)

Hierarchical Task Analysis - Practical

•

Express the task of fitting an electric plug using HTA

–

Goals

–

Operations Objectives to be achieved (e.g. Fit plug) Actions required to achieve the goals (e.g. Strip outer casing, Twist exposed wires)

–

Plans Conditions under which the actions are carried out (e.g. fit the fuse before closing up the plug to the cup)

Hierarchical Task Analysis - Guidance

•

State an overall goal (box at the top)

•

Breakdown each goal or sub-goal one at a time (i.e. finish one box before moving on)

•

Ensure all the actions under a goal are relevant and would actually achieve the stated goal

•

Keep the order and logic in the plans (make the plans specific to the goal)

Work in Pairs - 10 minutes

A solution on pink sheet

Wiring a plug – HTA one solution (1 of 4)

Wiring a plug – HTA one solution (2 of 4)

Wiring a plug – HTA one solution (3 of 4)

Wiring a plug – HTA one solution (4 of 4)

Human Reliability Assessment Process

ERROR IDENTIFICATION

General HRA Process – Kirwan, 1994

Error Identification - General

•

Task Analysis describes the activities necessary to achieve a goal

•

An Error Taxonomy (classification scheme) can be used to identify specific errors

•

Many errors will be possible, so need to understand

– –

Error effects (relating to the task goal) Failure criteria (goal failure)

•

Produce a list of identified errors, which lead to goal failure

•

Organise the information in a Tabular Task Analysis

Error Identification - Tabular Task Analysis

• • •

Use the information from the HTA Create a Tabular Task Analysis (TTA) Error taxonomy (classification scheme) to identify errors

•

Understand

–

Error effects

–

Failure criteria

•

List of identified errors

ID

0 0.1

0.2

0.3

0.3.1

0.3.2

0.3.3

0.3.4

0.3.5

0.3.6

0.4

0.4.1

0.4.2

0.4.3

0.4.4

0.4.4.1

0.4.4.2

0.4.4.3

0.5

0.5.1

0.5.2

0.5.2.1

0.5.2.2

0.5.2.3

0.5.2.4

0.5.2.5

0.5.2.6

0.5.3

0.5.4

Tabular Listing from HTA

Task

Wire an electric plug Collect tools Unscrew plug cover Prepare lead Estimate length of stripped wire required to reach earth terminal Strip outer casing according to estimate Check yellow/green wire reaches earth terminal whilst outer casing exceeds holder by 5 mm Cut blue and brown wires to reach their terminals Strip each of the coloured leads to leave exposed wire Twist exposed wires on each coloured lead

Plan

do in sequence 1-5 do in sequence 1-6 do in sequence 1-3; If mismatch between required and in-situ fuse then do ( 4) Ensure correct fuse is in place Locate appropriate instructions for equipment Read fuse requirement Compare fuse requirement with given fuse Change the fuse Select the correct fuse Extract fuse from plug Insert correct fuse Attach plug to lead Thread lead through holder Fit each twisted wire to correct terminal Select one coloured lead Identify lead based on colour (earth, live, neutral) Locate correct terminal for selected lead Unscrew terminal Place exposed twisted wire through terminal hole Tighten terminal screw Secure main lead holder Replace plug cover do in sequence 1-3 do in sequence 1-4

ID

0.4

0.4.1

0.4.2

Task

TTA Example – Selected Activities

Ensure correct fuse is in place

Plan

do in sequence 1-3; If mismatch between required and in situ fuse then do ( 4)

Error ID Error Type Immediate effects of error Detection of Recovery of error error

Locate appropriate instructions for equipment Read fuse requirement 0.4.3

0.4.4

0.4.4.1

Compare fuse requirement with given fuse Change the fuse Select the correct fuse do in sequence 1-3 0.4.4.2

Extract fuse from plug 0.4.4.3

Insert correct fuse

Error Taxonomy

•

Classification scheme

•

Generic error types

•

Similar to HAZOP guidewords

•

Taxonomy can be made domain specific

Error Taxonomy – SHERPA (see handout)

•

Example error types for an action task

–

E3 Action Omitted

–

E4 Action too much

–

E5 Action too little

–

E9 Right action wrong object

–

E10 Wrong action right object

ID

0.4

0.4.1

0.4.2

Tabular Task Analysis Example - Practical

Task Plan Error ID Error Type Immediate effects of error Detection of Recovery of error error

Ensure correct fuse is in place Locate appropriate instructions for equipment do in sequence 1-3; If mismatch between required and in E3 E16 Action omitted Instructions not obtained Wrong information obtained Instructions for another device obtained Unable to confirm fuse type May not detect Re-start task with instructions Re-start task with instructions Read fuse requirement 0.4.3

0.4.4

Compare fuse requirement with given fuse Change the fuse do in sequence 1-3 0.4.4.1

Select the correct fuse 0.4.4.2

Extract fuse from plug 0.4.4.3

Insert correct fuse

Tabular Task Analysis - Guidance

•

See Handout (blank TTA)

•

Review each activity one at a time

•

Read through the generic errors in SHERPA

•

Add error types to the TTA and fill-in the remaining columns (see example for guidance)

Work in Groups (max. 5) - 10 minutes

Tabular Task Analysis – Solution to Practical

•

See handout sheet for example error types against each activity (example is not a comprehensive record)

Example TTA - Note this TTA is not comprehensive as additional error types may apply

ID

0.4

0.4.1

0.4.2

0.4.3

Task Plan Error ID Error Type

Ensure correct do in sequence fuse is in place 1-3; If mismatch between required and in situ fuse then do ( 4) Locate appropriate instructions for equipment E3 E16 Read fuse requirement Compare fuse requirement with given fuse E3 E16 E11 E16 Action omitted Wrong information obtained Action omitted Wrong information obtained Check omitted Wrong information obtained

Immediate effects of error

Instructions not obtained Instructions for another device obtained Fuse type not obtained Incorrect fuse information used Incorrect in-situ fuse not detected Incorrect in-situ fuse not detected

Detection of error Recovery of error

Unable to confirm fuse type May not detect Unable to complete step 0.4.3

May not detect - device could fail Re-start task with instructions Re-start task with instructions Return to instructions Repeat task with correct information May not detect - device could fail May not detect - device could fail Repeat task Repeat task 0.4.4

0.4.4.1

Change the fuse Select the correct fuse do in sequence 1-3 E3 0.4.4.2

0.4.4.3

Extract fuse from plug Insert correct fuse E9 E3 E3 Action omitted Right action on wrong object Action omitted Action omitted No replacement fuse Incorrect fuse selected Unable to complete step 0.4.4.3

May not detect - device could fail Repeat task Repeat task Repeat task Original fuse left in Unable to place complete step 0.4.4.3

No fuse fitted Device does not work Repeat task E17 Misalign Fuse fitted incorrectly Device does not work Repeat task You cannot read this, but . . .

see green handout

Error Identification - Question

• • • •

Assume the fuse must be changed Review the tasks to achieve the goal at 0.4

Use the error taxonomy (SHERPA) to identify :

–

Example of an error leading to no fuse being fitted

–

Example of an error leading to the incorrect fuse being fitted Remember

–

Error Effects

–

Failure Criteria

Work in pairs - 5 minutes

Error Identification Question - One Solution

•

Errors leading to no fuse being fitted

–

Step 0.4.4.3, Error E3 Action omitted

•

Errors leading to the incorrect fuse being fitted

–

Step 0.4.3, Error E11 Check omitted

–

Step 0.4.4.1, Error E9 Right action on wrong object

Human Reliability Assessment Process

Representation

General HRA Process – Kirwan, 1994

Human Reliability Assessment and Risk Models

•

Risk models will usually include human errors for quantification (human as mitigation)

•

Human Reliability Assessor will collaborate with the Risk Modeller

–

Further investigation may be needed in order to carry out Human Reliability Assessment

–

Additional errors may be identified for inclusion in the risk model

–

Changes to models may be necessary to represent human error

Risk Assessment - General

• • •

Risk = Frequency x Consequence/Severity Assessment of a complex system requires a structured process (Probabilistic Safety Assessment)

•

Operation of the system is represented by a model (risk model) Risk model represents features in the system that prevent or mitigate against serious consequences (e.g. safety systems, intervention from human operators)

Risk Models – A Whirlwind Tour

• • • • • •

Hazard identification process used to establish a set of initiating events (what can happen to the system) Frequency of each initiator is assessed Consider the effects of each initiator on the system Typically use event trees to model accident sequences System features are ‘modelled’ as events in an Event Tree (ask success/failure questions as top events) Fault trees used to investigate detailed causes of equipment/system/human failure

An Event Tree

S Initiating event – system leak F Respond to Alarm Shut Valve Start Pump Success Success (recovered) 1 - x x Failure 1 Failure 2 Failure probability = x Success probability = 1 -x

An Event Tree - Quantified

S Initiating event – system leak F Respond to Alarm Shut Valve 0.999

0.99

0.01

0.001

Start Pump 0.99

0.01

S1 S2 P(F) = F1 + F2 = (0.999 x 0.01 x 0.01) + 0.001 = 0.0011

P(S) = 1 – Failure = 1 – 0.0011 = 0.9989

F1 F2

Fault Tree+ from Isograph (http://isograph-software.com/ftpovereta.htm)

A Fault Tree

Failure probability = A + B + C – AB – AC – BC + ABC Valve Fails to Shut OR For

OR

use A U B U C For

AND

use A n B n C A B C Electrical signal to valve fails A Mechanical valve failure B Operator fails to demand valve to shut C

Fault Tree - Practical Example

•

Create a Fault Tree for incorrect fuse in place (i.e. 0.4) Two types of boolean operators 1. OR

OR Occurrence of ANY event below causes failure above

2. AND

AND Only the occurrence of ALL events below causes failure above

Fault Tree - Practical Example Guidance

• •

Use the errors identified as the branches to the trees Think about the HTA to give an indication of the layers required

•

Think about which operator to use

Work in Groups (max. 5) - 10 minutes

A solution on blue sheet

Appropriate instructions not found

Fault Tree - A Solution

Incorrect fuse is in place OR Fuse requirements not read Fuse requirement not compared with given fuse Fuse not changed OR Correct fuse not selected Fuse not extracted from plug Correct fuse not inserted

Human Reliability Assessment Process

QUANTIFICATION

General HRA Process – Kirwan, 1994

Human Error Probabilities - Quantification

• • •

Human Reliability Assessment exists to provide quantification of the probability of human error Human Error Probabilities are used in Probabilistic Safety Assessment (risk models) Obtaining or generating Human Error Probabilities requires a range of techniques

Ways to get Quantitative Data

•

Historical records

•

Collected data (direct or simulated)

•

Estimation techniques (constructive, comparative)

•

Judgement and experience

Historical Records / Collected Data

• • • •

Number of recorded events of interest over time provides frequency of error Number of recorded events of interest over number of chances for event to occur provides the probability of error Strengths

– –

specific to the error of interest data validity (true values) Weaknesses

– – – – – –

may not have recorded all instances of error (under estimate) may need a lot of data to get a fair answer hard to identify root of some errors collection method may affect reliability collection in simulators may not be realistic for actual errors design changes over time may affect reliability

A Selection of Estimation Techniques

•

Technique for Human Error Rate Prediction (THERP)

•

Human Error Assessment and Reduction Technique (HEART)

•

Success Likelihood Index Methodology (SLIM)

•

Paired Comparisons (PC)

Estimation – THERP (1)

•

Technique for Human Error Rate Prediction (NUREG/CR-1278, 1983)

•

Collected data from civil PWRs in the USA (mainly control room actions, some manual valve actions)

•

Presented as a database of Human Error Probabilities

•

Flowchart of options to navigate the database

•

Simple error taxonomy (omission, commission)

•

Includes human error dependence model

•

Construct HEPs from basic error data

Estimation - THERP (2)

•

Strengths

–

Powerful method with good auditability and supporting qualitative material

–

Well suited to proceduralised, structured assessments

•

Weaknesses

–

Resource intensive (better with more experience)

–

Not adaptive

–

Limited error reduction information

Estimation - HEART (1)

•

Human Error Assessment and Reduction Technique (Williams, 1990)

•

Generic Task (GT) data in 9 categories

•

List of 38 Error Producing Conditions (EPC)

•

Select GT based on descriptions and examples (each task has a reliability attached)

•

Modify base reliability by considering EPCs

•

Advice included on possible error reduction measures

Estimation - HEART (2)

•

Strengths

–

Simple method, not resource intensive

–

Error reduction suggestions

–

Versatile (generic nature adapts to many tasks)

•

Weaknesses

–

Does not model dependence

–

Results can vary greatly dependent on initial assessment (GT selection) HEART – PC Demo http://www.ewe.ch/regional/regional_2_6.html

Estimation - SLIM (1)

• • • •

Success Likelihood Index Method (Embrey et al, 1984) Panel of assessors (including subject experts) Evaluate performance shaping factors (PSFs) for task of interest

•

Assign weighting as to the relative importance of the PSF to each other. Assign rating based on how useful the PSF is for the task.

•

SLI derived from the sum of the ratings and weightings (ranks the errors) Calibrate SLIs with known data to convert SLI to HEP

P4 SLI P1 P2 Known data points P3

Estimation - SLIM (2)

•

Strengths

–

Flexible technique, good theoretical method

–

Does not need task decomposition (task analysis and error taxonomies)

•

Weaknesses

–

Complex method, resource intensive

–

Lack of valid calibration data (known values)

Estimation - Paired Comparisons (1)

•

Paired comparisons (Hunns, 1982)

•

Panel of assessors (including subject experts)

•

Each assessor compares all possible pairs of error descriptions (decide which of the two is more likely for each pair)

•

Combine all comparisons made by all assessors to produce a relative scaling of error likelihood

•

Calibrate scaling with known data to convert to HEPs

Estimation - Paired Comparisons (2)

•

Strengths

–

Simple method, best use of limited known data

–

Provides information on the quality of data from individual assessors (internal consistency check)

•

Weaknesses

–

Comparisons not suited to complex tasks

–

Only suited to comparisons of similar tasks

Judgement / Experience

•

Expert judgement

Judgement/Experience – APJ (1)

•

Absolute Probability Judgement

•

Panel of experts to provide direct generation of HEPs (subject experts and HRA expert)

•

Assumes assessors are capable of making such estimates of reliability

•

Describe the tasks of interest

•

Describe errors and estimate HEPs

–

Individual estimates aggregated

–

Group consensus of estimates

Judgement / Experience - APJ (2)

•

Strengths

–

Simple method, allows constructive qualitative discussion

–

Practical error reduction measures can be discussed during the assessment

•

Weaknesses

–

Prone to biases, may have little face validity

–

Needs experienced experts

Criteria for Quantification Technique Selection

•

Availability of data

•

Applicability of data

•

Ease of use (time, cost, resources, information)

•

Data validity (justification)

•

Experience of assessor

•

Level of assessment needed

HEART - Practical Example

Use HEART to estimate the probability of fitting an electric plug to a device incorrectly

•

Assume :

–

Time shortage for the task

–

Written procedure (<10 steps) is used

–

Over the shoulder checking is carried out

HEART – See Handout

•

HEART paper

•

Generic task descriptions

•

Failure probabilities for each task description

–

(50th percentile value should be used, 5 th and 95 th percentiles indicate the uncertainty/range)

•

Error producing conditions and associated multiplication factors

HEART - Practical Example Guidance

• • • • •

Read the Generic Task Descriptions Consider the task complexity and difficulty by examining the identified errors from the TTA Select a Generic Task Review the EPCs and select the ones you believe are relevant Modify the Generic Task base HEP using factors for selected EPCs

– –

See the worked example in the HEART paper EPC equation developed to avoid negative probabilities

Work in Groups (max. 5) - 10 minutes

A solution on yellow sheet

HEART Practical - One (simple) Solution

•

Probability of fitting an electric plug to a device incorrectly

–

Generic Task F 0.003

–

EPC 2 time shortage, x 11

•

HEP = (0.003)(11) = 3.3E-2

Quantification Summary

• • • • • •

A range of HRA techniques is available Technique selection depends on the nature of the assessment Human Reliability Data can be difficult to obtain Human Reliability Data can be uncertain (range of probabilities) Information from the task analysis can be organised to suit the quantification technique (e.g. describe activities with HEART generic tasks in mind) Quantification must be based on detailed qualitative understanding

Process Acceptance

Capability Management Process for HRA CUSTOMER

Deliverables required

HR

Select & deploy team

System Requirements

Benchmarkin g External Sources

Other techniques/ external databases/ review/researc h

Regulatory Authorities

Process mgt.

Refine, tailor, what & when

Market watch Liaison Own, Maintain, Manage HRA Processes Recording mgt.

Data & process

Retrospective Analysis Training Process

BU REPS

Best practice, peer review

Database

HRA data, lessons learnt, assessments, techniques

Measures

Criteria (e.g. time, labour involved, etc.)

Product design & development process Design HRA Data Experience/ Lessons Learnt Knowledge Capture

Reporting and debriefing

Maturity Feedback

HE assessors, safety team certify

SAFETY TEAM

accident/incident

Safety Assessment Reporting

PRODUCT

Prospective Analysis HRA (concept) HRA (design) HRA (validate)

Human Reliability Assessment - Summary

•

Understand the actions being investigated

•

Use a structured approach to investigate and represent the actions (task analysis)

•

Consider the level of detail needed (compare with description detail of available data)

•

Understand the failure criteria

•

Select an appropriate technique(s)

•

Represent the identified errors in a risk model

Objectives – Re-visited

•

Understand the Human Reliability Assessment process

•

Gain practical experience of a simple task analysis

•

Gain practical experience of error identification

•

Gain practical experience of risk models

•

Gain practical experience of quantifying error probabilities in a simple example

Useful References

•

The following books provide a comprehensive guide to Human Reliability Assessment and Task Analysis A Guide to Practical Human Reliability Assessment, Barry Kirwan (1994), Taylor and Francis ISBN 07484-0111-3 A Guide to Task Analysis, Barry Kirwan & Les Ainsworth (1992), Taylor and Francis ISBN 07484-0058-3