Transcript Assessment instruments for safety culture: what are we

ASSESSMENT INSTRUMENTS
FOR NUCLEAR SAFETY
CULTURE: WHAT ARE WE
MEASURING?
Antonio C. O. Barroso IPEN – CNEN/SP
PRESENTATION ROADMAP
Origen of the term
 IAEA guidance
 Has the concept permeated?
 Assessment instruments
 What are we measuring?

ORIGEN OF THE TERM … CHENORBYL
From the aftermath, at IAEA
Equipments
Systems
Procedures
Controls
Management
People

Organizacional Culture
A socio-technical
system
ORIGEN OF THE TERM - IAEA IN 1986
INSAG1 – The Chernobyl Accident
“A vital conclusion drawn from this behaviour is the
importance of placing complete authority and
responsibility for the safety of the plant on a senior
member of the operations staff of the plant. Of equal
importance, formal procedures must be properly
reviewed and approved must be supplemented by the
creation and maintenance of a “nuclear safety
culture”.
Let’s get the proper
understanding of
PRESENTATION ROADMAP
Origen of the term
 IAEA guidance
 Has the concept permeated?
 Assessment instruments
 What are we measuring?

IAEA GUIDANCE - BASED ON SCHEIN
Monica Haage at
ICTP 2010
[email protected]
IAEA GUIDANCE - DURING THE 90’S
IAEA INSAG 1 IAEA INSAG 4 IAEA INSAG 7 IAEA SRS 1
1992
1998
1991
1986
IAEA SRS 11
1998
A “definition”
“Safety Culture is that assembly of
characteristics and attitudes in organizations
and individuals which establishes that, as an
overriding priority, nuclear plant safety
issues receives the attention warranted by their
significance”.
IAEA GUIDANCE
AND
...MARCH 2011
2011-03-11
IAEA TECDOC
1321 2002
INSAG Series 15
2002
IAEA SRS 74
2012
IAEA TECDOC 1707
2013
PRESENTATION ROADMAP
Origen of the term
 IAEA guidance
 Has the concept permeated?
 Assessment instruments
 What are we measuring?

HAS THE CONCEPT PERMEATED?
 Humans
alone and in groups are the root
agents of organizations and, as such, we
could build upon the definition stated in
INSAG 4
 Safety Culture is the assembly of systems,
characteristics, mindset and attitudes at the
organizational and individual levels which
assures that:
(a) as an overriding priority, nuclear safety issues
receive the attention warranted by their
significance; and
 (b) adequate resources, information and
actionable knowledge are empowered at the
decision / action points where safety issues
are dealt with.

HAS THE CONCEPT PERMEATED? A
BIBLIOMETRIC VIEW
 From
Scopus data base until Dec/2011
From 1997 -2011: nuclear safety culture seemed to
be left out of the agenda of the nuclear industry.
 In relevant journals and conferences, the interest on
the subject has been near stagnant, both in terms of
publications and citations.
 At the same time for the other areas taken together,
there is a vigorous increasing interest on the subject

HAS THE CONCEPT PERMEATED? A
BIBLIOMETRIC VIEW
Safety culture in
the nuclear area
years
Publications
Citations
Average
Slope
Average
Slope
First period
1990-2000
1.82
-0.08
4.91
-0.91
Second period
2001-2011
7.45
0.06
0.55
-0.05
Overall
1990-2011
4.64
0.38
2.73
-0.42
Safety culture
outside the
nuclear area
years
Publications
Citations
Average
Slope
Average
Slope
First period
1988-94
4.14
0.86
0.00
0.00
Second period
1995-2003
27.67
4.58
55.22
19.13
Third period
2004-11
170.88
20.30
1078.63
282.44
Overall
1988-2011
68.54
10.19
380.25
71.13
HAS THE CONCEPT PERMEATED? A
BIBLIOMETRIC VIEW

It could be unfair to compare numbers of nuclear area to all
others, but it is wise and fair to compare the growth rates
Safety culture
in the nuclear
area
Overall
years
1990-2011
Publications
Citations
Average
Slope
Average
Slope
4.64
0.38
2.73
-0.42
Are we happy with that?
Safety culture
outside the
nuclear area
years
Overall
1988-2011
Publications
Citations
Average
Slope
Average
Slope
68.54
10.19
380.25
71.13
PRESENTATION ROADMAP
Origen of the term
 IAEA guidance
 Has the concept permeated?
 Assessment instruments
 What are we measuring?

ASSESSMENT INSTRUMENTS
IAEA documents provide thorough, diverse and valuable
guidance, but they are only sketchy concerning quantitative
assessment models.
Note that there are other assessment methods and we need them
all, but I am going to focus on the quantitative ones.
ASSESSMENT INSTRUMENTS - REFLECTIVE OR
FORMATIVE

Is safety culture a reflective or formative construct?
Health Diet (reflective)
Health conservation (formative)
Health
Diet
Health
Cons.
ASSESSMENT INSTRUMENTS - REFLECTIVE OR
FORMATIVE
Blood alcohol
content
Drunkness level
Formative construct
# Wine
glasses
# beer
cans
# vodka
drinks
Reflective construct
# whisky
doses
Observable variables (indicators)
not necessarily correlated
(should be complete)
Blow test
Physical
reaction test
Clocked
memory test
Observable variables (indicators)
that should be correlated
(some are enough)
ASSESSMENT INSTRUMENTS – RELIABILITY
AND VALIDITY
ASSESSMENT INSTRUMENTS – RELIABILITY
AND VALIDITY
Dimensionality of the construct
 Content validity
 Convergent validity
 Discriminant validity
 Predictive validity
 Reliability

“validation is cumulative”
multiple studies, different settings …
ASSESSMENT INSTRUMENTS - “VALIDATED”
MODELS OF NUCLEAR SAFETY CULTURE?
Web of science –
Apr/2014
PRESENTATION ROADMAP
Origen of the term
 IAEA guidance
 Has the concept permeated?
 Assessment instruments
 What are we measuring?

WHAT ARE WE MEASURING?
Article 1 - LEE, T. Assessment of safety culture at
a nuclear reprocessing plant. Work and Stress, v.
12, n. 3, p. 217-237, 1998.
Safety culture assessment of Sellafield
reprocessing plant focusing safety related
attitudes. Initial questionnaire had 172 itens and
gather data from 5926 participants. Initially EFA
by PCA resulted in 38 factors, but later reduced to
19 and 81 items. Predictive validity was done
against the number of reported accidents with
significant time loss.
The model was not further evolved and it would be
unpractical by todays standards
WHAT ARE WE MEASURING?
Article 2 - Lopes de Castro, B.; Gracia, F. J.; Peiró, J.
M.; Pietrantoni, L.; Hernández, A. Testing the
validity of the International Atomic Energy Agency
(IAEA) safety culture model. Accident Analysis and
Prevention, v. 60, p. 231-244, 2013.
They have used as model the 5 characteristics
(factors) and 37 attributes (indicators) recommended
by AEA Safety Report Series no. 42 (2005) and have
tried to validate it.
The model failed to present adequate psychometric
capacity. Most of the indicators were not good
manifestations of their factors, low face validity and
only moderate content validity. Most alarming was
the lack of discriminating validity.
WHAT ARE WE MEASURING?
• Not much, or
• Not nuclear safety culture
• But other segments are!
• For instance in Hospitals and similar organizations
many steps of the “cumulative” validation have been
done
• a model for patient safety culture, developed in
2005, has been applied, adapted and validated in
different settings (10) and countries (5)