Transcript Module 1 - International Atomic Energy Agency

IAEA
International Atomic Energy Agency
Outline
• Learning Objectives
• Definitions
• Formulation of
Observations
• Experience from Past
Missions
IAEA
Basic IRRS Training - Evaluation of Observations
2
Learning objectives
After finishing this lecture the trainee will:
• be aware of the observation types to be offered in
IRRS missions;
• be able to select the observation-type adequate
to the review findings;
• know the process of formulating observations in a
mission;
• have some idea about the statistical properties of
the observations.
IAEA
Basic IRRS Training - Evaluation of Observations
3
DEFINITIONS
IAEA
Basic IRRS Training - Evaluation of Observations
4
Observations and their types
In brief: observations
Observation: focused results of the review process,
based on facts and related to possible ways of
improvement or achievements to recognise, with
reference to requirements in IAEA Safety Standards
IAEA
• Observations lead to:
•
•
•
•
Recommendation
Findings
Suggestion
Good Practice
(Commendable Performance)
Basic IRRS Training - Evaluation of Observations
5
Recommendation (1)
• Recommendations are proposed where
aspects relative to the IAEA Safety
Requirements are missing, incomplete, or
inadequately implemented.
• Recommendations should be
• Specific
• Realistic
• Designed to result in tangible improvements to
regulatory effectiveness (within the individual
context of the Host Country)
IAEA
Basic IRRS Training - Evaluation of Observations
6
Recommendation (2)
• Recommendations should also
• be based on IAEA Safety Requirements, with the basis
documented (Safety Guide may not be a basis);
• be formulated in such a way that they are succinct and
self-explanatory;
• clearly specify the responsible party;
• use “should”-language.
IAEA
Basic IRRS Training - Evaluation of Observations
7
Recommendation (3)
When a recommendation is
formulated it should
• be based on checked and
evidenced factual information
• advise on what improvement in
the regulatory regime should be
made;
• not advise on how it should be
completed;
• address the root causes, rather
than the symptoms
IAEA
Basic IRRS Training - Evaluation of Observations
8
Suggestion (1)
• Suggestions are proposed when
opportunities for improvement not
directly related to inadequate
conformance with IAEA Safety
Requirements are identified.
• Suggestions may contribute to
improvements in national regulatory
arrangements but are primarily intended
to make the regulatory body’s
performance more effective or efficient
IAEA
Basic IRRS Training - Evaluation of Observations
9
Suggestion (2)
Suggestions
• should stimulate the RB’s management
•
•
•
•
•
and staff to consider new approaches;
may be proposed in conjunction with a
recommendation;
should be based either on IAEA Safety
Requirements or Safety Guides, with the
basis documented;
formulated in such a way that they are
succinct and self-explanatory;
clearly specify the responsible party;
use “should consider”-language.
IAEA
Basic IRRS Training - Evaluation of Observations
10
Good Practice (1)
• A good practice is identified in recognition
of an outstanding organization, arrangement,
programme or performance superior to those
generally observed elsewhere
• A good practice
• goes beyond the fulfilment of current requirements
•
•
•
•
or expectations;
is worthy of the attention of other regulatory bodies
as a model
might be applied elsewhere;
shall reference a basis in IAEA Safety
Requirements or Safety Guides;
shall be clearly documented in the mission report.
IAEA
Basic IRRS Training - Evaluation of Observations
11
Good Practice (2)
Thus a good practice should have the
following characteristics:
•
•
•
•
•
•
IAEA
Has proven benefit;
Be replicable and applicable elsewhere;
Be effective and creative;
Produce good results;
Does not contradicts a finding;
Be novel or unique.
Basic IRRS Training - Evaluation of Observations
12
Commendable Performance
• Where
an innovative activity is
observed which clearly strengthens
regulatory
effectiveness
and/or
organisational efficiency but is not
definably ‘superior to those generally
observed elsewhere’ it may be
referred
to
as
commendable
performance (CP).
• CP will be worthy of the attention of
other regulatory bodies as a model in
the general drive for excellence
• Not yet officially introduced
IAEA
Basic IRRS Training - Evaluation of Observations
13
Important warning
The number of Recommendations,
Suggestions and Good Practices is
no way a measure of the status of the
regulatory body. Comparison of such
numbers between IRRS missions to
different countries should not be
attempted!
IAEA
Basic IRRS Training - Evaluation of Observations
14
FORMULATION OF
OBSERVATIONS
IAEA
Basic IRRS Training - Evaluation of Observations
15
Recording Observations
• During interviews and direct observations,
reviewers should make detailed notes of
all relevant information gained.
• On the techniques and content of taking
notes see Training Lecture No. 5
(Methodology of the review)
• Note here too that the number of
observations in an IRRS mission does
not represent a measure of the current
performance of the reviewed RB relative
to any other RB.
IAEA
Basic IRRS Training - Evaluation of Observations
16
Formulation of Observations
• The reviewer is expected to formulate
in writing his/her observations daily
• Observations of the day need to be
presented on the Daily Team Meeting.
• Be aware that the first set of findings
(R+S) needs to be ready as early as
on the third day of the mission!! (For
more details see Training Lecture No.
8 on Scheduling IRRS Missions)
• Good practices may be formulated
later
IAEA
Basic IRRS Training - Evaluation of Observations
17
Format of an Observation
A description of what the reviewer observed,
that has lead to the present finding or good
practice
Quotation from the relevant IAEA Safety
Standard that is to be / has been followed.
(There may be more than one Bases)
Recommendation / suggestion / good
practice offered by the reviewer
IAEA
Basic IRRS Training - Evaluation of Observations
18
Finalization of an Observation
• Each observation is discussed and agreed upon by
the team in Daily Team Meetings
• Agreed observations are finalized and inserted into
the mission report by the reviewer who formulated
them
• Observations agreed by the team should be
discussed by the reviewers with the host
counterparts for ensuring factual correctness of the
observation
• No agreement on the findings with the host
counterpart is needed or sought!
IAEA
Basic IRRS Training - Evaluation of Observations
19
EXPERTS SAY
IAEA
Basic IRRS Training - Evaluation of Observations
20
Recommendation of Experienced Reviewers
Give particular attention to:
R,S & GP should be adequately justified in the text The text should be
based on fact and evidence (not assumptions) and the conclusions
justified against IAEA’s Standards. The text should be succinct and
self-explanatory – not an exhaustive explanation but should include
some examples.
The focus for balance, harmonization and consistency of R, S &
GPs should reflect the main purpose of the mission which is to improve
the regulation of nuclear safety for the Host Country and the
International Regulatory Community
Some of the observations may fit in principle with the concept of
Suggestions but are considered by the team to be beneath the
threshold for inclusion as formal Suggestions – these should be
considered for inclusion as observations in the main text only
In the formulation of the mission findings the team leaders should
consider the application of the principle of graded approach
IAEA
Basic IRRS Training - Evaluation of Observations
21
EXPERIENCE FROM PAST
MISSIONS TO NUCLEAR
COUNTRIES
IAEA
Basic IRRS Training - Evaluation of Observations
22
Collections of Observations
IAEA
Basic IRRS Training - Evaluation of Observations
23
Statistics of Observations (1)
Ranges of observations in initials
Ranges of observations in initials
Recommendations-EU
Recommendations-nonEU
Suggestions-EU
Suggestions-nonEU
Good Practices-EU
Good Practices-nonEU
Recommendations
Good Practices
35
30
30
25
25
20
20
15
Suggestions
15
Ranges of observations in follow-ups
Recommendations
10
Suggestions
Good Practices
10
14
12
5
5
10
8
0
0
Minimum
6
Average
Maximum
Minimum
Average
Maximum
4
2
IAEA
0
Basic IRRS Training - Evaluation of Observations
Minimum
Average
Maximum
24
Statistics of Observations (2)
The number of
findings is negatively
correlated with that
of Good Practices
Correlation of observations
All observations
R+S
GP
80
Number of observations
70
60
50
40
30
20
10
0
S R
J Q P T M E N U H O A F V L
Mission
IAEA
C D
I
G B K
Basic IRRS Training - Evaluation of Observations
25
Statistics of Observations (3)
• The number of
Correlation of Recommendations and Suggestions
R
S
recommendations and
that of suggestions are
positively correlated
(with exceptions)
35
Number of findings
30
25
20
15
10
5
Correlation of Recommendations and Good Practices
0
A
B
C
D
E
F
G
H
I
J
K
L
Mission
M
N
O
P
Q
R
T
V
R
GP
Number of findings
25
20
15
10
5
0
O
IAEA
U
30
• The number of
recommendations and
that of good practices
are strongly negatively
correlated
S
E
S
R
J
G
T
F
H
U
P
I
Mission
Basic IRRS Training - Evaluation of Observations
N
Q
L
V
A
M
C
D
K
B
26
Warning
Note once more:
• The number of observations have certain
definite statistical properties,
• as such their relative values may have
orientating merits within a given mission,
• yet their absolute values should not be
used for any comparison of separate
missions.
IAEA
Basic IRRS Training - Evaluation of Observations
27
Summary (1)
• Three types of observations have been defined:
• Recommendation
• Suggestion
• Good Practice
• Observations need to refer to IAEA Safety
Standards (Requirements or Guides)
• The various observations have specific features,
yet all need to be based on factual information
and not on assumptions
IAEA
Basic IRRS Training - Evaluation of Observations
28
Summary (2)
• Reviewers have to take notes and formulate their
observations daily
• The first set of observations is expected to be
ready at the end of the third day of the mission
• Observations are inserted into the mission report
in a standard table format
• The number of recommendations, suggestions
and good practices is in no way a measure of the
status of the regulatory body. Comparison of
such numbers between IRRS missions to various
countries should not be attempted
IAEA
Basic IRRS Training - Evaluation of Observations
29
THANK YOU FOR YOUR
ATTENTION
This activity is conducted by the IAEA, with funding by the
European Union. The views expressed in this presentation do not
necessarily reflect the views of the European Commission
IAEA
Basic IRRS Training - Evaluation of Observations
Photo: V. Friedrich
30
Discussion
• What is the difference between a
Recommendation and a Suggestion?
• When will you offer a Good Practice?
• How can the numbers of various
observations be used as some kind of
measures of performance?
IAEA
Basic IRRS Training - Evaluation of Observations
31