Transcript Performance Measurement of Risk Management

John Knott Associates Ltd
Performance Measurement of Risk Management
A. Cost E.
Southern Branch Technical Meeting
27th October 2009
John Knott
Principal Consultant
John Knott Associates Ltd.
John Knott Associates Ltd
What the Specification said!
Performance Measurement in Risk
Management
A classic example of one of the most common project & business
risks – seeing what we want to see and not what was asked for!
Start from the beginning!

Why Manage Risk in the first place?
John Knott Associates Ltd
Typical reasons







The customer Expects it
It is a contract requirement
Corporate governance
Company procedures
To reduce fire-fighting
To meet the key schedule milestones
To minimise overall costs
John Knott Associates Ltd
Other Reasons






Maximise utilisation of resources (Especially cash)
Manage Release of contingency / reserve to profit
Release Resources to be used on other projects or
objectives.
To Assure our future
Protect Our Reputation / Credit Rating
Think of your next job application. What do you want to
be associated with?
John Knott Associates Ltd
What is at risk?
Key Tangible Assets:





John Knott Associates Ltd
Property and Facilities
Manufacturing plant
Material stockpiles
Profit
And!

Key Intangible Assets






John Knott Associates Ltd
Business systems
Reputation (Yours and the company’s)
PR & HR
Credit Rating (and cost of capital)
CSR / Environment
Long Term Objectives (Yours and the company’s)
So?

Where do we start looking for measures of
performance?
John Knott Associates Ltd
Risk Management Maturity Level?
(M Hopkinson, De-Risk, et al)

Measures:






how often, if at all
who does and who should
ad-hoc or structured,
embedded in business practice or tacked on
Etc
Does not actually measure performance
John Knott Associates Ltd
Risk Management Maturity - 1
Acknowledgement to
De-Risk.
Strategy

1 - Initial
2 - Repeatable
3 – Defined
4 – Managed
5 – Optimised
Little or no focus on Risk
Management
Individual approaches
established and
repeatable
Consistent approach,
shared understanding
Measures and
controls established
Focus on continuous
improvement
•
Strategy does not
explicitly address risk
management
•
Concept of risk
management built into
strategy / Project Charter

Comprehensive RM
vision defined in
strategy / Project
Charter
•RM Strategy
established jointly with
clients

Project Wide
planning reflects RM
Strategy
•Strategy / Project
Charter documents
RM measures /
metrics required
•
Team R&R's do not
address RM
•Significant duplication of
effort.
•
RM roles determined by
individual projects
•Minimal duplication of
effort

RM R&R's consistent
across projects
•Programme
Management defines
roles.
•
Team RM
effectiveness is
regularly measured
and corrective action
taken
•

Team

John Knott Associates Ltd
Strategy / Project charter
documents RM continuous
improvement measures.
Team R&R's include
evaluation against
continuous improvement.
•R&R's regularly reviewed
for improvement.
Risk Management Maturity - 2
Process
1 - Initial
2 - Repeatable
3 – Defined
4 – Managed
5 – Optimised

Focus is on issues rather
than risks.
RM process is not
documented.
Driven by audit
requirements
•Little planned mitigation

RM process established
for individual projects.
Regular Risk Reviews
undertaken
Coordinated audit
preparation
•Mitigations followed
through

Consistent process
across projects.
Escalation to
programme / business
effective.
•Fully coordinated with
audit.
•
Metrics collected to
measure RM
effectiveness as part
of the process


No tools or simple
spreadsheets used in some
areas.
•No policy on use of IT
tools.

Spreadsheet or standalone
databases used on each
project
•Policy for IT support
established

Common IT RM
system used across all
projects
•Systems linked to
enable risk data sharing
and escalation
•Significant automation
to reduce administration

Project-wide
automated RM metric
generation
•Automated exception
based risk warning
indicators.

Reluctance to
acknowledge risk RM
based on management
experience and intuition
•
Individual project
managers have clearly
embraced RM

Project wide
leadership /
endorsement of
common RM principles
•Programme / Business
Management have
clearly embraced RM

Programme /
Business management
act appropriately on
metrics
•Top-to-Bottom "Walk
the talk" evident
•
IT
Culture
•
John Knott Associates Ltd
RM process focus is on
process improvement.
Process is regularly
reviewed for potential
improvement.
•Risk knowledge is
captured and re-used.
IT supports knowledge
management.
•IT supports all required
new metrics
All project team members
feel empowered to
contribute to process and
tool improvements.
•"Think Risk" culture.
Risk Management Maturity - 2
Process
1 - Initial
2 - Repeatable
3 – Defined
4 – Managed
5 – Optimised

Focus is on issues rather
than risks.
RM process is not
documented.
Driven by audit
requirements
•Little planned mitigation

RM process established
for individual projects.
Regular Risk Reviews
undertaken
Coordinated audit
preparation
•Mitigations followed
through.

Consistent process
across projects.
Escalation to
programme / business
effective.
•Fully coordinated with
audit.
•
Metrics collected
to measure RM
effectiveness as
part of the process


No tools or simple
spreadsheets used in some
areas.
•No policy on use of IT
tools.

Spreadsheet or standalone
databases used on each
project
•Policy for IT support
established

Common IT RM
system used across all
projects
•Systems linked to
enable risk data sharing
and escalation
•Significant automation
to reduce administration

Project-wide
automated RM metric
generation
•Automated exception
based risk warning
indicators.

Reluctance to
acknowledge risk RM
based on management
experience and intuition
•
Individual project
managers have clearly
embraced RM

Project wide
leadership /
endorsement of
common RM principles
•Programme / Business
Management have
clearly embraced RM

Programme /
Business management
act appropriately on
metrics
•Top-to-Bottom "Walk
the talk" evident
•
IT
Culture
•
John Knott Associates Ltd
RM process focus is on
process improvement.
Process is regularly
reviewed for potential
improvement.
•Risk knowledge is
captured and re-used.
IT supports knowledge
management.
•IT supports all required
new metrics
All project team members
feel empowered to
contribute to process and
tool improvements.
•"Think Risk" culture.
Risk Management Maturity - 2

Metrics collected to measure RM
effectiveness as part of the process
John Knott Associates Ltd
Risk Management Performance?

What Metrics to collect?





John Knott Associates Ltd
Minimising executive time spent?
Maximising Coverage?
Capturing the Important Risks?
Reducing overall Impact?
Minimising mitigation costs?
Balance?




Risk Management Maturity =
Are you doing the right THINGS?
Risk Management performance =
Are you doing things RIGHT?
John Knott Associates Ltd
What can we measure?





Risks occurred but not identified in advance.
Risks identified but occurred
Risks identified but not managed
Risks identified and successfully managed
Number of risks in each case vs total score in each
case
John Knott Associates Ltd
What we cannot measure!

Unidentified risks which did not occur!

(Unknown unknowns which remain unknown and
hence immeasurable?)
John Knott Associates Ltd
Even if we do we still might hear:



How could they have let that happen???
Could they not see that coming?????
Are they blind or stupid??

John Knott Associates Ltd
If you don’t spot risks you will be lucky to manage
them.
DNV/AIRMIC Study





Very few organisations manage all categories of risk in an integrated
fashion.
RM Software used by many organisations but many also use
spreadsheets
Risk policies influence consistency but as a rule have little direct
influence on amount of risk accepted.
There is scope for improving the quality of risk identification, even in
some of those organisations that are best at risk management.
(AIRMIC – Association of Insurance & Risk Managers)
John Knott Associates Ltd
DNV/AIRMIC Study




Very few organisations manage all categories of risk in an integrated
fashion.
RM Software used by many organisations but many also use spreadsheets
Risk policies influence consistency but as a rule have little direct influence
on amount of risk accepted.
There is scope for improving the quality of risk identification, even in
some of those organisations that are best at risk management.
John Knott Associates Ltd
Business Continuity Example

BCM HIGHLIGHTS PAINFUL TRUTHS

BY FOCUSING ON:






Company’s business and operational model
Key value creating products and services
Key dependencies –assets and processes
How the company will respond strategically to loss of threat to
any of these
What are the main threats today and on horizon
Evidence that response strategies and plans will work in practice.
L. Bird – Business Continuity Institute – Risk SIG 11-06-09
John Knott Associates Ltd
Business Models

BCM HIGHLIGHTS PAINFUL TRUTHS

BY FOCUSING ON:
 Company’s business and operational model
 Key value creating products and services
 Key dependencies –assets and processes



How the company will respond strategically to loss of threat to
any of these
What are the main threats today and on horizon
Evidence that response strategies and plans will work in practice.
L. Bird – Business Continuity Institute – Risk SIG 11-06-09
John Knott Associates Ltd
Need for structure!

Projects have used requirements breakdown,
WBS, contractor breakdown, etc structures for
years. Use these as a template for risk workshops.

Now (at last) becoming recognised that a
business model is essential for identifying
business continuity risks.

Structured Risk Identification helps avoid
blind spots.
John Knott Associates Ltd
The right approach!
This
John Knott Associates
Ltd
software will only let you add risks to a structure item
Still no quantifiable measurement!



Using a structured approach will assist in
identifying relevant risks and….
not addressing irrelevant ones.
(Every risk must have a requirement or deliverable
on which it will impact)
John Knott Associates Ltd
Where else can we look?
John Knott Associates Ltd
NAO/OGC top causes of failure
1.
2.
3.
4.
5.
6.
7.
Lack of Clear Link with strategic priorities
Lack of clear senior management ownership and leadership.
Lack of effective engagement with stakeholders.
Lack of skills and proven approach to Project and Risk Management.
Lack of understanding of/contact with supply industry at senior
levels.
Evaluation of proposals driven by initial price rather than long term
value for money. (especially securing delivery of business benefits).
Too little attention to breaking down development and
implementation into manageable steps.
John Knott Associates Ltd
Actual Case Study!
John Knott Associates Ltd
All 16 Causes of 668 Risks
Available Technology
Standards.
Adverse w eather
Non Risk.
Lessons not learned.
Systems Integration
Link to Objectives
Interfacing & Integration Companies and Personnel.
Project and Risk
Management.
ARM of Systems and
Equipment.
Requirements Definition.
Sub contractor / supplier
management
Engagement w ith
stakeholders.
Availability of SQEP
Adequacy of Historic Data
John Knott Associates Ltd
Inaccurate assumptions
Top 10 Causes – 617 Risks
Inadequate Interfacing &
Integration - Companies and
Personnel.
Availability, Reliability &
Maintainability of Systems
and Equipment.
Lack of clear link to Project
Objectives
Lack of skills and proven
approach to Project and Risk
Management.
Lack of adequate
Requirements Definition.
Inadequate Sub contractor /
supplier management
Availability of SQEP
Lack of effective
engagement w ith
stakeholders.
Adequacy of Historic Data
Inaccurate assumptions
John Knott Associates Ltd
Case Study 1
23 High Level actions to effectively mitigate 600 Risks =
Real Risk Management Efficiency.
Risk : Action Ratio = 26.8 : 1
This is an actual number that can add meaning as a measure of Risk Management
Performance.
John Knott Associates Ltd
Case Study 2
John Knott Associates Ltd
Case Study 2
John Knott Associates Ltd
Case Study 2

444 Key Risks from 7 main causes and 22 High
Level actions,

Risk Action Ratio = 20.2
John Knott Associates Ltd
Financial Viability, 1
Case Study 3
Standards, 2
Contract Change, 1
Legislation, 3
A Weapons Platform
189 Risks - 20 Causes
Obsolescence, 4
Technical Novelty, 5
Business Continuity Failure,
Accidents, Incidents, Disasters,
5
GF*, 33
Facilities, 5
Concurrent Engineering Policy,
5
Implementation of Design, 7
Y2K , 10
Requirements Definition, 29
Commercial Policy, 11
Interfacing & Integration, 12
System Design, 16
Information Flow, 13
Management Policy, 14
Human Resources, 13
John Knott Associates Ltd
Case Study 3
Y2K, 10
Commercial Policy, 11
GF*, 33
151 Risks(80% ) - 9 Causes
31 High Level Actions
Interfacing & Integration, 12
Information Flow, 13
Requirements Definition, 29
Human Resources, 13
Management Policy, 14
John Knott Associates Ltd
System Design, 16
Case Study 3

151 Risks from 9 causes and 31 high
level actions

Risk Action Ratio = 4.87
John Knott Associates Ltd
Why Manage Risk?
 It is not possible to manage risks as such - they are uncertain
events which we do not plan to occur.
 It is possible to:
 1 - avoid a risk by changing our approach or policy in
such a way that the risk cannot occur
 2 - manage the recovery of the project after a risk has
occurred.
 3 - manage the underlying causes of the potential risk
before it has a chance to occur.
 Of the above only items 1 and 3 can be considered as risk
management Item 2 is normally known as “Fire Fighting
John Knott Associates Ltd
”
Don’t Manage Risk

Identify Common Causes

Manage Key Causes

Maintain paper trail between risk/cause/action.
John Knott Associates Ltd
And Remember!

The management of common causes of risk will
provide the greatest benefit in terms of senior
and middle management time, involvement and
overall cost.

Managing key causes of risk will probably
manage risks which have not even been
identified.
John Knott Associates Ltd
Support!

“…..a single treatment may
adequately address the risk in relation
to more than one objective.”

John Knott Associates Ltd
HM Treasury “Orange Book”
Which system
This system can guide you to common causes of risk.
John Knott Associates Ltd
Sources

www.Perrox.net


www.Community.businessballs.com


Free software & guidance downloads
Perrox_Risk_Manager - guidance downloads
(Resources)
[email protected]
John Knott Associates Ltd
Thank you for listening.
John Knott Associates Ltd