Transcript Performance Impact of Learning

Establishing a strategic approach to determine the measures
that reliably connect performance and training inputs
Bob Megens
BMO Financial Group Institute For Learning
STEP: May 3, 2007
16/07/2015
1
Why focus on measuring the impact of learning on performance?
BMO has set up an Our Way Model as a way of achieving the goal of a high performance culture.
Tracking performance is an essential component of BMO’s enterprise strategy of becoming a high
performing culture.
Harvard Business Review
Special Double issue – July/August 2005
BMO Financial Group
Our Way Model
The High Performance Organization
Editor Thomas A. Stewart’s
Key Things for a High Performance Organization
1.
Leadership
2.
Teams
3.
Organizational Design
4.
Learning
What are the outputs that will be used to valid the
achievement of high performance?
To what degree of accuracy are we able to
measure the contribution of these inputs to the
achievement of the outputs being measured?
16/07/2015
2
The Conference Board of Canada
Learning and Development Outlook 2007 (May), Ch. 6, Learning Evaluation
“The lack of detailed evaluation may explain [training, learning and
development] professionals’ inability to obtain higher training budgets: they
simply do not have the analytical basis to make the business case for
increasing budgets.
Although many [training, learning and development] professionals expect
their budgets to increase, their evaluation efforts suggest that they are not
fully engaged in demonstrating the impact of their current training
investments.
This is likely to reduce spending: as evaluation efforts decrease, budgets
might be expected to decrease.”
16/07/2015
3
Lexicons and Paradigms: Kirkpatrick or time for a change?
High-Impact Learning Measurement, Bersin and Associates, Nov. 2006
Level
Measure
1
Reaction
2
Learning
3
Behaviour
4
Results
5
Investment
(ROI)
While many organizations make a valiant effort to measure
specific high-value programs, most organizations do not
have a repeatable, high-value measurement process for
measuring training impact.
We believe that one of the reasons for this problem is the
limitations of the existing well-known measurement
“models,” primarily the Kirkpatrick Model – which describes
four levels for measurement: learner satisfaction, learning, job
impact and business impact. A fifth “level” (ROI or return on
investment) was added through the work of Jack Phillips4.
While Donald Kirkpatrick does not necessarily believe that ROI
is the fifth level, it is often recognized as the ultimate measure
of training impact.
While these models are widely understood and have done
much to help training managers understand some of the
possibilities of training measurement, we believe that they
tend to limit an organization’s thinking and make the
measurement process difficult to implement…
16/07/2015
While we do not want to confuse the industry with yet
another model, as a result of this research we found it
necessary to develop one of our own.
4
3 Key Criteria for linking performance and training
There were 3 criteria (which stood out from the research) that needed to be in place to
determine the impact of learning on performance.
1.
Alignment of learning with
enterprise priorities
Systematic approach
2.
Conditions that maximize
the learning investment
Concerted monitoring
Business performance
measurement and reporting
systems in place to identify
the metrics that are to be
used to isolate the
business performance
impact of learning
Consistent process
3.
16/07/2015
5
Current Approach
Consistent collection and reporting on Level 1 evaluation. While there is collection of data at higher
levels, evaluation and reporting are less consistent and not systematic.
Level 4:
- ad hoc collection
- late in the process
- no management reporting
Level 4
Level 3
Level 2
Level 1
Approach
16/07/2015
Amount of activity
Level 3:
- follow up survey
- no management reporting
- employee survey (Learning Index)
reported annually
Level 2:
- dependent upon initiative
- no management reporting.
Level 1:
- evaluation consistently applied
- quarterly reports.
What is measured and management reporting
6
2 dimensions for capturing the impact of learning on performance
Establish processes that identify opportunities to measure the impact of learning at an enterprise
level and consistently capture the impact of learning for specific initiatives and programs
1.
What are the opportunities for the collection of non-survey data?
2.
If survey data is collected, who would provide the most valid data?
Dimension
Metrics: measuring the impact of learning on business performance
1. Enterprise
Aggregate Enterprise Measurement
- human capital assessment, measuring employee performance and
capability for the entire BMO Workforce (e.g. customer service
professionalism)
- influence Customer Satisfaction surveys to capture the impact of learning
2. Specific Initiatives
Specific Strategic Learning Initiatives
- metrics to determine the impact of learning for strategic initiatives within
specific businesses (e.g. sales and service training to increase revenue)
Increase Customer Service metrics for learning at enterprise level
- establish a link with Market Research to formalize a process to align with Market Research’s
initiative in linking customer survey results to business results
- develop customer service metrics/questions that target specific learning and development
outcomes for integration into customer surveys
16/07/2015
7
Proposed Approach
The proposed approach places more emphasis on identifying robust and appropriate metrics for
determining the impact of learning on performance (Level 4) at the beginning of the engagement
process , making the agreement to collect these metrics explicit components of learning initiatives.
Level 4
Level 2
Level 3: (financial, non-financial and survey data)
- employee performance metrics
- survey executives and managers to assess the impact of
learning on employee performance
- index of employee survey performance questions
- index of performance questions from follow-up evaluations
BENCHMARKING
Level 3
Level 4: (financial, non-financial and survey data)
- focus on strategically relevant metrics (no more than 10% of
all learning initiatives)
- leverage business performance measurement systems and
coaching tools to isolate or associate the impact of learning
- survey executives and managers to assess the impact of
learning on business
- establish reporting that feeds into tracking systems
Level 1
Level 2: dependent upon initiative
Level 1: evaluation system reviewed and reported quarterly.
Approach
16/07/2015
Amount of activity
What is measured and management reporting
8
3 approaches for measuring the impact of learning on performance
Developing more rigorous metrics to demonstrate the impact of training on performance requires a
declaration of priority, sustained by a focused and managed approach.
•
1.
Attention
and
Priority
2.
Process
and
Focus
3.
Monitor
Conditions
16/07/2015
•
Track and aggregate (where possible) metrics for the impact of learning as a performance
measure for the learning function
Develop the expertise for client engagement, analysis and collection of measurement data
within the learning function
•
develop skills of individual project manager/course owners
•
establish a centre of competency for evaluation to work with projects/learning
initiatives
•
Leverage performance measurement systems used by the business
•
learning function cannot create systems to measure performance within the business
•
must leverage the financial and non-financial data collected by the business
•
begin at the scoping stage of the project, start with Level 4 metrics and seek agreement
with business client(s) on measures used
•
document measurement decisions into the project charter
•
Collect metrics that isolate or associate the impact of learning on business performance for
relevant strategic initiatives
•
validate alignment with learning objectives
•
select initiatives that best demonstrate the business value of learning (up to 10% of all
learning initiatives)
•
focus on sales and service, and leadership and performance management
•
Monitor the conditions that maximize the impact of learning on business
performance by monitoring them
9
Conditions directly affecting the impact of learning on performance
Research indicates that specific conditions need to be in place to realize the investment in
learning as a sustainable performance lift.
Conditions for Business Performance
• Reward and Recognition
• Appropriate Measurement
• Hiring Strategies
• Business process improvement and
technology
Assumed to be in place.
Raised only if they present barriers to
performance.
Business Conditions Required to
Maximize Learning Investment
 Leadership Engagement
(Funding and Support, e.g. Diagnostic)
 Appropriate target audience properly
prepared for learning
 Follow up Coaching
Potential to maximize learning’s impact on
business performance.
Monitor to ensure presence.
• Conditions taken from Phillips, Jack. Return on Investment in Training
and Performance Improvement Programs, 2nd Ed, 2003.
16/07/2015
10
Metrics that measure business and employee performance
Measuring the impact of learning on business performance will leverage business performance
measurement and reporting systems.
The following table lists metrics at the business and employee performance levels that can be used to
isolate the performance impact of learning. These metrics are particularly effective when applied to
sales and service, especially in the area of customer service.
Business Performance
Employee Performance
Increased revenue
Increased customer service
Decreased time to performance
Increased quality
Decreased cost
Increased retention
Benefit to Cost Ratio
Return on Learning Investment
Other (payback period, audit results, etc.)
Increased revenue
Increased customer satisfaction
Decreased time to performance
Increased quality
16/07/2015
11
Deriving verifiable and specific metrics
Research shows that deriving verifiable level 4 metrics to determine the performance impact of
learning using specific figures is only viable and recommended about 10% of the time.
The following criteria can be used by the learning function and the business sponsors to determine
when it is more appropriate to pursue robust data as opposed to a more associative level of evaluation
for particular learning initiatives:
- Business leader engagement
- Highly strategic initiative
- Large audience
- Greater allocation of resources
- Comprehensive and specific business
performance measurement systems
- Appropriate baseline measurements
Level 4 evaluations require greater
engagement of the business and
dedication of resources over an
agreed upon time period.
Level 4
Level 3
Level 1 and 2 Evaluations will
remain in high use due to the
relative low dedication of
resources and the ease of
implementation immediately after
the learning is completed
Level 2
Level 1
• from Phillips, Jack. Return on Investment in Training and
Performance Improvement Programs, 2nd Ed, 2003.
16/07/2015
12
Tracking impact on performance
Metrics on the impact of learning on business performance should be used as an input into
tracking of the performance of the learning function.
Business Metrics
Performance Dashboard
Financial and non financial data

use business performance measurement systems

impact of learning isolated or associated with specific
business performance

leverage business coaching tools used by business to
evaluate employee performance




Business Performance Metrics (Level 4)
Customer Satisfaction Surveys (Level 4)
Sales and Service Index (Level 4)
Employee Performance Metrics (Level 4 and Level
3)
Survey Metrics
Managerial evaluation:

surveys sent to managers to assess the impact of
learning on
 business performance
 employee performance
Employee evaluations

employee survey results

follow-up evaluations

course evaluations
16/07/2015
Managerial Summary Reports

Business performance (Level 4)

Employee performance (Level 4 and Level 3)
Employee Summary Reports

Index of AES performance questions (Level 3)

Index of performance question for follow-up
evaluations (Level 3)

Index of performance questions in course
evaluations (Level 1)
13