Transcript Introduction to RMAFs

What Difference do We Want to Make;
Developing a Strategic Framework and
Measurement Strategy for Sport
Manitoba’s PSO Funding Unit
June 5, 2008
Jennifer Birch-Jones, Senior Associate
Performance Management Network Inc.
Tel: (613) 236-2320
Fax: (613) 236-8644
Email: [email protected]
Today’s Objectives

Increased awareness and understanding of:








performance based planning, measurement and evaluation
what difference the PSO Funding Unit is trying to make
the interplay btw the PSO Funding Unit’s outcomes and the outcomes of
other Sport Manitoba programs
measurement considerations
evaluation considerations
learning considerations, including ways in which Sport Manitoba can help
build capacity internally and in PSOs
Consensus on a strategic framework / logic model
Consensus on Next Steps
2
Today’s Approach

Learn from yesterday evening’s session

Confirm key concepts and terms

Build on the good work that already has been done

Opportunity to discuss / explore issues as they relate to success

Interactive – we will build the strategic framework together

Ask lots of questions, especially if any of the terms / jargon not clear

“Why” questions not meant to challenge, but to help get to an outcome
/ key result

Parking Lot

Logistics
3
Strategic Management Cycle
Strategic Analysis
Where are we now?
Strategic Framework
Monitoring &
Evaluation
Implementation
Where do you want to be?
How do we get there?
Are we getting there?
Source: Kent and Wilkinson, Applied Strategic Planning, 1991.
4
Key Terms and Concepts




Performance-based planning combines performance measurement
and evaluation concepts and approaches with strategic planning
Move away from a focus on how busy we are (activity-based) to
what difference we are making (outcome-based)
Uses an integrated approach to planning, measurement and
reporting
Can strengthen the planning process by



Clearly articulating and agreeing on what success looks like and how
to measure and report on progress
Focusing scarce resources on measuring progress for only the most
important aspects
Adapting the level of detail to meet the information needs of the
users, e.g., Board, Minister, CEO, Director, etc.
5
Key Terms and Concepts
 Managing for results entails:




Knowing what difference you are trying to make
(strategic framework and targets)
Knowing what progress / success you are having
(monitoring and evaluation)
Finding out why something is not working (monitoring
and evaluation)
Making adjustments in your work to improve
performance (continuous improvement / learning)
6
Overview of Process
1. Develop strategic framework
2. Develop measurement / ongoing monitoring strategy


Priority performance areas for monitoring
Measurement strategy – indicators, data sources, data
collection methods, timing, responsibility, resources, whether
baseline data is needed
3. Develop evaluation framework




Type of evaluation
Evaluation questions
Timing of evaluation
Evaluation strategy - indicators, data sources, data collection
methods, timing, responsibility, resources, whether baseline
data is needed
7
Overview of Process
4. Develop learning strategy – ways in which PSOs can
better use and learn from monitoring and evaluation
– their own and Sport Manitoba’s
5. Develop / refine existing processes and tools
6. Implement, learn, refine and share
Engage PSOs at every opportunity!
Logic model / strategic framework is the basis of both
monitoring and evaluation
You can adapt monitoring and evaluation
to meet your needs (not vice versa)!
8
Key Terms and Concepts

A Strategic Framework includes:



Inputs  Activities  Outputs  Goals / Outcomes
Goals / outcomes can be Short-Term  Medium-Term  and
Long-Term Goals / Outcomes
Also known as a results chain, logic model, performance
framework

diagram showing the links from the activities through the
sequence of outcomes to the final outcomes (or results
commitments)
9
Key Terms and Concepts
inputs
HOW?
WHAT
WHO?
do we
WHERE?
want?
activities
users / clients /
direct and
final
co-deliverers /
intermediate outcomes
beneficiaries
outputs
WHY?
outcomes
“Operational”
“Behavioural Change”
“State”
10
Strategic Framework
LONG-TERM OUTCOMES
REACH
Reach is defined as
the group, or
groups, which are
reached by
program / service
outputs. The reach
category can be
segmented into
different groups,
e.g.,
Primary targets
or clients
(State level)
Long-term outcomes should relate to the mission, mandate and key results of the program/service
provider.
SHORT- AND MEDIUM-TERM OUTCOMES
(Behavioural Change level)
Direct Outcomes are the initial effect or consequence of an output. Direct outcomes include early
behavioural changes in the target group or clients, reflected in changes in awareness, client
satisfaction, acceptance and support. These lead to intermediate outcomes, such as new
knowledge, improved capability, improved behaviour (compliance) and decision making.
Action / Adoption
Ability / Competence (Capacity Building)
Awareness / Perception / Understanding
Active Partner Support (Commitment)
Partners or
Co-deliverers
Stakeholders
OUTPUTS and ACTIVITIES
(Operational level)
11
11
Key Terms and Concepts
Characteristics of a Strategic Framework




Sequence of variable length extending from inputs to final
outcomes (results commitments)
Outputs mark transition between internal and external
results (outcomes)
Partnerships externalize delivery process and lengthen
sequence of outcomes
What is achievable is directly related to:


Resources
Timeframe
12
Spheres of Influence
WHY?
(State)
End
Outcomes
Your environment of indirect influence
e.g., individuals and/or
communities of interest where you do not
make direct contact
Performance needs to
be considered in terms
of its differing spheres
of influence. Actions
in the operational
sphere should directly
lead to changes in
targeted groups which
should in turn affect
the desired ‘state’.
WHAT do we want
by WHOM?
(Behavioral Change)
Your environment of direct influence
e.g., people, groups and organizations in
direct contact with your operations
Intermediate
Outcomes
Immediate
Outcomes
HOW?
(Operational)
Your operational
environment
You have direct control
over the behaviors within
this sphere
Outputs
Sources: Van Der Heijden (1996), Montague (2000)
13
Strategic Framework
WHY
REACH
WHAT BY
WHOM
HOW
14
Office of the Auditor General – Basic Results Chain
END OUTCOMES
Office contributes to better managed government programs and better accountability to
Parliament and the public.
INTERMEDIATE OUTCOMES
Audits and studies are relevant. Critical issued and problems are addressed by audits and
agendas are changed.
Audits result in better informed legislature, government, entities, and public.
IMMEDIATE OUTCOMES
Office support for its role and its work maintained.
Clients and stakeholders engaged in audit process.
OUTPUTS AND OPERATIONAL PROCESSES
Source: Office of The Auditor General, pg. 12, Departmental Performance Report 2001
15
Measurement Considerations
 Determine the priority / important performance
areas for monitoring and why
 Identify what should be the key indicators for
these:

What would you see or hear if the expected results /
outcomes you have described are being achieved?
 Determine how, how often and who will collect this
information
 Determine the means by which to implement
16
Measurement Considerations
Sample Measurement Strategy
Performance Area
/ Outcome
Perf.
Indicator
Data
Source
Collection
Method
Responsibility
Timing
Increased
awareness of
the Health
Check (HC)
program by
consumers
The
percentage
change in the
no. of
consumers
aware of the
HC program
btw 2008 and
2011
Canadian
consumers
Consumer
survey
HC Program staff
Baseline in
April 2008,
Follow-up
in March
2011
17
Measurement Considerations
 Performance indicators (or measures) are the
specific items of information that track a program’s
success.
 PIs describe observable, measurable characteristics
or changes that represent achievement of an
outcome, e.g., number / percentage of Canadian
consumers who are aware of the Health Check
program
18
A Basic Template for Defining
Indicators
Results can be operationally defined as follows:
The Measure of
Change
By Whom
In What
Such As:
Population Target Group
Awareness
# Proportion / Percentage
Program Participant
Attitude
Rate
Client
Perception / Satisfaction
Ratio
Individual
Knowledge
Amount
Organization
Skill
‘Level’
Agency
Behaviour / Practice
Community
Example:
The percentage
of clients
Who express positive
satisfaction with the
services / advice rendered
19
Evaluation Considerations


Over and above the PSO Funding Units ongoing monitoring, is there a
need for periodic evaluation?
If so, should the focus be:





Formative?
Summative?
Issue-based?
Some combination of the above?
What would you like to learn most from an evaluation of the PSO
Funding program?
20
Evaluation Considerations
Non-profit Work and the Nature of
Evaluation (Festen and Philbin, 2007)
You do work. When you evaluate how well
you do what you do, it’s called a
process evaluation.
Your work has results. When you evaluate the
results of your work, it’s called an
outcome evaluation.
Lots of work produces multiple outcomes
over time. This equals
impact or long-term outcomes.
21
Evaluation Considerations

Formative evaluations:



Are associated primarily with analysis of program implementation
and early results / progress
Provide feedback so that implementation can be improved and
barriers to improved performance can be identified and removed
Summative evaluations:


Deal with whether the program has achieved its’ intended
outcomes.
Focuses on;



Relevance / ongoing need
Success
Cost-effectiveness and Alternatives
22
Evaluation Considerations
Sample Evaluation Strategy
Evaluation
Question
Perf.
Indicator
Data
Source
Collection
Method
Responsibility
Timing
How
successful
has PSO
funding been
in providing
a stable
environment
within which
to deliver
sport in
Manitoba?
Fluctuations
in annual
funding
amounts
(overall
budget and
to PSOs)
Extent PSOs
view funding
as providing
a stable
environment
PSO
Funding
Unit Files
File Review
PSO Funding
unit staff /
Evaluator
Summative
evaluation
in 2012
PSOs
Interviews
(Sample of
20)
Evaluator
Summative
Evaluation
in 2012
23
Evaluation Vs Monitoring
Program Evaluation
Monitoring
Episodic
Ongoing
Issue Specific
Broad issues
Measures are usually customized for
each program evaluation
Measures are developed & data are
usually gathered through routinized
process
Attribution is generally assumed
Attribution of observed outcomes is
usually a key question
Targeted resources are needed for
each program evaluation
Program evaluators are not program
managers
Source: Adapted from McDavid & Hawthorn, 2006.
Resources are usually a part of the
program
Program managers play a key role in
developing and using performance
measurement / monitoring system
Learning Considerations
 Considering how the PSO Funding Unit plans to use
monitoring and evaluation:





What are the capacity implications for PSOs?
Are we being realistic?
What can we do to make it easier for PSOs to meet our PSO
Funding Unit accountability requirements?
What can we do to help PSOs being more intentional about
using evaluation as a tool for learning?
How can we use / share the learnings with our PSOs?


Learnings about how to do monitoring and / or evaluation
effectively?
Learnings from the results of the information gathered and
analysed from our monitoring and evaluation?
25
Logic Model Worksheet
HOW?
Activities / Outputs
Def’n; An activity is an
operation or work process
internal to an organisation,
which uses inputs to
produce outputs. E.g.,
Training, research,
construction, negotiation,
investigation, etc…
WHAT WE WANT?
Direct Outcomes
Intermediate Outcomes
Final Outcomes
Def’n: An immediate outcome is
an outcome that is directly
attributable to a policy, program
or initiative's outputs. In terms of
time frame and level, these are
short-term outcomes and are
often at the level of an increase in
awareness of a target population.
Def’n: An intermediate outcome
is an outcome that is expected
to logically occur once one or
more immediate outcomes
have been achieved. In terms
of time frame and level, these
are medium term outcomes and
are often at the change of
behaviour level among a target
population.
Def’n: Final Outcome is the
highest-level outcome that can
be reasonably attributed to a
policy, program or initiative in
causal manner, and is the
consequence of one or more
intermediate outcomes having
been achieved. These
outcomes usually represent the
raison d'être of a policy,
program or initiative. They are
long-term outcomes that
represent a change of state of a
target population.
The direct outcome we would
expect to see happen / occur as a
result of each key activity of the
Office.
The outcomes which flow from
the direct outcomes and which
contribute to the final outcomes.
The end outcome(s); what we
are ultimately trying to achieve
through the contribution of our
work.
Def’n: An output is what is
produced as a
consequence of the activity,
e.g., build a website
(activity) / website (output)
Key Activities and outputs
of the Office
WHY?
Def’ns adapted from the Treasury Board Secretariat Results-based Management Lexicon.
26
Building the Strategic Framework

What is the time frame for the Strategic Framework?

What level of resources are allocated for the PSO Funding
Unit?



Existing / New?
O&M? / Other?
Staff / FTEs?
 What is the reach of the PSO Funding Unit?
27
Building the Strategic Framework
 Over the longer-term, what is the “why” of the PSO Funding
Unit?
 What are the key activity areas / outputs?
 For each of the key activity areas, what are the immediate
outcomes?
 What are the intermediate outcomes?
28
Building the Strategic Framework
 Logic check - Is the logic of the
strategic framework clear and robust:

Any logic leaps?

Any key gaps?

Any of the “reach” missing?
29
Next Steps

A good performance-based planning and reporting system
takes time, patience and persistence.

Strategic discussions along the way are as important as the
end product.

Next Steps and Timing
30
Additional Definitions and Examples
Key Definitions and Examples
Activities:
An operation or work process internal to an organisation,
intended to produce specific outputs (e.g. products or
services). Activities are the primary link in the chain
through which outcomes are achieved.
Examples:
Development of …
Preparation of …
Coordination of …
Identification of …
32
Key Definitions and Examples
Outputs:
Direct products or services stemming from the activities of a
policy, program, or initiative, and delivered to a target group
or population. Usually things you can count.
Examples:
Plans for …
Tools for …
Workshops / training sessions / etc.
Websites
Documents
Etc.
33
Key Definitions and Examples
Reach:
The individuals and organisations targeted and directly
affected by a policy, program or initiative.
Examples:
Farm organizations
Producers
Landowners
Etc.
34
Key Definitions and Examples
Outcomes:
An external consequence attributed to an organisation,
policy, program or initiative that is considered significant in
relation to its commitments. Outcomes may be described
as: immediate, intermediate or final, direct or indirect,
intended or unintended.
Examples of immediate outcomes:
Better understanding of …
Easy access to more information on …
Change in attitudes regarding ...
Enhanced participation of … in …
Increased capability of …
Opportunities for employment for … in ...
Etc.
35
Key Definitions and Examples
Examples of intermediate outcomes:
Continued employment (after end of project)
Use of information / knowledge gained / etc.
Improved decision-making
Sustained change in (behaviour or practice)
Etc.
Examples of final outcomes:
Improved socio-economic conditions
Increase in exports
Improved environment / health
Etc.
36