Transcript Program Evaluation and the Logic Model

Program Evaluation and the
Logic Model
Research Methods for Public
Administrators
Dr. Gail Johnson
Dr. G. Johnson,
www.researchdemystified.org
1
What to Evaluate?
 Projects: a single intervention in one
location or a single project implemented in
several locations.
 Programs: an intervention comprised of
various activities or projects which are
intended to contribute to a common goal.
 Organizations: multiple intervention
programs delivered by an organization.
Dr. G. Johnson,
www.researchdemystified.org
2
When to Evaluate?
 Before program starts:
 To
improve design
 During the implementation:
 To
improve implementation
 Identify barriers to be removed
 Lessons learned about implementation
Dr. G. Johnson,
www.researchdemystified.org
3
When to Evaluate
 Mid-term evaluation
 Relevance, effectiveness, efficiency
 Lessons learned: management tool
 Impact evaluation
 Either at the end of the project or a few years after the
program has been operating: assessing a mature
program
 Can also look at: effectiveness, efficiency, early signs
of impact and sustainability
 Lessons learned: future projects
Dr. G. Johnson,
www.researchdemystified.org
4
Why Is Evaluation Useful?
 Feedback
 Accountability
 Learning
 Improvement
 Results
 Testing underlying assumptions or theory
 Funding decisions
Dr. G. Johnson,
www.researchdemystified.org
5
Fear of Evaluation
If evaluation is so useful, why do some people
fear evaluation?
Dr. G. Johnson,
www.researchdemystified.org
6
Evaluation Questions
Did the promised
Compliance/
Accountability
Questions
“How” Questions
Impact
Questions
activities actually take
place as they were
planned?
What was the sequence
or processes that led to
successful (or not)
outcomes
Did the program achieve
the desired results?
Dr. G. Johnson,
www.researchdemystified.org
7
Types of Evaluations
 Auditing: accounting for money

Is the money being spent according to plan?
 Efficiency and effectiveness.
 Monitoring: measuring implementation and
results
 Is the intervention producing the intended
results?
 Process: measuring operations and service delivery
 Are there problems in service delivery?
Dr. G. Johnson,
www.researchdemystified.org
8
Types of Program Evaluations
 Feasibility evaluations
 Before
the program begins
 Intended to improve program design
 Evaluability assessments
 Assesses potential usefulness of the evaluation
 Used to test out different strategies for
conducting an evaluation
 What is doable given the situation?
Dr. G. Johnson,
www.researchdemystified.org
9
Evaluability Assessment
 Helps to define the actual objectives,
implementation and management of a
program.
 The
actual objectives may differ from those
initially planned.
 Determines the coherence of the program:
are goals, activities, program infrastructure
linked?
Dr. G. Johnson,
www.researchdemystified.org
10
Evaluability Assessment
Key steps in the process:
 Interview key program staff to actual program
mission, goals, objectives and activities.
 Site visits to observe and get a sense of what is
going on.
 May include interviews with key stakeholders.
 Observe program delivery.
Dr. G. Johnson,
www.researchdemystified.org
11
Evaluability Assessment
 Reach agreement as to:
 Whether
to conduct the evaluation.
 Scope and objectives of the evaluation.
 The decision could be to not conduct the
evaluation.
Dr. G. Johnson,
www.researchdemystified.org
12
Evaluability Assessment:
Challenges
 Key components of the program may not be
well defined:
 Lack
of agreement on program objectives.
 Lack clear, measurable indicators of
performance and/or impact.
 Target group may not be clearly defined.
 The delivery system is poorly articulated.
Dr. G. Johnson,
www.researchdemystified.org
13
Types of Program Evaluations
 Formative evaluations
 During
implementation
 Feedback about operations and processes
 Used to make mid-course corrections
Dr. G. Johnson,
www.researchdemystified.org
14
Definition: Performance Monitoring


Performance monitoring:
 the continuous process of collecting and
analyzing data to compare how well a
project, program or policy is being
implemented against expected results.
Traditional:
 focus on inputs, activities and outputs.
Dr. G. Johnson,
www.researchdemystified.org
15
Types of Evaluation:Monitoring
 On-going review:
 On-time
 On-budget
 On-target
 Linked with on-going management
 Measured
against established baselines
 Indicators of progress toward targets
Dr. G. Johnson,
www.researchdemystified.org
16
Types of Program Evaluations
 Summative Evaluations




At the end of the program or after the program has been
running long enough to achieve its goals (with
“mature” programs)
Identify lessons learned
Other issues: unintended outcomes, program
sustainability, program efficiency, costs and benefits
Sometimes called impact evaluations and ex-post
evaluations
Dr. G. Johnson,
www.researchdemystified.org
17
Program Evaluation
 Summative Evaluation Question:
Do Public Programs Work?
 Implied cause-effect relationship
 Did the program cause a desired outcome?
 Performance-based:
Focus on outcomes, results, impacts, goal
achievement.
Dr. G. Johnson,
www.researchdemystified.org
18
Differences
 Formative Evaluations
 Project
Monitoring
 Early Years of Implementation
 Key Question:

Are we doing things right?
–
–
–
–
Have we hired the right people with the right skills?
Have we marketed the program effectively?
Have we met our strategic objectives?
Have we spent our money according to our plan?
Dr. G. Johnson,
www.researchdemystified.org
19
Differences
 Summative Evaluations
 Measuring Results or Impacts
 A longer time before results or impacts are visible
 Key Question:


Are we doing the right thing?
This gets back to the theory or underlying assumptions of the
program:
– We can do an excellent job at training people but if the problem
is not about the larger structural economic issues, a training
program, no matter how well implemented, may show little
result.
Dr. G. Johnson,
www.researchdemystified.org
20
Working with Models
 Visualize a program in context
 Systems
approach, within an environment
 Identify the relationships between various
components
 Identify cause and effect
 Identify key assumptions
Dr. G. Johnson,
www.researchdemystified.org
21
Models: Cause and Effect:
Did the program cause something to
happen?
Education
Employment
Dr. G. Johnson,
www.researchdemystified.org
22
Hierarchy of Objectives
Reduced Poverty
Improved Quality of Life
Increased Income
Job
Training
Unemployed
Dr. G. Johnson,
www.researchdemystified.org
23
Logic Models
 The focus is on results or impacts rather than
inputs and activities



We are not training people just for the sake of training
people
We believe if we train the chronically unemployed, then
there quality of life will be improved and poverty will
decrease.
Our goal is to reduce poverty
 Also called Program Outcome Model or
Measuring for Results
Dr. G. Johnson,
www.researchdemystified.org
24
Logic Model
Impact
Outcomes
Outputs
Activities
Inputs
Dr. G. Johnson,
www.researchdemystified.org
25
Elements of the Logic Model
Inputs: what resources are used
 University inputs: budget, number of
faculty, number of staff, number of
buildings, number of classrooms
 Activities: what the program does
 University activities: teaching, research,
and service
Dr. G. Johnson,
www.researchdemystified.org
26
Elements of the Logic Model
 Outputs: the services or products produced
 University
outputs: number of students that
graduate, number of articles and books
published by faculty
 Outcomes: what happened: immediate results
 Graduates are sought after, get good jobs, active
alumni who donate big bucks
 Faculty well-known, obtain big grants, enhance
rating of university
Dr. G. Johnson,
www.researchdemystified.org
27
Elements of the Logic Model
 Impacts: the “so what.” Larger, long term
results, usually tied to program goals.
 A more informed and engaged citizenry,
preserves democratic institutions, future
leaders. Faculty research contributes to
knowledge.
Dr. G. Johnson,
www.researchdemystified.org
28
Logic Model
Logical Connections:
 Inputs
to do activities
 Activities lead to outputs
 Outputs lead to one or more outcomes
 Outcomes lead to impacts
Dr. G. Johnson,
www.researchdemystified.org
29
Logic Model: Training Program
Activities
Outputs
•Training
Resources Programs
Products
Inputs
•money
•staff
•Supplies
•mentors
•Dress for
success
coaching
•Interview
coaching
•Resume
assistance
•Number
of
graduates
per training
session
•%
graduate
rate
Outcomes
Benefits
changes
•Increased
skills
•% Obtain jobs
•% Obtain high
paying, quality
jobs
•Increased
self-esteem
Dr. G. Johnson,
www.researchdemystified.org
Impacts
Goals
•Increased
income
•Selfsufficiency
•Family
stability
•Reduction
in poverty
30
Takeaway Lesson
 Program evaluation takes many forms but
all follow the same research planning
process.
 Evaluation of programs have shifted from
reporting inputs and activities to attempting
to measure results: the difference the
program actually made.
Dr. G. Johnson,
www.researchdemystified.org
31
Takeaway Lesson
 Measuring results is harder than it appears.
 Program
goals and objectives may be fuzzy.
 Sites may vary in how they have implemented
the program.
 It takes money to collect and analyze data.
 Results may not be observable for many years.
 The operating environment may make it hard to
see results.
Dr. G. Johnson,
www.researchdemystified.org
32
Takeaway Lesson
 Measuring results is harder than it appears.
 But
there is much that can be learned from
engaging in this process and doing the best job
possible.
 Remember: do not quickly conclude a program
does not work just because you cannot measure
the result. The research tools available may not
be up to the job.
Dr. G. Johnson,
www.researchdemystified.org
33
Creative Commons
 This powerpoint is meant to be used and
shared with attribution
 Please provide feedback
 If you make changes, please share freely
and send me a copy of changes:
 [email protected]
 Visit www.creativecommons.org for more
information
Dr. G. Johnson,
www.researchdemystified.org
34