Transcript Using Logic Models to Enhance Evaluation - osep

Using Logic Models to
Enhance Evaluation
WESTAT
Center to Improve Project Performance (CIPP)
Office of Special Education Programs
Amy A. Germuth, Ph.D.
Compass Consulting Group
Durham NC
1
Today’s Schedule
12:30-3:30 Workshop
12:30-2:00
2:00-2:15
2:15-3:30
Workshop
Break
Workshop/Wrap-up
Please feel free to ask questions at any time
or take a break if you need it.
2
What we Will Cover

What a logic model is



Why we develop logic models
The components of a logic model
Vocabulary associated with logic models



How to develop a logic model



What is an output versus an outcome
Why satisfaction is (almost always) an output versus an outcome
Simple logic model
Complex logic model
How logic models can guide formative and summative
evaluation efforts


Formative versus summative evaluation
Connecting evaluation to the logic model
3
Learning Goals
1.
2.
3.
4.
Understand what a logic model is and why
you might want to develop one for your
program.
Understand the components of a logic
model.
How to develop a logic model for your
program.
How to use your logic model to guide
evaluation.
4
What is a Logic Model?


A depiction of a program showing what the program
will do and what it is to accomplish.
A series of “if-then” relationships that suggest
(domino effect):


If the program is implemented as intended (and
designed)
desired outcomes will be realized
A systematic and visual way to depict the
relationship that exists among the resources,
activities, and desired results of a project or
program. Thus, a completed logic model depicts
how the project will work and is sometimes referred
to as the program theory.
5
Why are Logic Models used?




To describe group work, team work, communitybased collaboratives and other complex
organizational processes.
To guide clarity on sequencing.
To guide planning, implementation, evaluation and
communication.
To help depict the theory of change associated
with a program.
6
Constructing Logic Models




Examining program descriptions, missions,
visions, plans, or extracting them from a
narrative (useful if program in place)
Forward mapping—starting with activities and
asking “so what” to generate expected outputs
and outcomes (useful if program in place)
Reverse mapping—starting with outcomes and
asking “how best to” identify the activity that is
most likely to generate them (useful if
developing a program)
Importance of incorporating Evidence Based
Practices
7
Simple Logic Model
Inputs → Outputs → Outcomes
Example:
Inputs
→ Outputs
→ Outcomes
Books
→ New vocabulary words
→ Increased reading comprehension
8
More Complex Logic Models
9
10
Another Logic Model
11
Our Logic Model Framework - 1
Objectives/Goals
Inputs→→
Activities→→
Outputs→→
Outcomes→→
Impacts
Assumptions
External Factors / Context
12
The Visual…
Need
Goal(s)
Inputs
Activities
Outputs
Outcomes
Impacts
Evidence-based practices are used to:
1)establish goals as effective and efficient ways to meet needs;
2)establish activities as effective and efficient ways to operationalize
goals;
3)establish the direct outcomes likely to result from activities; and/or
4)establish that direct outcomes eventually will lead to impacts in some
proportion of the target population.
13
Our Logic Model Framework-2



Inputs = what the program needs to invest in order to
provide activities
Activities + Outputs = what the program does
Outcomes + Impacts = who or what will change and how


Needs reflected in Legislation = The legislation that provides
resources for a program is grounded in identified or observed
needs
Assumptions + External Factors / Context = Factors that
influence getting to outcomes
14
Logic Model Definitions
1.
2.
3.
4.
5.
6.
7.
NEEDS – Inspire legislation.
GOALS / OBJECTIVES– The primary changes that
are necessary to serve the need.
INPUTS - Resources, contributions, investments
(financial and other) that go into the program.
ACTIVITIES - What you do as part of the program.
OUTPUTS - Activities, services, events and products
that reach people who participate or who are targeted.
These are often discrete.
OUTCOMES - Results or measures of change for
individuals, groups, communities, organizations, or
systems. Can be short-term and long-term.
IMPACTS – Overall result of multiple outcomes acting
together.
15
Activity 1- Output or Outcome?
Directions:
Worksheet 1 contains multiple outputs and outcomes
selected from a diverse range of policy, personnel, and
project / program evaluations.
1.
2.
On your own, identify each as an Output or
Outcome.
Next compare your results with those at your table,
and be prepared to present your findings to the
broader group.
16
Developing a Logic Model
Let’s start by developing a simple logic model
(Inputs →Outputs →Outcomes), assuming
the situation we are addressing is:
Developing and operating a science lab.
1.
2.
3.
What are possible inputs?
What are possible outputs?
What are possible outcomes?
17
Inputs:
Developing and Operating a Science Lab




Funds to buy lab equipment
Lab procedures curriculum
Guide to lab procedures and safety rules
Safety equipment
18
Outputs:
Developing and Operating a Science Lab





Number of labs completed
Number of different procedures completed
Number of times each procedure done
Number of safety rules learned
Number of safety measures taken
19
Outcomes:
Developing and Operating a Science Lab



Improved lab skills
Increased understanding of safety needs
Increased number of safety precautions
taken
20
Activity 2 –
Another Simple Logic Model

Situation: Family vacation. Let’s distinguish between
inputs and activities on this one.
Inputs – what we invest
Activities – what we do
Outputs – who we reach and how
Outcomes – what results
What would a potential logic model look like?
21
Adding More Detail:
Short- and Long-term Outcomes
The USED Math-Science Partnership
Program was developed to increase students’
achievement in math and science. The
assumption behind this program is that by
providing teachers professional development
in math and science content and pedagogy,
teachers would be better prepared to help
their students attain greater achievements in
these areas.
22
MSP Program Logic Model
Inputs: $$$, trainers, college professors, materials,
technology
Activities: Professional development in content and
pedagogy, shared planning time, departmental meetings,
book discussions
Outputs: Attendance at PD, feedback on PD, # persons
trained, average number of hours of training received
Outcomes: ??????
Impact: Increased student achievement
23
MSP Program Logic Model:
Short- and Long-term Outcomes
What are some short-term outcomes associated
with teachers?
What about with students?
What are some long-term outcomes associated
with teachers?
What about with students?
24
Additional Considerations

Assumptions - the beliefs we have about the program, the
people involved, the context in which the program is operating ,
and the way we think the program will work.

Objectives –our goals / objectives for our program – these are
based on our assumptions and guide our inputs and activities.

External Factors / Context - the context or environment in
which the program exists. This includes external factors that
interact with and influence the program action (e.g., need,
interest, economic factors, etc.). They may guide your
assumptions and choices of inputs and activities, but also may
be things that affect outcomes and impacts.
25
Activity 3:
Putting it all Together
In groups, please review the scenario/ program assigned to
you and identify the following:
Assumptions
External Factors / Context
Objectives / Goals
Inputs
Activities
Outputs
Outcomes (short-term and long-term)
Impacts
The more detail / specific, the more useful your
logic model will be, particularly for guiding your evaluation!
26
Using Logic Models
to Guide Evaluation
Logic Models are extremely useful in
guiding both formative and summative
evaluation efforts.
- Formative evaluation is undertaken to assess whether
the program is proceeding as planned, the fidelity of
implementation to program design, and the degree to
which changes need to be made.
- Summative evaluation is conducted to asses whether
planned outcomes have been achieved and what
impacts (intended and unattended) have occurred.
27
Logic Models and
Formative Evaluation
Logic Models identify the key inputs, activities, and outputs
associated with the program. This information in turn identifies
data sources and key data that can be used to answer such
questions as:
1.
2.
3.
4.
5.
Are key inputs being utilized as expected? As designed?
How are activities being received by participants?
Are key outputs occurring at the desired level?
What do participants and stakeholders like about the current
program?
What changes do they suggest should be made?
What other questions might we ask?
What other data sources can we consider?
What data should we collect?
28
Logic Models and
Summative Evaluation
Logic Models also identify key outcomes or measurable changes
and impacts associated with the program. This information in turn
identifies data sources and key data that can be used to answer
such questions as:
1.
2.
3.
4.
5.
6.
7.
8.
What outcomes (expected and unexpected) have occurred?
What expected outcomes have not occurred?
Where is change the greatest?
What impacts (expected and unexpected) have occurred?
What expected impacts have not occurred?
To what degree have outcomes occurred?
What is the unique contribution of the program to the observed
change?
What is the cost / benefit of these outcomes?
What other questions might we ask?
What other data sources can we consider?
What data should we collect?
29
Updating our Logic Model
These questions, data sources, and data to
be collected can easily be added to our logic
model.
30
Activity 4 –
Adding Evaluation Components
Using the same scenario/program from
Activity 3, work in your group to identify
potential formative and summative evaluation
questions, and data / data sources needed to
answer them.
Be prepared to share these with the broader
group.
31
References (Web)

Kellogg Foundation Logic Model Development Guide
http://www.wkkf.org

CDC Guide on Logic Model development
http://www.ojp.usdoj.gov/BJA/evaluation/guide/docume
nts/cdc-logic-model-development.pdf

University of Wisconsin Extension Program Logic
Model Development Guide
http://www.uwex.edu/ces/pdande/evaluation/evallogicm
odel.html

InnoNetTool for Planning and Evaluation
http://www.innonet.org/index.php?section_id=64&conte32
nt_id=185
References (Texts)

Logic Modeling Methods in Program
Evaluation: Research Methods for the Social
Sciences - J Frechtling

Theory Driven Evaluations – H. Chen
33
The End
Questions?
34
CIPP Hotline Information
Email: [email protected]
Toll-free: (888) 843 - 4101
35