Transcript Planning the evaluation

Planning and Design of Program Evaluation
Plan Your
Program Evaluation
Murari Suvedi, Professor
www.msu.edu/~suvedi
October 21, 2010
Identify A Program You Like to Evaluate and
Ask:
What is the goal of your program?
Who is your audience (most important one)?
What are the indicators of change, or what will show that
your project is making progress towards your goal.
What data do you need to monitor the change?
Who should collect data, when, from whom, what cost?
How to analyze data and to whom you will present
findings?
Assess Evaluation Feasibility
Ask the following questions:
 Have stakeholders agreed upon the intended use of
your evaluation?
 Is there a commitment to use evaluation findings?
 Is this evaluation a legal requirement?
 Does your program have enough impact and
importance?
 Will this evaluation provide valid and reliable
information?
 Is your program ready to be evaluated?
 Are there sufficient resources for this evaluation?
Conditions Unfavorable for Evaluation
 When the program has few routines and little stability
 When people involved cannot agree on what the
program is trying to achieve
 When sponsors of evaluation or program managers sets
stringent limits
 When there is not enough money, staff time or qualified
persons to conduct evaluation
Consider Hierarchy of Program Performance
SEE Change
Practice Change
KOSA Change
Reactions
Participation
Activities
Resources
Social, economic, environmental conditions
Patterns of behaviors, procedures, or actions
Knowledge, Opinion, Skills, Aspirations change
Positive/negative reactions
Individuals, families, groups, organizations, communities
Meetings, workshops, radio/TV programs, newsletters, etc.
$, staff time, & material resources in order to conduct program activities
Outcome Hierarchy
System/Circumstance
Behavior
Skills
Attitude
Knowledge
Awareness
Adapted from Claude Bennett , 1976
Consider Logic Evaluation Framework
INPUTS
OUTPUTS
Activities
Staff
Volunteers
Time
Money
Materials
Equipment
Technology
Partners
Workshops
Meetings
Camps
Curriculum
Publication
Media
Web site
Projects
Test plots
Field days
Research
OUTCOMES-IMPACT
Participation
Who needs to
- participate?
- be involved?
- be reached?
Number
Characteristics
Reactions
Short term Medium term
Long term
LEARNING
ACTION
IMPACT
Awareness
Behavior
Social
Knowledge
Practice
Economic
Attitudes
Decisions
Environmental
Skills
Policies
Ecological
Aspirations
Social Action
Technological
Context
Influential factors
Logic Model and Common Types of Evaluation
Needs/asset
assessment:
What are the
characteristics,
needs, priorities of
target population?
What are potential
barriers/facilitators?
What is most
appropriate to do?
Process evaluation:
How is program
implemented?
Are activities delivered
as intended? Fidelity
of implementation?
Are participants being
reached as intended?
What are participant
reactions?
Outcome evaluation:
To what extent are
desired goals and
target met?
Who is benefiting/not
benefiting? How?
What seems to work?
Not work?
What are unintended
outcomes?
Source: University of Wisconsin-Extension, Program Development and Evaluation
Impact evaluation:
To what extent can
changes be attributed
to the program?
What are the net
effects?
What are final
consequences?
Is program worth
resources it costs?
Choose Indicators for Your Program’s Evaluation
An indicator is a marker that can be observed to show that
something has changed.
Indicators should be…





Relevant to the objective
Simple and unambiguous
Realizable
Conceptually well-founded
Limited in number
Criteria for choosing
indicators:
 Measurable
 Relevant and easy to use
 Provide a representative picture
of the program
 Show trends
 Responsive to changes
 Has a reference
 Measured at a reasonable cost
Examples of Outcome Indicators
 Nutrient Use and Management
Nitrogen fertilizer use:
 Amount of decrease/increase __ lbs/acre
Use of Cover Crops:
 Amount of decrease/increase __ acres
Well water quality:
 Change in nitrate/pesticide levels __ ppm
Examples of Outcome Indicators
Quality of life/social benefits
No. work hours per day: ___ hrs/acre or head
Time for community activities: ___ hrs/week
Marketing of farm produce locally: ___% of total
 Personal & family health: ___ no. of sick days/yr
Consider Utilization-Focused Methods
 Selection of methods should be guided by:
 Goal of evaluation
 Intended use by intended users
 Credibility of information source
 Believable and understandable data
 Reliable measure: A measure is reliable to the extent that
essentially the same results can be reproduced repeatedly,
as long as the situation does not change.
Let us Plan Your Evaluation:
Complete This Worksheet and Get Ready to Share!
Major Goal of the
Program
Outputs
Activities
Program’s Name:
Major Goal or
Objectives
Participation
What are the Planned
Outcomes?
Indicators of Program ‘s
Merit/Change
Method(s) of Data
Collection
How will data be analyzed?
How will evaluation
information be
shared? Utilized?
Short- Term:
Medium-Term:
Long Term:
Will a sample be used? If yes, what
kind? How many?
Who will gather data? From whom? When?
(Before, during, or after the program?)
If you like to read further about the evaluation
models, please visit the following websites:
 Program Logic Model
See full description at:
http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html
 Targeting Outcomes of Programs (TOP) Model
See full description at:
http://citnews.unl.edu/TOP/