Transcript Assessing Program Impact - Littleton Public Schools

Assessing Program
Impact
Chapter 8
Impact assessments
answer…
• Does a program really work?
• Does a program produce desired effects
over and above what would have occurred
without the intervention?
• Are there important unintended effects?
Impact Assessments
can be relevant at many points
• Policy formation
• Evaluating a pilot program
• Ongoing programs – may need to modify or
refine
• high costs of programs – must continually
evaluate their efficacy
• demonstrations of effectiveness are required to
renew grants
• justify uses of public funds
Prerequisite conditions
• impact assessments build on earlier forms
of evaluation
• the program’s objectives must be well
articulated and have plausible,
measurable outcomes
• intervention is well implemented and has
been in place for sufficient time
Equivalence
•
•
•
identical composition - same mixes in
terms of their program-related and
outcome-related characteristics
identical predispositions - equally
disposed toward the project and equally
likely to attain any given outcome status
identical experiences – over the time of
observation
If two groups are “equivalent”, both are
subject to the same degree of change
induced by factors outside of the
program.
Any difference in outcome between
them, therefore, should represent the
effect of the program.
Two classes of Approaches:
Experimental vs Quasi-Experimental
Research Designs
Randomized field experiment
• the “gold standard” research design
• Participants are randomly sorted into at least two groups
– a control group and an intervention group
Quasi-experiments
•
Nonrandomized in which participants (given
intervention) are compared with nonparticipants (the
controls)
Randomized Field Experiments
Principal advantage:
• it isolates the effect of the intervention being
evaluated by ensuring that intervention and
control groups are statistically equivalent except
for the intervention received
Best way to achieve equivalence!
Randomization is just not a “shot in
the dark
• Allocating targets to intervention and
control groups requires considerable care
• The evaluator must use an explicit
chance-based procedure
• Be careful when using lists
• use statistical significance testing to judge
whether a specific difference is likely to
have occurred simply by chance
Data Collection Strategies
Two strategies can improve the estimates of
program effects
• Make multiple measurements of the
outcome variable
• Collect data periodically during the course
of an intervention
Units of Analysis
• the units on which the outcome measures
are taken
• should be based on the nature of the
intervention and the target units
Be careful!
• Observational vs. experimental units
• Assumption of independence
Limitations of Randomized
Experiments
• These research designs are not applicable to all
program situations
• The program usually has to be stable and
operationally mature for authentic results
• If the program changes during the course of the
experiment, it is difficult to differentiate between
what program version produced what effects.
• Difference between Experimental and Actual
program delivery
Ethical Considerations
• ethical qualms about randomization –
deprives control groups of positive benefits
If program resources are scarce
• do you allot services by chance?
• or insist that the most needy targets
receive priority?
The next chapter that will look at alternative designs.
“Perfect” vs “Good Enough”
•
•
•
•
•
•
time and resource constraints
intended use of the results
feasibility of design
ethical considerations
credibility
differences between experimental and actual
intervention delivery
• high turnover in policy considerations
• integrity of a randomized experiment is
easily threatened
Impact Assessments Overview
• All impact assessments are comparative – with a
group receiving alternative services or no
treatment.
• Ideally, the conditions being compared should be
identical in all respects except for the
intervention
• All assessment involves establishing control
conditions
• The most valid results generally require more
skills, more time to complete and more cost.
Beware! Rigorous Impact Assessments
involve technical and managerial
challenges and significant resources sometimes has political pressures
Is it justified by the circumstances?