Chapter 9 – Assessing Program Designs Alternative Design Michael Wurdeman

Bias in Estimation of Program Effects  Bias comes when either the measurement of the outcome with the program exposure or the estimate of what the outcome would have been without program exposure is higher or lower the the corresponding “ true ” value.

 Problems occur when it is not possible to view the target without the program.

Bias

Vocab Development

Selection Bias  Comparing target groups to a control group.

 Works when both groups would be the same had they both not received program, random assignment.

 Not randomly assigned groups or

Nonequivalent comparison design

Selection Bias  Volunteer  Subtle differences  Attrition  Targets drop out of the intervention or control group and cannot be reached  Targets refuse to cooperate in outcome measurement.

Other Sources of Bias  Secular Trends  Long-term trends  Interfering Events  Short-term trends  Maturation  Time

Quasi-Experimental Impact Assessment  Quasi-Experimental are designs that do not involve randomly assigned intervention and control groups.

 Use of quasi-experiments requires extreme care to ensure as much equivalence as possible between groups that will be compared.

Constructing Control Groups by Matching  Choosing Variables to Match  Use research literature to match important variables  Matching Procedures  Constructed through either individual or aggregate matching.

Equating Groups by Statistical Procedures  Multivariate Statistical Techniques  A statistical model that represents the overall set of relationships among the control variables and outcome variables.

 Modeling the Determinants of Outcome  Multiple Regression based on dependent variable.

 Modeling the Determinates of Selection  Dependent variable is selection criteria instead of outcome.

Regression-Discontinuity Designs  A quasi-experimental design in which selection into the intervention or control group is based on the observed value on an appropriate quantitative scale, with targets scoring above a designated cutting point on that scale assigned to one group and those scoring below assigned to the other. Also called a cutting-point design.

Reflexive Controls  Simple Pre-Post Studies  Before and after study  Time-Series Designs  Strongest reflexive control design  Consists of multiple observations over time. Not just pre and post.

Some Cautions About Using Quasi-Experiments for Impact Assessment  It is appropriate for evaluators to use quasi-experimental designs for impact assessments when randomized designs are not feasible, but only with considered efforts to minimize their potential for bias and acknowledgement of their limitations.