Transcript Third-Party Evaluation Studies as a basis for Determining

Third-Party Evaluation Studies as a
Basis for Determining Program
Effectiveness and Improvement
Needs
Center for Research and Reform in
Education
Johns Hopkins University
Steven M. Ross, Ph.D.
Professor and Evaluation Director
The School of Education and CRRE
We want SOE to be the place where education
companies and others go for research and
development and instruction.
The Center for Research and
Reform in Education:
Evaluation Services
Independent studies of program
implementation, products, and outcomes
 Literature reviews and research papers
on selected topics
 Best Evidence Encyclopedia (BEE)

Recent and Ongoing CRRE
Evaluations












Parent Engagement and Partnership Program (EIA)
Middlebury Interactive Program (EIA)
JUMP Math in NYC
Middle School Matters
National Institute of School Leaders
Three principal preparation programs
Women’s Initiative Fellowship Program
High school reform in Minnesota
Social-emotional learning in Northern Ireland
The Leader in Me Program in two schools
Pre-K and K Early Literacy
English Language Learners in Texas
Growing Demand for Evidence

Publishers
◦ Evidence-Based Acquisition Standards

State/Local Education Agencies
◦ Contract RFP Evaluation Criteria

Federal Government
◦ Rising Standard of Efficacy

Companies
◦ Product Differentiation

Investors
◦ Culling Criteria
A Hierarchy of Program
Evaluations

Level I: Design Study
◦ What is the quality of the program design with
regard to instructional theory, logic model,
pedagogy, etc.?
◦ Focus: Systematic program review relative to
rubrics and standards.
A Hierarchy of Program
Evaluations

Level II: Development Study
◦ What is the quality of program implementation,
user satisfaction, and refinement needs? Is the
program ready for broader implementation?
◦ Focus: Case study in one or a few selected
application contexts (e.g., classrooms or schools)
A Hierarchy of Program
Evaluations

Level III: Efficacy Study
◦ What is the potential of the program to
produce educational benefits in selected
target contexts?
◦ Focus: Treatment-control comparison in a
small number of selected application
contexts (e.g., 3 program schools vs. 3
control schools. )
A Hierarchy of Program
Evaluations

Level IV: Effectiveness Study
◦ What are the effects of the program to
produce educational benefits in a broad
range of target contexts?
◦ Focus: Highly rigorous treatment-control
comparison in a large number of application
contexts (e.g., 20 program schools vs. 20
control schools)
Types of Evaluation Studies

Simplest and Least Costly
◦ Case Study
Example: Examining a middle school’s
use of a new computer program for
supplementing math instruction
Types of Evaluation Studies

Simplest and Least Costly
◦ Survey/Interview Study
Example: How 325 principals who
participated in online leadership
training react to the program and their
application of the skills taught
Types of Evaluation Studies

Simplest and Least Costly
◦ Achievement Profile Study
Example: Descriptive analysis of
posted state assessment scores for 25
schools before and after using a new
after-school program in E/LA
Types of Evaluation Studies

Medium Rigor and Cost
◦ Mixed-Methods Control Group Study
Example: Program Schools A and B are
compared on district science
assessments to Control Schools C
and D
Types of Evaluation Studies

Medium Rigor and Cost
◦ Quantitative Control Group Study
Example: Using statistical controls,
comparisons are made on school-level
AP scores in chemistry between 26
program schools and 50 control
schools
Types of Evaluation Studies

Medium Rigor and Cost
◦ Qualitative Control Group Study
Example: Through observations,
interviews, and surveys, teaching
methods and student engagement are
compared at two schools receiving
professional development in projectbased learning and two control
schools
Types of Evaluation Studies

Most Rigorous and Costly (Often
funded by federal grants)
◦ Mixed-Methods Matched Comparison Study
Example: 10 schools that elected to
use a new program are compared on
student-level test scores and qualitative
measures to 10 matched schools
serving as control sites
What Determines Rigor?

Multiple measures (triangulation)
Standardized measures (unbiased/objective)
 Treatment-control group comparisons


Equivalent comparison groups
What Determines Cost?

Accessibility of data

Cooperativeness of participants

Travel
Major Considerations

What questions do you want to answer?

How quickly do you need the answers?

What resources are available to fund the
study?

How accessible are participants and data?
Steven M. Ross
Evaluation Director, CRRE
[email protected]