Transcript Chapter 4 The Designing Research Consumer High Quality Research: Evaluating Research Design High quality evaluation research uses the scientific method to investigate the effectiveness.

Chapter 4
The Designing Research Consumer
1
High Quality Research: Evaluating Research
Design
High quality evaluation research uses the scientific
method to investigate the effectiveness of programs
and practices.
Some evaluation studies are higher quality than
others, and the research consumer must learn to
distinguish among them.
One important index of quality is the rigor of the
study’s research design.
2
Research Design: The Structure of a Study
An evaluation’s research design is its structure.
At its most stingy, the structure consists of
 the new program
 study participants
 timing of outcome measures
The research consumer must be able to distinguish
between experimental and observational designs
and what makes them internally and externally valid.
3
The Randomized Controlled Trial (RCT): Going
for the Gold
An RCT is an experimental study in which eligible individuals
or groups of individuals (e.g., schools, communities) are
assigned at random to receive one of several programs or
interventions.


The group in an experiment that receives the specified
program is called the experimental group.
The term control group refers to another group assigned to
the experiment, but not for the purpose of being exposed to
the program.
The performance of the control group usually serves as a
standard against which to measure the effect of the program
on the experimental group.
4
The Randomized Controlled Trial: Going for
the Gold (Continued)


The control program may be typical practice (“usual care”),
an alternative practice, or a placebo (a treatment or program
believed to be inert or innocuous).
Random assignment means that people end up in the
experimental or in the control group by chance rather than
by choice.
In some randomized studies, the participants and investigators
do not know which participants are in the experimental or the
control groups: This is the double-blind experiment. When
participants do not know, but investigators do, this is called
the blinded trial.
5
The Randomized Controlled Trial: Going for
the Gold (Continued)
Randomized controlled trials are sometimes called
true experiments because, at their best, they can
demonstrate causality.
That means that, in theory, the researcher can
assume that if participants in an RCT achieve
desirable outcomes, the program caused them.
6
Randomized Controlled Trials in Action
Two commonly used randomized control designs
are:
1. Concurrent controls in which two (or more)
groups are randomly constituted and they are
studied at the same time (concurrently).
2. Wait-list controls in which one group receives the
program first; if the program appears to be effective,
participants on the wait list receive it.
Participants are randomly assigned to the experimental
and wait-list groups.
7
Non Randomized Controlled Trials (QuasiExperiments) and Observational Studies
True experiments (randomized controlled trials) are contrasted
with non randomized controlled trials and observational
studies.
In non randomized controlled trials, the control group is
predetermined (without random assignment) to be comparable
to the program group
Non randomized controlled trials are also called quasi
experiments.
In observational designs, the evaluator does not intervene but
studies the effects of already existing programs

Observational designs are sometimes referred to as
descriptive.
8
Non Randomized Controlled Trials
Non randomized controlled trials rely on participants
who -1) volunteer to join the study OR
2) are geographically close to the study site OR
3) conveniently turn up (at a clinic, school) while the
study is being conducted
Because the study groups are opportunistically
rather than randomly composed, study group
characteristics (age, sex) may not be balanced
before (at baseline) the study begins.
Baseline differences between groups may confound
the study’s results.
9
Quasi Experimental Designs or Non
Randomized Controlled Trials (Continued)
Typical confounding variables include age, educational
level, motivation, severity of illness, social structure, and
income.
Evaluation researchers worry that study groups in non
randomized trials will differ from one another at baseline,
and the study’s findings will be compromised.
They aim to create study groups that are as similar to one
another as possible (equivalent) at baseline or before
“treatment.”
Among the strategies commonly used to ensure equivalence
is one called matching.
10
Quasi Experimental Designs or Non
Randomized Controlled Trials (Continued)
Matching requires selecting pairs of participants or
clusters of individuals who are comparable to one
another on important variables.

A researcher who is interested in comparing the
acuity of vision among smokers and non smokers
can try to balance the two groups by selecting
pairs of smokers and non smokers who are same
age, sex, and have the same medical history
Statistical methods such as analysis of covariance
and propensity score analysis are sometimes used
to deal with the problem of confounding after the
data are collected for the study.
11
More Quasi Experimental Designs
Time-series Designs

Time-series designs are longitudinal studies that enable the
researcher to monitor change from one time to the next.

They are sometimes called repeated measures analyses.
Interrupted or Single Time-series

The interrupted or single time-series design without a
control group involves repeated measurement of a variable
(e.g., reported crime) before and after implementation of a
program.

The goal is to evaluate whether the program has
"interrupted" or changed a pattern established before the
program's implementation.
12
Even More Quasi Experimental Designs
Self-Controlled or Pretest-Post Test Designs

Each participant is measured on some important program
variable and serves as his or her own control.

Participants are usually measured twice (at baseline and
after program participation), but they many be measured
multiple times afterward as well.
Historical Controls

Investigators compare outcomes among participants who
receive a new program with outcomes among a previous
group of participants who received the standard program.
13
Observational Designs
Cohort Designs
A cohort is a group of people who have something in common
and who remain part of a study group over an extended period
of time.
In public health research, cohort studies are used to describe
and predict the risk factors for a disease and the disease's
cause, incidence, natural history and prognosis. They tend to
be extremely large studies.
Cohort studies may be prospective or retrospective. With a
prospective design, the direction of inquiry is forward in time,
while with a retrospective design, the direction is backward in
time.
14
More Observational Designs
Case Control Designs
 Case-control designs are generally retrospective.
They are used to explain why a phenomenon
currently exists by comparing the histories of two
different groups, one of which is involved in the
phenomenon.
 For example, a case control design might be used to
help understand the social, demographic, and attitudinal
variables that distinguish people who at the present time
have been identified with frequent headaches from those
who do not at the present time have frequent headaches.
15
More Observational Designs
Cross-Sectional Designs


Cross-sectional designs result in a portrait of one
or many groups at one period of time.
They are sometimes called descriptive, pre
experimental, or survey designs.
16
Internal and External Validity:
Quality
The Route to
Internal Validity is the ability to make accurate
inferences about a program’s outcomes and
effectiveness. (Program A caused Outcome A.)
External validity refers to the extent to which the
results are applicable to other programs,
populations, and settings. Another term for external
validity is generalizability.
17
Internal Validity: How it is Jeopardized
The research consumer wants to be sure that
Program A which was conducted in Setting A
1. caused Outcome A (internal validity) and
2. will be effective in Settings B and C, etc. (external
validity)
Study design flaws can “threaten” internal and
external validity.
18
Internal Validity and Its Threats
Threat to Validity
Explanation
Selection
Group characteristics (e.g., age,
gender) are not evenly balanced
History
Unanticipated outside events
occur while study is in progress
Maturation
Participants change or mature
Testing
Taking one test effects
performance on the second
19
Internal Validity and Its Threats (Continued)
Threat to Validity
Explanation
Instrumentation
Changes occur in measures or
observers
Statistical regression
Participants who are selected
because of extremely high or
low scores return to the average
over time
Attrition
Participants drop out of the
study in a non random manner
Expectancy
Participants or researchers
have expectations of the
program and study activities
20
External Validity and Its Threats
Threat to External Validity
Explanation
Interaction between selection
and experiment
A unique mixture is created that
is unlikely to occur again
Reactive effects of testing
Completing the baseline
measures influences program
participation uniquely
Reactive effects of
experimentation (Hawthorne
Effect)
Study participants behave
uncharacteristically because
they are being observed
Multiple program interference
Complementary activities may
influence participants’ behavior
and study outcomes
21