Transcript Adjusted Marginal Means

DYS StARR Project
March 2010
SREE Presentation, Washington DC

Ohio Department of Youth Services
Striving Readers Grant Funded by IES
Project Evaluation By Ohio State University
Issues on Design and Impact from a
Randomized Control Trials Study
 William E. Loadman, PI
 Raeal Moore
 Jing Zhu
 Richard Lomax

Local Context
Incarcerated youth ages 12-21
 7 High Schools (6 Male 1 Female)
 Highly restrictive environments
 Students with limited reading ability

– 2/3 below grade level
Average length of stay 10.5 months
 Students with concomitant problems
 No consistency on entry and exit of
students

Mobility Issues

ODYS release and return
– 16% of Ineligible youth left and returned
– 8% of Read 180 youth left and returned
– 7.5% Traditional youth left and returned

Inter School movement
– 24% Ineligibles moved across schools at least once
– 27% of Read 180 and Traditional youth moved across
schools at least once
– Ineligible youth moved as many as 5 times and Read
180 and Traditional youth as many as three times

At any point in time there are approximately
1400 - 1700 youth across all DYS schools
Targeted Intervention Logic Model
Program
Inputs/Activities
Classroom Practices:
Intermediate Outcomes
Leadership Training
provided for Principals,
Literacy Coaches, and
District Staff [Initial and
follow-up; 5 hours total
in Y1]
A Maximum of 15 students
are scheduled for 2-45
minute periods consisting of
the 5 components of
Read180: whole group,
individualized learning,
computer activities, small
group (each 20 min), wrap
up (10 min)
R180 Training for
Teachers/Teacher
Aides [initial 2 days
and Semi-annual
follow-up; 15 hours
total in Y1. Two 5 hour
PD sessions (10 hrs
total) in Y2]
Scholastic Software for
classroom including
supplemental
materials. [including
Classroom
supplies/materials]
Targeted Strategies Used
include:
-Sequencing
-Identifying main idea/detail
-Summarizing
-Identifying cause/effect
-Making Inferences
Student Outcomes
Short-Term
Long Term
Use of strategies
to increase
comprehension/
vocabulary
Positive impact
upon return to
home/community
Increased
literacy, fluency,
comprehension
and confidence.
Reading at Grade
Level
Increased
engagement in
educational tasks.
Reduce
recidivism
Scholastic R180 Software
Adapts instruction for
Students needs
Students are assessed using
R-skills and Scholastic
Reading Inventory
Literacy Coach on classroom
practices
Increased student
self-efficacy
Implementation Data Collection
 Daily
implementation logs
 Weekly observation data (OSU)
 Quarterly feedback from Scholastic
observations
 Professional development attendance
– Literacy coaches
– Read 180 teachers/aides
– All teachers
– Principals
Targeted Intervention
Implementation Results
Facility
PD Attendance
Year 1
PD Attendance
Year 2
Instruction
Year 1
Instruction
Year 2
1
Moderate
High
Needs Improv.
Moderate
2
Moderate
High
Moderate
Needs Improv.
3
High
High
Moderate
High
4
High
High
High
High
5
High
High
Moderate
Moderate
7
High
High
Moderate
Moderate
8
High
High
Moderate
High
Total
High
High
Moderate
High
Targeted Intervention
Implementation Results
 OSU
Observation Results
– Read 180 implemented on Model
– Variation across schools
– Youth interested and engaged
– Differentiated instruction observed
– Youth disrupted class on 60% of early
visits, but youth not removed from
classroom
– When either aide or teacher not in
classroom, instruction very difficult
Targeted Intervention
Implementation Results
 Scholastic
observations
– Reported as on model/protocol
– Better able to meet 90 minute model in
year two compared with year one
– Encouraged use of more strategies to
help keep youth engaged (year two)
– Encouraged use of SAM reports to
better inform instruction (year two)
Impact Design
Targeted Intervention

Random assignment of student to
condition (100% reading below grade
level, 6 month stay, no GED/diploma)
– Condition 1 Read 180
– Condition 2 Traditional English class

Project start-up baseline testing
– CAT Reading and Math
– SRI Reading Lexile scores

End of each term assessments on SRI (4
per year)
Targeted Intervention Sample:
Classrooms/Teachers

8-12 students per classroom across
facilities and quarters
One-on-one interaction was greater in
Read180 relative to Traditional classes
 One Read 180 teacher and aide per
building/classroom
 All teachers certified in English/Language
Arts

100
Targeted Intervention Sample:
Students
95.2 96.5
90
80
70.168.3
70
60
55
57.8
50
45
42.2
40
30
24
26.1
20
10
3.1 3.7
2.8 1.7
0 0.2
4.8 3.5
Race
Gender
R180
Traditional
Yes
No
Female
Male
Multiracial
White
Hispanic
Black
Asian
0
Special
Education
Targeted Intervention Sample:
Students
100
100
90
90
80
80
70
70
60
60
50
50
43.7 42.2
40
45.8
41.7
40
22.5
20
7.3
0.5 0.4 1.1 1.8
4.6 5.3
Age *
Grade
R180
Traditional
graduated
12
11
10
9
8
22
0
Vi
TB
I
SL
D
2.3 1.1 1
0.4
0 (Pre-enrolled)**
Traditional
SL
OI
n
OMa
j
MD
ED
Df
(M
R)
Au
CD
OMi
R180
9.3
21
0.3 0.4 0.3 0
5
20
0.3 0 0.3 0 1 0
2.8
1 1.3
14
0
1 0.8
0.3 0
9.4
10
19
0.3 0.4
5 4.5
14.615.1
12.6
18
10
13.114.8
15
20
16
17.2
28.2
24.2
25.9
23.1 23.8 22.3
21
18.9
30
17
30.1 29.1
30
Targeted Intervention Analyses

Treatment of the Treated (TTT)
– This study defines TTT based on being present
in class for at least 5 weeks in each of 2 or
more quarters
– Estimated 5 HLM analyses using the number of
quarters of actual treatment received
– The base model (grand mean centered)
included the following variables (white, age,
Math CAT, Read CAT, disability, grade level,
institution and mobility)
– The dependent variable was the SRI score
– Variables with p values of .08 or less were
retained in each final model.
TTT Descriptive Results
Time Plot of the Mean Responses for the READ 180 Group, the Comparison
Group, and the Overall including All Youth
840
Comparison
READ 180
820
SRI Score

Overall
800
780
760
740
720
0
1
2
3
4
5
Number of Terms
6
7
8
TTT Descriptive Results
Time Plot of the Mean Responses for the READ 180 Group, the Comparison Group,
and the Overall for Youth with Less Than Two Quarters of Treatment
Comparison
900
READ 180
850
Overall
SRI Score
800
750
700
650
600
550
500
0
1
2
3
4
5
Number of Terms
6
7
8
TTT Descriptive Results
Time Plot of the Mean Responses for the READ 180 Group, the Comparison Group,
and the Overall for Youth with At Least Two Quarters of Treatment
840
SRI Score
820
800
780
760
Comparison
READ 180
740
Overall
720
0
1
2
3
4
5
6
7
8
Number of Terms
Overall
Read 180
Comparison
n
col %
n
col %
n
col %
Slope > 0
405
62.40%
248
72.09%
157
51.48%
Slope  0
244
37.60%
96
27.91%
148
48.52%
Total
649
100%
344
100%
305
100%
1000
1000
0
500
0
2
4
# of Terms
6
0
8
500
0
2
4
# of Terms
6
0
8
0
2
4
# of Terms
6
8
0
2
4
# of Terms
6
8
0
2
4
# of Terms
6
8
1000
1000
1000
0
500
0
2
4
# of Terms
6
0
8
500
0
2
4
# of Terms
6
0
8
1000
1000
1000
500
0
SRI Score
1500
SRI Score
1500
SRI Score
1500
500
0
2
4
# of Terms
6
8
Traditional
500
SRI Score
1500
SRI Score
1500
SRI Score
1500
Read 180
500
SRI Score
1500
SRI Score
SRI Score
1000
1500
Overall
TTT Descriptive Results
1500
0
500
0
2
4
# of Terms
6
8
0
TTT HLM Results

Youth (n = 542) who received between two and eight
quarters of treatment.
Level 1: yij  i  j i   ij , for i  1,2,...,n and j  0,1,2,...Ni
Level 2:  i   0  1 (AGEi  AGE.)   2 (MAT HCATi  MAT HCAT.)
  3 (READCATi  READCAT.)   4 (DISBi )   5 (GRDLVL i  GRDLVL.)  b0i
i  1 (READCATi  READCAT.)  2 (TRTGRPi )  b1i
Age is continuous; grand mean centered
Math CAT is continuous; grand mean centered
Read CAT is continuous; grand mean centered
Disability Status is dichotomized (0= not disabled;
1 = disabled)
Grade level is continuous and ranges from 8-12;
grand mean centered
TTT HLM Results
Estimates for the Fixed & Random Effects Using 2- 8 Quarters of Treatment
Fixed Effect
Estimate
SE
t-ratio
p-value
Effect Size
Intercept
α0
782.9
10.253
76.36
<.0001
--
Age
α1
15.9294
5.4662
2.91
0.0037
0.02
MathCAT
α2
10.4414
3.8851
2.69
0.0074
0.01
ReadCAT
α3
37.6965
3.7074
10.17
<.0001
0.18
Disability
α4
-28.3319
14.8666
-1.91
0.0572
0.01
Grade Level
α5
17.3026
7.2463
2.39
0.0173
0.01
ReadCAT*Time
β1
3.458
1.1505
3.01
0.0028
0.02
TRTGroup*Time
β2
22.342
3.4899
6.4
<.0001
0.12
Random Effect
b0
b0
18440*
b1
-846.83

b1
2181.18*
18820*
Regression Adjusted Means
2006-2007 (Year 1)
2 Quarters or more
3 Quarters or more
READ
180
Comparison
READ
180
Comparison
Baseline
Mean
784.34
773.27
783.88
773.36
Estimated
Post-test
Mean
856.95
827.95
879.99
820.12
Difference
score
+72.61
+54.68
+96.11
+46.76
a
Regression Adjusted Means
2006-2008 (Year 1 & 2)
2 Quarters or
more
3 Quarters or
more
READ Compar READ
180
ison
180
4 Quarters or
more
Comparis READ Comparis
on
180
on
766.98 753.64
764.71
757.11
756.17
759.64
Estimated 813.02 768.34
Post-test
Mean
823.28
766.79
842.02
757.46
Difference +46.04
score
+58.57
+9.68
+85.85
-2.18
Baseline
Mean
+14.7
Summary of Issues
Staff turnover
 Release dates
 Time in treatment
 Student attendance
 Student mortality
 Local building level perturbations
 Student cooperation
 Variability of SRI measure
 Psychometric properties of SRI
 Use of CAT as an additional outcome
measure

Summary of Methodological Issues
Tracking student entry, movement, exit
and return
 Power
 SRI measure quality/variability
 Missing data
 Losing data
 Maintaining student confidentiality
 Reporting issues
 Use of data

Summary of Issues Cont’d
 Student
disruptive behavior in class
 Student’s placed in isolation
 Student’s removed from class
 Student receipt of treatment
 Student assessment and cooperation
 Teacher cooperation
 Student release date
Summary of Findings







Read 180 being implemented on model and according
to protocol
Randomized control trials able to verify causal
relationship
Approximately 72% of Read 180 students
demonstrate a positive slope of increasing reading
scores over time, compared with about 51% for the
traditional group
Read 180 adds about 22 Lexile points per quarter of
treatment over and above the traditional group
performance
Students in Read 180 significantly out perform
students assigned to the traditional group
Effect sizes vary according to model, but most are
trivial
Most Read 180 students gain, but are still not reading
at grade level
Future Research
Student perception of Read 180 content
and structure
 Impact analysis with CAT as an outcome
measure.
 Sub-analysis determining the influence of
disability status on program intervention
impacts.
 Analysis of the influence of program on
recidivism.
 Sub-analysis of reason for incarceration,
reading improvement, and recidivism.
