Transcript Evaluation of South Carolina’s Reading First (SCRF) Initiative

Evaluation of South
Carolina’s Reading First
(SCRF) Initiative
Achievement and Survey Results
from the First Three Years
Presented at the American Educational Research
Association’s Annual Meeting
March 27, 2008
Tammiee S. Dickenson, PhD
Robert L. Johnson, PhD
Heather L. Bennett, MSW
Katie A. Sesso-Dahlke, MSW
Brett Ermer
Mugdha Galande
Joanna Gilmore
Jessalyn Smith
Diane M. Monrad, PhD
Sarah J. Gareau, MEd
Jennifer Gay
Diana Mindrila
Anita Rawls
Pam Wills
Patricia Branham
Becca Driggers
Diane Dunham, EdS
Overview

SCRF evaluation

Data sources

Achievement data

Participant group survey results

Conclusions
SCRF
Evaluation
SCDE
Planning
Data Collection
Data Analysis
Reporting
OPE
SCEPC
Participants

2004-2005: served 52 schools in 24 districts
(approximately 12,000 students)

2005-2006: served 51 schools in 23 districts
(approximately 11,000 students)

2006-2007: served 49 schools in 23 districts
(approximately 10,500 students)
Data Sources

Stanford Reading First assessment

Palmetto Achievement Challenge Test
(PACT)

Participant group surveys

School Leadership Team (SLT) surveys

Professional development workshop surveys
Annual SLT Presentation

Results presented from previous year

Stanford Reading First assessment results


Cohort analysis of all students in previous year

Matched analysis of students in all years

Individual school reports provided
Participant group survey results

Formative feedback for program improvement
Stanford Reading First
Assessment

Measures achievement in grades 1 through 3

Sections:

Multiple choice (components – phonemic
awareness; phonics; vocabulary development;
reading fluency; and reading comprehension
strategies)

Oral fluency (components – speaking vocabulary
and oral reading fluency)
Results: At Grade Level (AGL)
Grade 1
Percentage of SCRF Grade 1 Students Scoring At Grade
Level (AGL) on Stanford Reading First and Annual Gains
Fall
Spring
Annual Gain
Year 1
Year 2
17.0%
18.8%
48.2%
51.2%
31.2%
32.4%
Year 3
21.1%
54.3%
33.2%
Results: At Grade Level (AGL)
Grade 2
Percentage of SCRF Grade 2 Students Scoring At Grade
Level (AGL) on Stanford Reading First and Annual Gains
Fall
Spring
Annual Gain
Year 1
30.8%
35.5%
4.7%
Year 2
35.7%
40.4%
4.7%
Year 3
38.0%
44.6%
6.6%
Results: At Grade Level (AGL)
Grade 3
Percentage of SCRF Grade 3 Students Scoring At Grade
Level (AGL) on Stanford Reading First and Annual Gains
Fall
Spring
Annual Gain
Year 1
21.0%
41.5%
20.5%
Year 2
25.2%
46.2%
21.0%
Year 3
27.6%
49.0%
21.4%
Average NCE Scores on Stanford
Reading First for SCRF Students
with AGL Reference Line
At Grade Level
SCRF Students
60
NCE Score
50
40
30
20
10
0
Fall Gr1
SESAT2
Spr Gr1
Fall Gr2
Primary 1
Spr Gr2
Fall Gr3
Primary 2
Spr Gr3
Primary 3
Average Scale Scores on Stanford
Reading First for SCRF Students
and the Norm Group
Norm Group
SCRF Students
700
Scale score
650
600
550
500
450
400
Fall Gr1
SESAT2
Spr Gr1
Fall Gr2
Primary 1
Spr Gr2
Fall Gr3
Primary 2
Spr Gr3
Primary 3
Participant Group Surveys

Respondent groups: interventionists, literacy
coaches, principals, and teachers

Sections: implementation, level of support,
roles and responsibilities, professional
development, and program effectiveness

Comparison across years (2005-2006
compared to 2006-2007)
Results: Participant Group
Surveys




Better understanding of program goals
Reported a high rate of involvement in professional
development activities and found those activities
helpful
Decrease in the need for professional development
about the program and use of assessments
Increased need for professional development on
effective instructional strategies to use for students
below grade level
Implementation Survey Items
Principal
Literacy Coach
Interventionist
Teacher
I support the SCRF
Initiative.
I understand the goals of
the SCRF Initiative.
I understand my roles
and responsibilities
within the SCRF Initiative.
0%
20%
40%
60%
% Agree or Strongly Agree
80%
100%
Implementation Survey Items
(Cont.)
Principal
Literacy Coach
Interventionist
Teacher
SLT members regularly
share important information
w ith our faculty about the
SCRF Initiative.
Teachers, interventionists,
administrators, and the
school-based SCRF literacy
coach(es) are w orking
together to implement the
SCRF Initiative.
0%
20%
40%
60%
% Agree or Strongly Agree
80%
100%
Describe One Benefit of the
SCRF Initiative

Commonly cited benefits across groups
Focus on assessment



Increased awareness of students’ needs/strengths/weaknesses
Progress monitoring; Dominie
Improved use of assessments; use of assessment data to
guide instruction
Resources


Additional books and materials
Extra funding/money
Professional development; study groups
Increased collaboration
Learned new strategies
Results: Participant
Recommendations
Teacher
Interventionist
Literacy Coach
Principal
60.0%
Percentage
50.0%
40.0%
30.0%
20.0%
10.0%
0.0%
Changes to study groups
Changes related to
assessments
Recommendations
Increase writing emphasis
Results: Effectiveness Ratings
Teacher
Interventionist
Literacy coach
Principal
100.0%
83.1% of teachers rated SCRF as effective or very effective.
 88.1% of interventionists rated SCRF as effective or very effective.
 81.8% of literacy coaches rated SCRF as effective or very effective.
 90.9% of principals rated SCRF as effective or very effective.

Percentage
80.0%
60.0%
40.0%
20.0%
0.0%
Not Effective
Somewhat Effective
Effective
Effectiveness Rating
Very Effective
Conclusions: Achievement

Students performed better on the Stanford Reading
First assessment in the 2006-2007 school year as
compared with the 2005-2006 school year.

Students’ scores on the Stanford Reading First
assessment improved from fall to spring in all three
years for all grade levels.

The largest gains were made by students in grade 1
and the lowest gains occurred in grade 2 in 20062007, which is also consistent with the last two
years.
Conclusions: Participant
Groups

Program participants have a better
understanding of program goals.

Participant groups find professional
development useful.

Over 80% of each participant group rated the
initiative as either effective or very effective in
Years 2 and 3.
Recommendations



More time for sustained reading for students
in grade 2
Additional strategies for working with
students below grade level
Participant groups’ suggestions regarding
study groups
Contact Information
South Carolina Educational Policy Center
University of South Carolina, College of Education
Wardlaw Suite 010, Columbia, SC 29208
P: 803-777-8244
F: 803-777-0220
E: [email protected] (Diane Monrad)
Office of Program Evaluation
University of South Carolina, College of Education
Wardlaw Suite 023, Columbia, SC 29208
P: 803-777-3402
F: 803-777-8838
E: [email protected] (Tammiee Dickenson)