Transcript Exploring Data Use & School Performance in an Urban School

Exploring Data Use &
School Performance
in an Urban School District
Kyo Yamashiro, Joan L. Herman, &
Kilchan Choi
UCLA Graduate School of Education & Information Studies
National Center for Research on Evaluation,
Standards, and Student Testing (CRESST)
CRESST Conference
UCLA
September 8, 2005
Context & Background
• Large urban school district in the
Pacific Northwest
• Value-added Assessment System
implemented in District
• Need for more info on schools’ use of
data (VA and other)
Data Use &
Evidence-based Practice



Data use at the heart of test-based
reforms (NCLB) & continuous
improvement efforts
Little evidence of effects of data use
on performance
Some evidence shows limited access
and capacity of schools to use data
Study Components
CRESST conducts multi-year, multifaceted study of data use:
• Transformation Plan Review - content
analysis of school improvement plans
• Interviews, surveys, and observations
from site visits of case study schools
• Analysis of district achievement and
survey data
• Observations of school presentations
about progress
Sampling
• Latent variable, multilevel analyses used
to estimate gains (student-level,
longitudinal ITBS data in reading & math)
• Gains based on growth from 3rd to 5th
grade for 2 cohorts in each school:
•
3rd graders in 1998
•
3rd graders in 2001
• Within each cohort, 3 performance
subgroups (average, low, high)
Sampling (cont’d)
13 Schools met the following criteria:
•
Greater than district average % of low-SES
students
•
Starting point below district average
“Beat the Odds” Sample (7):
•
Higher than average gains
•
Relatively more consistent across:
• 2 cohorts (98 & 01)
• reading and math
• performance subgroups (hi, avg, lo)
Sample
Extremely diverse set of 13 small,
elementary schools
• African American student populations
between 11 - 81%
• Asian American student populations
between 2 - 59%
• White student populations between 559%
• Enrollment range: 134 to 533
Transformation Plan Review
TP Review Rubric (Rating of 1 to 3)
• Types of evidence or indicators used
•
Breadth; depth; VA data; technical sophistication
• Identification of goals/objectives or needs
analysis
• Identification of solution strategies
•
Specificity; based on theory/ research/data
• Analysis of progress
• Inclusion of stakeholders
Case Study Site Visits
2-day visits to 4 case study sites:
• Interviews/focus groups:
• Principal
• Building Leadership Team (BLT)
• Teachers (primary, upper)
• Teacher Survey
Additional
Achievement Analyses
Latent Variable Multiple Cohort (LMC)
Design (with SEMs)
• Estimating gains on ITBS based on data across 5
cohorts (1998 to 2002)
• Gains for performance subgroups:
•
Average (students starting at school mean initial status)
•
High (students starting at 15 points above school’s average)
•
Low (students starting at 15 points below school’s average)
• Patterns of growth differ from 2-cohort analysis
Results:
Achievement
Differences between Pre- and PostTransformation Plan Reform
• High/Avg: 4 schools - consistent
growth across rdg & math & subgroups
• Low: 6 schools - left some subgroups
behind in math and/or rdg
• Very Low: 3 schools - no growth or
negative gains
Results: Data Use
• Data Use Is Improving but Still Varied
•
Over 3 years, schools increased use of assessment
results and other evidence
•
Schools increased mention of VA data
• Data Review Process is Inclusive When
Capacity Exists
•
Principal often conduit (filter, interpret)
•
However, many schools developed collaborative
processes for data review
• Transf Planning Process May become More
Centralized (Less Inclusive) in Later Years
Results: Data Use (cont’d)
• Accessible and Excessive Data
•
Teachers use data for schoolwide reform and (to lesser
degree) instructional planning
•
Teachers are overwhelmed with amount of data
• More Capacity Needed
•
Whether schools integrate data into instructional decisions
tended to be person- or climate-driven
•
Principals need help, too
• More Diagnostic, Instructionally Sensitive Data
Needed
•
•
State testing data not seen as useful, valid, timely, or
interpretable
•
lack of continuity in tests (from grade to grade)
•
lack of diagnostic info (item analyses)
•
lack of individual growth info (pre-post)
District assessments seen as more helpful to instruction
Results:
Data Use & Achievement
Pre-Post Gains & Data Use Practices
Pre-Post Differences
Data Use Practices
Truman
High Growth
High
Polk
High Growth
Low
Wilson
Average Growth
High
Hoover
Average Growth
High
Jefferson
Low Growth
Medium
Tyler
Low Growth
Low
Van Buren
Low growth
High
Carter
Low growth
High
Harding
Low growth
Medium
Fillmore
Low growth
Low
Kennedy
Very Low growth
Low
Lincoln
Very Low Growth
Medium
Pierce
Very Low Growth
Low
Results: Data Use &
Achievement (cont’d)
• Ratings overlap for 7 of 13 schools
• For the most discrepant case
(Polk):
• showing high gains but low data use
• school in chaos, with new leadership
• For remaining 5 moderate
discrepancies, no case study data
Conclusions
• Less use of data for instructional planning probably
a function of:
•
type of data provided
•
leadership & climate
•
capacity
• Principals and teacher leaders need more help in
interpreting and using data
• Data use and gains appear to have a moderate link
for struggling schools; more case study info needed
• Need for more research on how to use value-added
(gains) in an accountability setting