Transcript From Research to Practice: An Analysis of the Interaction

Forty Years of Public School Reform:
What the Data Tell Us
Randy Keyworth
Jack States
The Wing Institute
Is Kansas Flatter Than a Pancake?
A value of 1.0 = perfect flatness
Pancake = .957
Kansas = .999
Texas State University
Arizona State University
National Assessment of Educational Progress (NAEP)
Only nationally representative and continuing assessment of what
America’s students know and can do in: math, reading, science,
writing, the arts, civics, economics, geography, and U.S. History.
Common metric for all states
Stays the same each year with carefully documented changes
Administered by National Center for Education Statistics (NCES) within the
Institute of Education Sciences within the U.S. Department of Education
Provides a clear picture of student academic progress over time
Tests are continually scrutinized for reliability and validity by panels of
technical experts within NCES and by external groups.
Is Kansas Flatter Than a Pancake?
A value of 1.0 = perfect flatness
Pancake = .957
Kansas = .999
40 years NAEP reading data = .996
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National
Assessment of Educational Progress (NAEP), 1992, 1994, 1998, 2000, 2002, 2003, 2005, and 2007 Reading Assessments.
Grade
12
Grade 8
Grade 4
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National
Assessment of Educational Progress (NAEP), 1992, 1994, 1998, 2000, 2002, 2003, 2005, and 2007 Reading Assessments.
Is Kansas Flatter Than a Pancake?
A value of 1.0 = perfect flatness
Pancake = .957
Kansas = .999
40 years NAEP reading data = .996
5 years IES graduation rate data = .992
IES, National Center for Education Statistics
Today’s Question:
What’s going on????
CAPSES “Street Cred”
1978 - 2004
Operated a large nonpublic school in SF Bay Area
six campuses
adult programs
residential programs
employment supportive
public school consultation
teacher training campus
services
CAPSES Board Member
President 1993-95
GAC Committee
Regional Coordinator
local SELPA liaison
The Wing Institute
2004 - present
independent, non-profit operating foundation
promote evidence-based education policies and practices
act as a catalyst to facilitate communication, cooperation
and collaboration between individuals and organizations
currently engaged in evidence based education
The Wing Institute’s Strategic Vision
Identify exemplars in evidence-based education
Develop networks to facilitate collaboration
Provide support for new ideas, research, and
publications
Facilitate cross-discipline cooperation
www.winginstitute.org
Today’s Discussion
What does performance monitoring data tell us
about the effectiveness of 40 years of school
reform?
What does education research data tell us about the
effectiveness of 40 years of school reform?
REACHING AMERICA'S HEALTH POTENTIAL: A STATE-BY-STATE LOOK AT ADULT HEALTH Commission to Build a
Healthier America May 2009 U.S. Census Data: American Community Survey (2007) U.S. Census Data: Behavioral Risk
Factor Surveillance System Survey Data (2005-2007)
REACHING AMERICA'S HEALTH POTENTIAL: A STATE-BY-STATE LOOK AT ADULT HEALTH Commission to Build a
Healthier America May 2009 U.S. Census Data: American Community Survey (2007) U.S. Census Data: Behavioral Risk
Factor Surveillance System Survey Data (2005-2007)
Obesity in Adults by Education Level
30%
25%
20%
Not a High School Graduate
High School Graduate
15%
Some College or Associate Degree
Bachelor's Degree or Higher
10%
5%
0%
1991
SOURCE:
Department of Health and Human Services (2003)
2001
Past Month Cigarette Use Among Persons 18 or Older
(2002)
40%
35%
30%
25%
20%
15%
10%
5%
0%
Not a High School Graduate
High School Graduate
SOURCE:
Department of Health and Human Services (2003)
Some College or Associate
Degree
Bachelor's Degree or Higher
Median Earnings by Level of Education (2003)
Professional Degree
Doctoral Degree
Masters Degree
Bachelors Degree
Associate Degree
Some College, No Degree
High School Graduate
Not a High School Graduate
$-
U.S. Census Bureau, 2004
$10,000
$20,000
$30,000
$40,000
$50,000
$60,000
$70,000
$80,000
$90,000 $100,000
University of Maryland, Department of Sociology
Source: U.S. Department of Justice (2003)
Education and Delinquency
The odds of delinquency, given low academic
performance, were about 2.1 times higher than those
given high academic performance.
Intervention Studies showed that improvements in
academic performance co-occurred or followed
improvements in the prevalence of delinquency.
Maguin & Loeber, 1996
SOURCE: U.S. Department of Commerce, Census Bureau, Current Population Survey (CPS), October 1967 through October 2007.
•
IES, National Center for Education Statistics
SOURCE: U.S. Department of Commerce, Census Bureau, Current Population Survey (CPS), October 1967 through October 2007.
Education is the civil rights issue of
our generation.
Arne Duncan
Secretary of Education
2009
PERFORMANCE FEEDBACK
Performance feedback is one of the most important tools we
have to assess and correct performance at all levels of the
education system:
Student performance
Staff performance
Systems performance
STUDENT PERFORMANCE FEEDBACK
No Child Left Behind:
Adequate Yearly Progress (AYP)
graduation rates
state test scores in reading and math
States can establish standards and goals
Graduation Rates (Reported and Cohort)
North Carolina
New Mexico
South Carolina
California
127,292 high school students dropped out....70% were black or hispanic
US
1,252,396 high school students dropped out....53% were black, hispanic, or native american
0%
10%
20%
30%
State Reported Grad Rates 2002-03
The Education Trust (2005)
40%
50%
60%
70%
80%
Cumulative Promotion Index Grad Rates 2000-2001
90%
100%
Comparison of State AYP Requirements
THOMAS B. FORDHAM I NSTITUTE (2008)
STUDENT PERFORMANCE FEEDBACK
Department of Education Initiatives
Adopting common standards and assessments
Building longitudinal data systems that measure
student growth and success to support instruction
STUDENT PERFORMANCE FEEDBACK
Standardized Graduation Rate calculation
DOE amended regulations require states to adopt the
“cohort model” by 2010-11:
four-year adjusted cohort graduation rate
disaggregated by subgroups
STUDENT PERFORMANCE FEEDBACK
Standardized Tests
Nation-wide standardized tests
43 States (so far)
2014 - 2015
Race to the Top
Build robust data systems to aggregate and disaggregate
student outcome data
link teachers to students outcomes -- to distinguish
between effective and ineffective teachers
to link effective teachers to their colleges of education
– to distinguish between effective and ineffective
programs
STAFF PERFORMANCE FEEDBACK
1. Performance measures must be reliable and valid.
2. Staff should know the variables being measured
3. Skills must be identified to attain these measures
4. Staff feedback should be frequent and specific
5. Feedback should trigger and guide effecting coaching.
18
Evaluation Requirements of Tenured Teachers
in the 50 Largest U.S. School System
17
16
14
12
11
10
8
6
7
6
6
4
3
2
0
Once a year Once every two
years
Once every
three years
Once every five
years
Unclear
Hiring, Assignment, and Transfer in Chicago Public Schools Report from The New Teacher Project July 2007
Not Stated
Hiring, Assignment, and Transfer in Chicago Public Schools Report from The New Teacher Project July 2007
93% of teachers
received superior or
excellent ratings
Only 3 out of 1,000
teachers rated
unsatisfactory
Hiring, Assignment, and Transfer in Chicago Public Schools Report from The New Teacher Project July 2007
87 Schools met criteria for being identified as
“failing schools”
69 (79%) of these schools did not issue a single
“unsatisfactory rating”
Hiring, Assignment, and Transfer in Chicago Public Schools Report from The New
Teacher Project July 2007
STAFF PERFORMANCE FEEDBACK
TRADITIONAL:
No
Performance measures must be reliable and valid.
No
Staff should know the variables being measured.
No
Skills must be identified to attain these measures.
No
Staff feedback should be frequent and specific.
No
Feedback should trigger and guide effecting coaching.
STAFF PERFORMANCE FEEDBACK
LAUSD
Yes*
Performance measures must be reliable and valid.
No
Staff should know the variables being measured.
No
Skills must be identified to attain these measures.
No
Staff feedback should be frequent and specific.
No
Feedback should trigger and guide effecting coaching.
SYSTEM PERFORMANCE FEEDBACK
Most systemic school reform strategies have focused on
“structural fixes”
Structural Fixes….Commonly Held Beliefs
SCHOOL RESOURCES
increase funding for students
higher pay for teachers
TEACHER QUALITY
more “Highly Qualified Teachers” (NCLB)
more teachers with credentials
more teachers with advanced degrees
more teachers with professional certifications (NBPTS)
comprehensive induction for new teachers
SCHOOL MODELS
smaller class size
school choice
charter schools
SCHOOL REFORM INITIATIVES
A Nation at Risk
Goals 2000
No Child Left Behind
Structural Fix: Increased Funding for Education
SOURCE: U.S. Department of Education, National Center for Education Statistics. (2009). Digest of Education Statistics, 2008 (NCES
2009-020), Chapter 2 and Table 179.
Structural Fix: Increased Pay for Teachers
IES National Center for Education Statistics Digest of Educational Statistics: 2008
Structural Fix: More Highly Qualified Teachers
Structural Fix: More “Credentialed” Teachers
Structural Fix: Advanced Degrees for Teachers
Master’s Degrees
Bachelor’s Degrees
SOURCE: National Education Association, Status of the American Public School Teacher, 2000-01. (This table was
prepared August 2003.)
Structural Fix: More Board Certified Teachers
Structural Fix: More Induction Support for Teachers
Percentage of Teachers Participating in
Induction Programs in Public and Private Schools
100
90
83%
80
70
60
Public
59%
60%
51%
50
40
30
Private
25%
24%
20
10
1990-1991
1993-1994
1999-2000
0
Smith & Ingersoll, 2004
Structural Fix: Class Size Reduction
32 States now have class size reduction programs in law:
Texas (1983)
Tennessee (1989)
Minnesota (1990)
California (1996)
New York (1997)
Florida (2002)
Structural Fix: Class Size Reduction
Structural Fix: School Choice
TRENDS IN THE USE of SCHOOL CHOICE: 1993 To 2007, IES
Structural Fix: More Charter Schools
5.0%
Charter School Growth As Percent of Total Schools in the
U.S.
Percent of Total Schools
4.0%
3.0%
2.0%
There are currently over 5,000 Charter
schools in the United States
1.0%
0.0%
1999-00 2000-01 2001-02 2002-03 2003-04 2004-05 2005-06 2006-07 2007-08 2008-09
U.S. Department of Education, National Center for Education Statistics
A History of Structural Fixes
SCHOOL RESOURCES
✔increase funding for students
✔higher pay for teachers
TEACHER QUALITY
✔more “Highly Qualified Teachers” (NCLB)
✔more teachers with credentials
✔more teachers with advanced degrees
✔more teachers with professional certifications (NBPTS)
✔comprehensive induction for new teachers
SCHOOL MODELS
✔smaller class size
✔school choice
✔charter schools
SCHOOL REFORM INITIATIVES
✔A Nation at Risk
✔Goals 2000
✔No Child Left Behind
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National
Assessment of Educational Progress (NAEP), 1992, 1994, 1998, 2000, 2002, 2003, 2005, and 2007 Reading Assessments.
National Assessment of Educational Progress (NAEP)
Basic denotes partial mastery of prerequisite knowledge and skills that are
fundamental for proficient work at each grade.
Proficient represents solid academic performance. Students reaching this
level have demonstrated competency over challenging subject matter.
Advanced represents superior performance.
Grade 4
Basic = 208
Proficient = 238
Advanced = 268
Grade 8
Basic = 243
Proficient = 281
Advanced = 323
Grade 12
Basic = 265
Proficient = 302
Advanced = 346
Trend in Fourth Grade NAEP Reading
Achievement-level Performance
1.2 million students below basic
2.4 million students below proficient
45%
% of Fourth Grade Students
40%
35%
30%
25%
20%
15%
10%
5%
0%
92
94
At or above Advanced
98
98
00
At or above Proficient
02
03
05
At or above Basic
07
09
Below Basic
347,900
Students
Structural Fix: Class Size Reduction
CSR Research Consortium
Legislatively mandated four year comprehensive study.
Major CA research orgs: AIR, RAND, PACE, WestEd, EdSource
Conclusions (Final Report):
1. The relationship of CSR to student achievement was
inconclusive…attribution to any gains in scores to CSR is
unwarranted.
2. Students in CSR received more individual attention,
similar instruction and curriculum.
3.
CSR remains very popular with parents and teachers.
but
What’s going on????
PERFORMANCE FEEDBACK
we have had lousy performance feedback systems
student performance
teacher performance
system performance
we have have been focusing on the wrong things
structural fixes haven’t worked
enormous time, energy and money has been wasted
there is hope on the horizon
better student longitudinal data systems
standardized formulas for graduation rate data
uniform standardized tests
better tools for evaluating teacher performance
THE END
The Education Culture
Education has been a non-evidence-based culture:
Teaching is an “art”, not a “science”
Constructivism, anti-science, anti-measurement
Ruled by ideology, preferences, hunches, fads
The teaching profession has been stuck in a “guild” mentality
Teacher tenure
Resistance to performance appraisal
Most school reform strategies are not based on scientific evidence
Lack of fundamental research into teaching, curriculum, systems
Special interests drive initiatives
The culture drives simplistic, commonly held “beliefs”
Lack of performance feedback
What is the relationship between
teacher experience and effectiveness?
Value Added Returns with Years Experience
1.070
Standard Deviations from Beginning Teachers
1.060
1.050
1.040
1.030
1.020
1.010
1.000
0.990
0.980
0.970
0.960
Beginning
1st year
2nd year
3rd year
Math
4th year
5th year
6th year
7+ years
Read
Kane, Rockoff, Staiger, 2006
What is the relationship between
teacher experience and effectiveness?
Value Added Returns with Years Experience
2.000
Standard Deviations from Beginning Teachers
1.800
1.600
1.400
1.200
1.000
0.800
0.600
0.400
0.200
0.000
Beginning
1st year
2nd year
Math
3rd year
Read
4th year
5th year
Log. (Read)
6th year
7+ years
STUDENT PERFORMANCE FEEDBACK
Sample of Graduation Rate calculations:
counting only seniors for graduation rates
(not freshman entering high school)
percentage of kids who got their diplomas in four
years or less
number of graduates compared to 12th grade
enrollment on the last day of school
Structural Fix: Major School Reform Initiatives
A Nation at Risk (1984)
Goals 2000 (1994)
No Child Left Behind (2001)
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National
Assessment of Educational Progress (NAEP), 1978, 1982, 1986, 1990, 1992, 1994, 1996, 1999, 2004, and 2008 Long-Term Trend
Mathematics Assessments.
What’s Going On????
TOOLS:
Roadmap: a model for identifying critical
components of evidence-based education
Data-mining: a mechanism for identifying,
analyzing,
and disseminating evidence
Evidence-based Education Roadmap
Research
Replicability Sustainability
Practice
Keys to Evidence-Based Education
RESEARCH
selecting interventions
PERFORMANCE MONITORING
evaluating interventions
The Wing Institute Services
• Publications
• Research
• Public Policy Analyses
• Information Clearinghouse
• The Evidence-based Education Knowledge Network
• Professional Forums