Transcript Title of Presentation - OSEP Project Directors' Conference

TQ Research & Policy Brief:
Challenges in Evaluating Special
Education Teachers and English
Language Learner Specialists
Lynn Holdheide and Dan Reschly, Ph.D.
Vanderbilt University
July 20, 2010
OSEP Project Director’s Conference
Copyright © 2010 National Comprehensive Center for Teacher Quality. All rights reserved.
About the National Comprehensive
Center for Teacher Quality
 The National Comprehensive Center for Teacher
Quality (TQ Center) is a federally funded
partnership whose mission is to help regional
comprehensive centers and states carry out the
teacher quality mandates of the Elementary and
Secondary Education Act (ESEA) as reauthorized by
the No Child Left Behind Act.
 Partners:
• Learning Point Associates
• Vanderbilt University
• ETS
2
www.tqsource.org
Today’s Goals
Seeks to build the capacity of participants to
• Articulate the challenges identified with
evaluating special education teachers through
value-added and other measures of teacher
evaluation.
• Actively participate in the creation or redesign of
teacher evaluation models that support the
development of strong, valid and reliable teacher
evaluation policies and practices that recognize
and promote the unique contribution of special
education teachers.
3
www.tqsource.org
The Purpose
Identify the specific challenges in evaluating
this population of teachers.
Determine the current status of state policy
and practice.
Identify promising evaluation practices and
instruments.
Provide guidance and policy recommendations
to districts and states.
4
www.tqsource.org
The Inquiry
 Review of
policy/literature
• Designed in collaboration
with Council for Exceptional
Children (CEC) and national
experts
 Survey inquiry
 Series of interviews
with state- and districtlevel practitioners and
researchers
• State and local survey
• Respondent pool: state and
local directors (identified within
CEC’s Council of Administrators
of special education listserve)
 Data collection period:
December 2009–April
2010
1,143
total respondents
5
www.tqsource.org
Modification of Evaluation Processes
for Special Educators
Percentage of Administrators
Who Report an Allowance in
Modification for Special Educators
Among the local administrators,
30%
25%
20%
15%
10%
5%
0%
81%
State
Local
reported that contractual
agreement prevented
modification in the evaluation
process.
Total
6
www.tqsource.org
Opinions Regarding Special
Education Teacher Evaluation
Strongly Agree or Agree
92%
32%
84%
7
www.tqsource.org
Evidence-Based Practices
Meeting the needs of “diverse” learners may
not attend to the following:
• Special skills (individualized education program
[IEP] facilitation, collaboration, secondary
transition, social and behavioral interventions,
compliance with legal mandates)
• Evidence-based instructional methods
(direct/explicit instruction, scientifically based
reading instruction, learning strategy
instruction)
8
www.tqsource.org
Multiple Measures
95+%
Use more than one measure.
Survey inquired about
current practice.
Respondents indicated
value-added models in
future evaluation efforts.
9
www.tqsource.org
Practical Example:
District of Columbia IMPACT
Special Education








Individual Teacher Value-Added Scores
Non-Value-Added Achievement
Teaching and Learning Framework
Commitment to the School
School Value-Added Scores
Core Professionalism
IEP Quality Plan
IEP Timeliness
10
10%
50%
5%
5%
15%
15%
www.tqsource.org
Opinions Regarding Use of Student
Achievement for Special Educators
Strongly Agree or Agree
60%
21%
73%
11
www.tqsource.org
Student Growth Measures
Practical Examples
 Austin Independent School District, Texas
• Student Learning Objectives
 One is targeted toward classroom performance.
 One is targeted toward particular skills or subgroups
of students.
 Norwell Public Schools, Massachusetts
• Progress on the IEP is factored into evaluation of
special educators.
 Both districts are heavily dependent on teacher
training and support.
12
www.tqsource.org
Observation Protocols
51%
Align to the
state’s professional
teaching standards.
85%
26% Didn’t know.
Use the same observation
instrument as that of general
education teachers.
“Our evaluation tool was
developed in the district over
40 years ago.”
12%
“Our current evaluation
system is outdated and
applied to nothing.”
Use a modified or different
observation instrument.
13
www.tqsource.org
Observation Protocol
Practical Example
Alabama Department of Education’s
Professional Education Personnel Evaluation
Program
• Slightly modified for the following:
 Specialty area systems (speech paths, library specialist)
 Teachers of students with significant cognitive disabilities
• Competencies added in certain areas (e.g.,
classroom is expanded to include community
settings, and academic content is expanded to
include functional life skills.)
14
www.tqsource.org
Expert Opinions Regarding
Evaluators
Strongly Agree or Agree
77%
61
%
15
60%
Require training for
evaluators.
12%
Require specialized
training.
www.tqsource.org
Practical Examples
 Toledo’s Peer Assistance and Review
• School-based teams evaluate.
 Norwell Public Schools, Massachusetts
• All teachers are evaluated using the same instrument.
• Two formative assessments are conducted:
 One with principal
 One with special education administrator
• Each evaluator focuses on expertise areas.
• Both work collaboratively to develop summative
evaluation.
16
www.tqsource.org
Opinions Regarding Attribution in
Coteaching Setting
Strongly Agree or Agree
85%
75%
13%
17
www.tqsource.org
Policy and Practice
Recommendations
 Include special education administrators when
revamping/designing evaluation frameworks.
 Identify a common framework that defines
effective teaching for all teachers, differentiating
for special educators as appropriate.
 Integrate evidence-based practices for students
with disabilities into evaluation models.
18
www.tqsource.org
Policy and Practice
Recommendations
 Improve data quality.
 In addition to─or, in some situations, in
the absence of─appropriate standardized
assessment data, incorporate other reliable
evidence of teachers’ contributions to student
learning into the teacher evaluation system,
such as progress toward accomplishing IEP
objectives and student learning objectives
across broad academic and behavioral domains.
19
www.tqsource.org
Policy and Practice
Recommendations
 Ensure that evaluator training includes explicit
training for evaluators of special educators
and/or consider establishing a model of peer-topeer observations or a model in which
evaluators are matched to specific disciplines.
 Collaborate with teacher preparation programs
to ensure that evidence-based practices are
incorporated into teacher preparation
coursework and professional development
activities.
20
www.tqsource.org
Evidence-based Practices in Special
Education and Practice
 Scientifically based instruction ESEA (2002) and IDEA
(2004)
 IES criteria and evolution to evidence-based practices
 Research supported evidence-based practices in
special education
• ABA and its many variations/application
• Direct instruction, big D and little d, reading and
mathematics
• Formative assessments with instructional decision making
• Learning strategies
21
www.tqsource.org
What Works? Research Foundations
From Meta-Analysis
Treatment
•
•
•
•
•
•
•
Effect Size
Applied Behavior Analysis
Formative Evaluation: Curriculum-Based
Measurement+Graphing+Decision
Rules+Reinforcement
Explicit Instruction and PS
Comprehension Strategies
Mathematics Interventions
Writing Interventions
Matching instruction to
learning styles?
Note: These effect sizes are stable across cultural groups.
22
+ 1.00
+ 1.00
+ .70 to 1.50
+ 1.00
+.60 to 1.10
+.50 to .85
0.00
Sources: Kavale, 2005
www.tqsource.org
Scientifically Based
Instruction in Reading
 Reading Curricula content (Snow, Burns, & Griffin, 1998)
•
•
•
•
•
Phonemic Awareness
Phonics
Fluency
Vocabulary
Comprehension
PLUS
• Direct, systematic instruction
• Universal screening and
formative evaluation
 Problem of teacher preparation
 Vanderbilt University/TQ Center innovation
configurations: reading, classroom behavior, inclusive
services, learning strategies (Reschly, Smartt, & Oliver,
2007)
23
www.tqsource.org
Learning to Read: The Great Debate
(Chall, 1967)
 Research review 1900–1965
 Early reading, K–3
 Code versus meaning
emphasis
 Phonics or whole word
 Code superior, especially
for struggling readers
 Lamented the generally poor
preparation of teachers to
teach reading
24
www.tqsource.org
National Council on Teacher Quality:
Reading Components Taught Well
Percentage
43%
N=31
11%
N=8
11%
7%
13%
N=9
N=8
N=5
Source: Walsh, Glaser,
and Wilcox, 2006
Number of Components
25
www.tqsource.org
NCES Reading Report Card 2009:
Categories
 < Basic: Less than partial mastery of
prerequisite knowledge and skills fundamental
to proficient work at the grade level
 Basic: Partial mastery of …
 Proficient: Solid academic performance and
demonstrated competency over challenging
subject matter
 Advanced: Superior performance
Source: National Center for Education Statistics, 2009
26
www.tqsource.org
Reading 2009 Grade 4
Percentage
Source: National Center for Education
Statistics, 2009, Table A-12
27
www.tqsource.org
Preparation of Special Education Teachers in
Scientifically Based Reading Instruction in 27
Institutions of Higher Education (IHEs)
28
www.tqsource.org
Reading Course Syllabi: Projects
Explain your philosophy of literacy.
Develop bulletin board to motivate children
to read.
Produce journal explaining personal
experience in learning to read.
Analyze the social justice implications of
literacy.
29
www.tqsource.org
TQ Research & Policy Brief
 Smartt, S. M., & Reschly,
D. J. (2007). Barriers to the
preparation of highly
qualified teachers in reading
(TQ Research & Policy Brief).
Washington, DC: National
Comprehensive Center for
Teacher Quality.
http://www.tqsource.org/pu
blications/June2007Brief.pdf
30
www.tqsource.org
Current Special Education Practice
With High-Incidence Disabilities
 Review of special education case files for the state
of Washington for a court case
 Randomly selected 900 special education students
• Ten districts were represented.
• All students were in special education for at least 12
months.
• Case files varied from 50 to 1,100 pages.
• The review evaluated individualized education programs
(IEPs) using checklist for required components and
evidence of formative evaluation.
• How many graphs?
31
www.tqsource.org
Review of Special Education
Case Files: Results
 Little evidence of systematic, direct instruction or
behavior interventions using problem solving
 Assessment and formative evaluation
nearly nonexistent (11 of 870 cases had
graphs.)
 Lots of test protocols documenting weaknesses
 Little objective evidence of positive outcomes
(i.e., benefits of special education are largely
undocumented in high incidence.)
 No assessment of progress
32
www.tqsource.org
National Mathematics Advisory
Panel (2008) Report Key Findings
 Conceptual understanding, computational and
procedural fluency, and problem solving skills
are equally important and mutually reinforce
each other.
 Students should develop immediate recall of
arithmetic facts to free the “working memory”
for solving more complex problems.
 Teachers’ regular use of formative
assessments can improve student learning in
mathematics.
33
www.tqsource.org
National Mathematics Advisory
Panel (2008) Report Key Findings
 Explicit instruction for students who
struggle with mathematics is effective in
increasing student learning.
 Teachers should understand how to provide
clear models for solving a problem type
using an array of examples, offer opportunities
for extensive practice, encourage students to
“think aloud,” and give specific feedback.
34
www.tqsource.org
Summary: Teacher Preparation
and Practice
 Insufficient use of evidence-based practices in
teacher preparation/comprehensive professional
development and practice
 TQ Center use of evidence-based innovation
configurations to address these issues (See the TQ
Center Special Education Resource List.)
 Improved implementation of evidence-based
principles leading to improved outcomes
 Major Challenge: Narrowing the gap between what is
known about evidence-based instruction and teacher
preparation and special education practice
35
www.tqsource.org
References
 Chall, J. S. (1967). Learning to read: The great debate. New York:
McGraw-Hill.
 Kavale, K. (2005). Effective intervention for students with SLD: The nature
of special education. Learning Disabilities, 13(4), 127–138.
 National Center for Education Statistics. (2009). The nation’s report card:
Reading 2009─National Assessment of Educational Progress at Grades 4
and 8 (NCES 2010-458). Washington, DC: Institute of Education Sciences,
U.S. Department of Education. Retrieved July 1, 2010, from
http://nces.ed.gov/nationsreportcard/pdf/main2009/2010458.pdf
 National Mathematics Advisory Panel. (2008). The final report of the
National Mathematics Advisory Panel. Washington, DC: U.S. Department of
Education. Retrieved July 1, 2010, from
http://www.ed.gov/about/bdscomm/list/mathpanel/report/final-report.pdf
36
www.tqsource.org
References
 Reschly, D. J., Smartt, S. M., & Oliver, R. M. (2007). Innovation
configurations to improve teacher preparation in reading, behavior
management, and inclusive practices. In C. A. Dwyer (Ed.), America’s
challenge: Effective teachers for at-risk schools and students (pp. 23–45).
Washington, DC: National Comprehensive Center for Teacher Quality.
 Smartt, S. M., & Reschly, D. J. (2007). Barriers to the preparation of highly
qualified teachers in reading (TQ Research & Policy Brief). Washington, DC:
National Comprehensive Center for Teacher Quality. Retrieved July 1, 2010,
from http://www.tqsource.org/publications/June2007Brief.pdf
 Snow, C. E., Burns, M. S., Griffin, P. (Eds.). (1998). Preventing reading
difficulties in young children. Washington, DC: National Academy Press.
 Walsh, K. Glaser, D., & Wilcox, D. D. (2006). What education schools aren’t
teaching about reading and what elementary teachers aren’t learning.
Washington, DC: National Council on Teacher Quality. Retrieved July 1,
2010, from http://www.nctq.org/p/docs/nctq_reading_study_app.pdf
37
www.tqsource.org
Lynn Holdheide, Vanderbilt University
P: 615-322-8150
E-Mail: [email protected]
Dan Reschly, Vanderbilt University
P: 615-322-8169
E-Mail: [email protected]
38
www.tqsource.org