Transcript Discussion of Identifying Effective Teachers Using

Discussion of
Identifying Effective Teachers Using
Performance on the Job
(Gordon, Kane & Steiger)
Henry Braun
Boston College
Teacher Quality Conference
Northwestern University
May 1 2008
GKS: A Modest Proposal
A plan to restructure the labor market and working
conditions for teachers: Making “performance” a more
significant factor in teacher advancement and
compensation – with value-added measures given
special prominence
•
•
•
•
•
Comprehensive
Timely
Scalable
Costed
Defended
What really matters?
… the success of U.S. public education depends
upon the skills of the 3.1 million teachers
managing classrooms in the elementary and
secondary schools around the country.
Everything else – educational standards, testing,
class size, greater accountability – is
background … Without the right people standing
in front of the classroom, school reform is a futile
exercise. “ (p.5)
An Ecological Perspective on Reform
• What are the highest priority education goals?
– e.g. Greater high school completion rates with students well-prepared
for life beyond high school
• What are the most critical requirements?
– e.g. A teaching force that is : (i) demonstrably effective, (ii) adequately
large, (iii) appropriately distributed
– e.g. Schools that are: (i) well led and collaborative, (ii) properly
resourced, (iii) learning organizations
• What is the context in which the proposed reforms will be
implemented?
– e.g. (i) a technically flawed accountability system that generates
perverse incentives, (ii) problematic union-management relationships,
(iii) concerns about over-reliance on test scores
• What are the warrants for “systems validity”?
– e.g. (i) alignment of ends and means, (ii) coherence of system
components, (iii) feasibility, (iv) designs to mitigate negative
consequences
Teaching: The Current Scene
• Initial entry based largely on credentials
• Tenure an early and semi-automatic decision
• Uniform salary schedules based on credentials and
years of experience
• Seniority rights
• High mobility rates (localized)
• High levels of out-of-field teaching (localized)
The Argument
• Significant variation among teachers in effectiveness
(as measured through value-added analysis)
• Existing barriers to entry are dysfunctional
• Tenure decisions are now generally information-poor
• Value-added measures of effectiveness are a credible
basis for decision-making
• Current distribution of teacher quality is sub-optimal
Recommendation 1: Reduce barriers to entry
• A well-documented need to increase access to the
teaching profession
• Current licensing standards are often “artificially” lowered
• How should we assess (pedagogical) content
knowledge? What standards?
• How would this work in the elementary grades?
• What about special education?
• What would be the function of the M. Ed.?
Reflections on Teaching
“ I came to understand that teaching is hard and that
being smart and well-educated doesn’t necessarily mean
one will be good at it.”
(Checker Finn, on his first year of teaching
social studies in Newton HS, Newton, MA)
“The evidence indicates that neither an extreme centralized bureaucratization nor a complete deregulation of
teacher requirements is a wise approach for improving
teacher quality.”
(Jennifer King Rice, 2008)
Recommendation 2: Modify tenure process
• A more rigorous evaluation for tenure is needed
• Outcome-based indicators should be considered
•
•
•
•
Better training for evaluators (e.g. Principals)
More equitable provision of support and mentoring
Lengthier probationary period
Salary consequences?
[The question we dare not ask: Why tenure?]
Recommendation 3: Introduce differential pay
• Inequitable distribution of experienced and more
effective teachers
• Use substantial bonuses to shift incentives
• Need a stable index for “ranking” hard-to-staff schools
that incorporates more than student disadvantage
• Avoid hard cut-offs
• “Bonus” teachers should bring school resource dollars to
jumpstart improved working conditions
• Must retain bonus-eligible teachers already in such
schools
• Countervailing pressure of NCLB sanctions
Recommendation 4: Change teacher
compensation
• Seven challenges are well worth considering!
• Improve credibility of all forms of evaluation
• Pay attention to a number of desired outcomes (i.e., aim
for balanced accountability)
• Don’t place excessive burden on test scores
• Idiosyncratic school-level conditions are problematic
– School leadership
– Out-of-field teaching
– Grade-shifting
Recommendation 5: Develop data systems
• Yes, yes, yes
[Molly Bloom (1922), Hillary Clinton (2008)]
• Federal government stepping up to the plate here!
• Need substantial investment (and commitment) to build
(and to maintain) data warehouses
• Many state education departments are under-staffed in
technology and data management
Using (Test-based) Performance
Indicators (I)
• Intuitively attractive and commonsensical
• Certain measures of performance contain useful
information
• If indicators are seen as (relatively) fair, then attaching
stakes to performance focuses attention and can be
constructive
• Provides a quantitative and comparable basis for
decision-making
Using (Test-based) Performance
Indicators (II)
• Cognitive outcomes are a subset of education goals
• Current (state) tests are proxies of variable quality for the
target cognitive outcomes
• Mis-alignment between proxies and targets can lead to
sub-optimal resource allocation (Murnane & Cohen,
George Baker, and others)
• Disagreement on effectiveness of extrinsic motivation
(Ed Deci, and others)
• Mixed findings on success of pay-for-performance
schemes (Podgursky, Harris, and others)
• Implications of “Campbell’s Law”
Value-added Modeling (I)
• VAMs attempt to isolate the contributions of teachers to
student learning as measured by test scores
• VAMs adjust current student performance for differences
in prior performance and certain background
characteristics (creating a “level playing field”)
• Superior to status models and growth models as a basis
for performance evaluation
• Many models are in use and have been studied
extensively with regard to their operating characteristics
and sensitivity to departures from assumptions
• Policy use of VAM results implicitly relies on a causal
inference
Value-added Models (II)
Fundamental technical problem in using VAMs
“In making causal attributions from observational data,
no statistical model, however complex, and no method of
analysis, however sophisticated, can fully compensate
for the lack of randomization.”
(Braun, 2005)
Practical Questions:
(i) How close can we come?
(ii) How do we know when we are close enough?
(iii) What are the limitations on use?
Value-added Models (III)
Other technical considerations in the use of VAMs
• Test quality (substantive)
• Test characteristics (psychometric)
• Balancing model complexity against test information
density
• Volatility
• Bias
How do we go beyond math and language arts?
Value-added Models (IV)
Operational and political considerations
• Administrative complexity
– VAM infrastructure
– Multiple review systems
– Training and auditing
• Fairness
– Transparency
– Level playing field
– NCLB/AYP
• Core opposition to use of student results
– Unions (e.g. New York state)
– Concern with over-reliance on tests
T’was Ever Thus
For more than a hundred years much complaint
has been made of the unmethodical way in
which schools are conducted, but it is only within
the last thirty that any serious attempt has been
made to find a remedy for this state of things.
And with what result? Schools remain exactly as
they were!
John Amos Comenius, 1632
(quoted by C. E. Silberman in
“Crisis in the Classroom”, 1970)
The Status Quo Is Not Viable
• Demographic trends and static literacy distributions
portend a downward shift in per capita human capital
(America’s Perfect Storm)
• Staggering opportunity costs in not increasing high
school graduation rates
(The Price We Pay)
• More than 50% of non-capital school spending goes to
teachers
(Understanding the Effectiveness of Teacher Attributes)
• Too many ships in the American convoy are taking on
water
(Any newspaper)
CONCLUSIONS
• This is a thoughtful proposal to address a serious national need,
grounded in the belief that data-based decision-making should play
a greater role in shaping the teaching profession.
• I agree
-- in principle!
• As always -- the devil is in the details!
• More attention is needed to:
–
–
–
–
Full design of the new system
Infrastructure
School realities
Consequential validity
Parting Thoughts
Teacher compensation is the elephant in the room.
We need a constructive debate around various
proposals to assist us in moving forward –
and rigorous evaluation to tell us what works and why!
Question yet to be answered:
Is the uniform salary schedule to teacher compensation
as democracy is to forms of government --- the worst
possible system, except for all the others?