Helping Teachers Become More Effective While Measuring Teaching Effectiveness:

Combining Multiple Measures

AASA Webinar, 2011 Allan Odden Strategic Management of Human Capital (SMHC) University of Wisconsin-Madison

Overview

1. Prime challenge is to improve student performance 2. Key strategy to attain that goal (the focus of today): talent and human capital management 3. Support tactic for talent management – multiple measures of effectiveness used in new teacher evaluation systems 2

Human Capital Management

• Obama and Duncan administration has made improving teacher and principal talent and their effectiveness central to education reform • Goal: put an effective teacher into every classroom and an effective principal into every school • To implement these practices and manage teachers (and principals) around them, develop multiple measures of teacher effectiveness (long-hand for new teacher evaluation systems) • New NEA and AFT policies that allow use of student data in teacher evaluation – Extract such measures from teacher improvement systems • Scores of states and districts working on this issue • These issues also central to ESEA reauthorization 3

Core Elements of the Strategy

• Multiple Measures to Evaluate Teachers and Assess Teaching Effectiveness 1.

Measures of instructional practice – several systems 2.

Indicators of impact on student learning • Use of those measures: a) In new evaluation systems, for teachers and principals b) For tenure c) For distributing and placing effective teachers d) For dismissing ineffective teachers e) For compensating teachers 4

Current Teacher Evaluations Useless

• Find 99+% of teachers satisfactory, accomplished, or outstanding • Even when student performance is dismal • Rarely use specific teaching standards and scoring rubrics with trained assessors • Until recently, did not include evidence of impact on student learning • Neither valid nor reliable; cannot be used for consequential decisions for teachers • Viewed as “waste of time” by teachers &administrators 5

New Directions in Teacher Evaluations

• So now there is a major nationwide push to change teacher evaluation systems • Desire to use BOTH measures of instructional practice (qualitative) AND indicators of impact on student learning gains (quantitative) • • Widespread support for these new directions

The question is not whether teacher evaluation will change but how it will be changed

6

Teacher Evaluation

Two major pieces of the evaluation: 1.

Qualitative Measures of instructional practice – Danielson Framework, INTASC, Connecticut BEST system, CLASS, PACT, National Board, the new North Carolina system – see Milanowski, Heneman, Kimball,

Review of Teaching Performance Assessments for Use in Human Capital Management, 2009 at www.smhc cpre.org

and go to resources

2.

a.

b.

Quantitative Measures of impact on student learning: Primary model at the present time is value added using end of year state summative tests Additional proposal is to use interim-short cycle (every 4-6 weeks) assessment data, aligned to state content standards, that show student/classroom growth relative to a normed (national or state?) growth trajectory 7

Measuring Educator Effectiveness

Measuring Educator Effectiveness

Measuring Educator Effectiveness

Specifically, focus on

short-cycle

assessments

Combining Multiple Measures of Teaching Performance

• Standard Prescription: Instructional practice measure (e.g., teacher evaluation ratings) + Gain, growth, or value added based on state standards-based assessments • But: – Practice ratings and assessment gain, growth, or value cancel added don’t measure the same thing; measurement error sources are different and don’t – Gain, growth, or value-added on state assessments are of limited use for teacher development

Advantages of Adding Short-cycle Assessments to the Mix

1.

2.

For teacher development: – – Because such assessments are frequent, teachers get feedback that they can use to adjust instruction

before

the state test Teachers can see if student achievement is improving, and if assessments are linked to state proficiency levels, whether students are on track to proficiency – – For teacher accountability: More data points allow estimation of a growth curve The growth curve represents learning within a single school year; no summer to confuse attribution – The slope of the average growth curve or average difference between predicted end points provides another indicator of teaching effectiveness – Combining with growth, gain, or value-added based on state assessments provides multiple measures of productivity – If linked to state assessments, can predict school year proficiency growth

What Short Cycle Assessments Show

Issues in Combining Practice & Student Achievement Measures

• Models: Report Card, Compensatory, Conjoint • When Combining Need to Address: – Different Distributions, Scales and Reference Points – Weighting in Compensatory Models • Equal • Policy • Proportional to reliability

Report Card Model

Performance Domain

Instructional Practice

Performance Dimensions

Planning & Assessment Classroom Climate Instruction Professionalism Cooperation Attendance Development Student Growth, Gain, or VA on State Assessments Math Reading/ELA Other Tested Subjects Student Growth on Short Cycle Assessment Math Reading

Score Levels

1-4 1-4 1-4

Requirement for Being Considered Effective

Rating of 3 or higher on all dimensions 1-4 1-4 1-4 Rating of 3 or higher on all dimensions Deciles or Quintiles in state/district distribution for each subject Being in the 4 th Decile or 3 rd Quintile or Higher for All Tested Subjects Avg. Growth Curve Translated into Predicted State Test Scale Score Change Predicted Gain Over Year Sufficient to Bring Student from Middle of “Basic” Range to “Proficient” 15

16

Scales, Distributions, & Reference Points for Value-Added vs. Practice

Putting Practice Ratings and Student Achievement on the Same Scale

Emerging Practice: Rescale growth, gain or value-added measure to match the practice rating scale – Standardize and set cut-off points in units of standard error, standard deviation or percentiles Category Distinguished (4) Proficient (3) Basic (2) Unsatisfactory (1) In S.D. Units >1.5 S.D. Above Mean +/- 1.5 S.D. Around Mean 1.51 - 2 S.D. Below Mean > 2 S.D. Below Mean Percentiles 70 th + 30 th to 69 th 15 th to 29 th Below 15 th

Compensatory (Weighted Average) Model for Combining Performance Measures

Dimension

Growth, Gain, Value Added on State Test Growth as Measured by Short-Cycle Assessment Practice Evaluation Total 1.0-1.75 = Unsatisfactory, 1.76-2.75 = Basic, 2.76-3.75 = Proficient, 3.76 += Distinguished

Rating

2 3 4

Weight

25% 25% 50%

Product

0.50

0.75

2.00

3.25

18

19

Conjoint Model for Combining 2 Measures

1 Student Outcome Rating 2 3 4 Teaching Practice

4 = Advanced 3 = Proficient 2 = Basic 1 =Unsatis factory 2 2 1 1 2 2 2 1 3 3 2 1 4 4 3 2

20

Conjoint Model for Combining 3 Measures

To Get a Summary Rating of 4 Need Scores of at Least:

4 on two measures and 3 on the other

3 2

2 on the practice measure and 4 on both the student achievement measures - or 3 on the practice measure and 3 on at least one of the student achievement measures 2 on the practice measure and 2 on either of the student achievement measures

1

1 on the practice measure and 1 on either student achievement measure

Teacher Evaluation in Tennessee

From Race to the Top to First to the Top E ducating O ur C hildren, E ngaging O ur P arents, E mpowering O ur S chools

Evaluation

The ultimate goal of all teacher assessments and evaluations should be…

TO IMPROVE TEACHING AND LEARNING E ducating O ur C hildren, E ngaging O ur P arents, E mpowering O ur S chools

First to the Top Law on Evaluation

• Requires annual evaluation of all teachers and principals • 50% student achievement data:  35% TVAAS where available, 15% other objective measures • 50% other qualitative data include:  Review of prior evaluations  Personal conferences re: strengths, weaknesses and remediation  For teachers, classroom or position observation followed by written assessment  For principals, additional criteria pursuant to their employment contract

E ducating O ur C hildren, E ngaging O ur P arents, E mpowering O ur S chools

General Guidelines

• Evaluations will be used to inform human resource decisions, including but not limited to:  Tenure and dismissal  Compensation  Assignment and promotion  Hiring  Professional development •

LEAs may develop alternative evaluation procedures which must be approved according to policies and rules adopted by the SBE. E ducating O ur C hildren, E ngaging O ur P arents, E mpowering O ur S chools

Categories of Educators

Teachers with TVAAS data untested subjects untested grades Teachers without TVAAS data Library Information Specialists Special Groups counselors social workers non-classroom educators Principals assistant principals

Not included in TEAC authority: central office staff

E ducating O ur C hildren, E ngaging O ur P arents, E mpowering O ur S chools

Criteria for Evaluations

Educator Evaluation 35% Student Growth 15% Student Achievement 50% Other Criteria

E ducating O ur C hildren, E ngaging O ur P arents, E mpowering O ur S chools

50% Quantitative Data

Teachers

35% Student Growth

• TVAAS where available • School-wide TVAAS for all other teachers • Developing alternative growth measures for non-tested subjects/grades

15% Student Achievement

• Selected from “menu of options” adopted/approved by SBE

Principals

35% Student Growth

• School-wide TVAAS

15% Student Achievement

• Selected from “menu of options” adopted/approved by SBE

E ducating O ur C hildren, E ngaging O ur P arents, E mpowering O ur S chools

Growth Measures for Non-tested

• TDE convened educator workgroups in 12 areas of non tested subjects and grades.

• Teams provided recommendations in February 2011.

• All recommendations are being vetted by the TDE and a technical advisory committee to determine validity, reliability and feasibility. • Until such measures are available, educators in non-tested subjects and grades will be evaluated using a TVAAS composite score for the growth component.

E ducating O ur C hildren, E ngaging O ur P arents, E mpowering O ur S chools

15% Student Achievement

• For the 15% achievement portion of the teacher evaluation, the State Board approved a menu of options from which teachers may choose, in cooperation with their administrator, by October 1. • The chosen measures should reflect the educator’s primary responsibility as directly as possible.

• Top 3 quintiles may use TVAAS score.

• Measures are under review for appropriateness and scalability.

E ducating O ur C hildren, E ngaging O ur P arents, E mpowering O ur S chools

Qualitative Appraisals

• For teachers the qualitative appraisal instrument must address the following domains:  Instruction  Planning  Environment  Professionalism • For principal/assistant principal the qualitative appraisal instrument will be based on Tennessee Instructional Leadership Standards (TILS).

E ducating O ur C hildren, E ngaging O ur P arents, E mpowering O ur S chools

Outlining the process

• TDE to provide user-friendly, manageable forms to document observations and personal conferences • Future goal: all forms and data entry will be done electronically • Minimum 4 observations for professional licensed teachers (2 -semester) • Minimum 6 observations for other licensure categories (3-semester) • Feedback from observation visits:  Detailed feedback, highlighting areas of strength and refinement  At least ½ of all observations must be unannounced  Written feedback within a week  In-person debrief scheduled within a week

E ducating O ur C hildren, E ngaging O ur P arents, E mpowering O ur S chools

Category

Guidelines for the Evaluations

35% Student Growth 15% Student Achievement 50% Other Mandatory Criteria (Minimums)

Teachers with TVAAS Teachers without TVAAS Apprentice Licensed Teachers Principals, Assistant Principals Individual TVAAS score School-wide value added; other identified or developed measures Menu of options; top 3 quintiles may use TVAAS score Multiple sources; 4 observations for professional licensed, 2/semester, minimum 60 minutes annually; at least half unannounced Menu of options; top 3 quintiles may use TVAAS score or growth score Multiple sources; 4 observations for professional licensed, 2/semester, minimum 60 minutes annually; at least half unannounced Individual TVAAS scores TVAAS composite; other identified or developed measures School-wide value added Menu of options; top 3 quintiles may use TVAAS score or growth score Multiple sources; 6 observations, 3/semester, minimum 90 minutes annually, (also other non professional licenses) Menu of options; top 3 quintiles may use TVAAS score Multiple sources; 2 onsite observations; qualitative appraisal based on TILS, review of teacher evaluation quality; surveys Special Groups School-wide value added; menu of Menu of options Multiple sources;

E ducating O ur C hildren, E ngaging O ur P arents, E mpowering O ur S chools

4 observations, 2/semester, options; other identified or developed measures minimum 60 minutes annually; at least half unannounced

Evaluations will differentiate educators into five effectiveness groups:

E ducating O ur C hildren, E ngaging O ur P arents, E mpowering O ur S chools

State Model

• The Tennessee Educator Acceleration Model (TEAM) has been adopted as the state evaluation model.

• TEAM utilizes the TAP rubric for observations.

• TEAM observers must complete a four-day training session and pass an online test to be certified as observers.

E ducating O ur C hildren, E ngaging O ur P arents, E mpowering O ur S chools

Other Evaluation Models

Alternative evaluation models developed and adopted: •Memphis—Teacher Effectiveness Measure (Gates supported based on IMPACT model) •Hamilton County—Project COACH •Association of Independent and Municipal Schools (AIMS) — Teacher Instructional Growth for Effectiveness and Results (TIGER)

E ducating O ur C hildren, E ngaging O ur P arents, E mpowering O ur S chools

Evaluation Appeals Process

Teachers may appeal: 1)Accuracy of data used in evaluation 2)Adherence to evaluation policies adopted by SBE

E ducating O ur C hildren, E ngaging O ur P arents, E mpowering O ur S chools

Evaluation Appeals Process

Three-step process: 1) 15 days to appeal to evaluator, who has 15 days to issue decision in writing 2) 15 days to appeal to director of schools or designee, who has 15 days to issue a written decision 3) 15 days to appeal to school board (final step), which has 30 days to conduct a hearing and 30 days to render a decision

E ducating O ur C hildren, E ngaging O ur P arents, E mpowering O ur S chools

Short Summary

State Action

• More than half the states have enacted legislation changing how teachers are evaluated • All require a

combination

of indicators including: – Measures of instructional practice – Student achievement data • State accountability test data • Other test data, that usually can include short cycle assessment data – Short cycle can comprise up to 35% of the data on student learning, so are important options

Advantages of Short Cycle Data

• Multiple kinds: – Renaissance Learning STAR assessments • online administration for immediate feedback, can be administered monthly, online instructional help – Several others – AIMS Web, NWEA Map, etc.

• Designed in the first instance to help teachers improve their instructional practice • Gives formative feedback during the year on how the class is doing • So short cycle assessments, designed to help teachers be more effective, can now also be used to measure teacher effectiveness

Contact Information

Dr. Allan Odden, University of Wisconsin-Madison [email protected]

[email protected]

Dr. Damian Betebenner, Center for Assessment [email protected]

Al Mance, Tennessee Education Association [email protected]