Tips for Using This Template

Download Report

Transcript Tips for Using This Template

Measuring teachers' contributions to student learning in the non-tested subjects and grades

Laura Goe, Ph.D.

Research Scientist, ETS, and Principal Investigator for the National Comprehensive Center for Teacher Quality

Summit II: Using Student Performance Measures in a Comprehensive System to Improve Teaching and Learning

Sunday, February 26, 2012  Phoenix, AZ

Laura Goe, Ph.D.

• Former teacher in rural & urban schools   Special education (7 th & 8 th grade, Tunica, MS) Language arts (7 th grade, Memphis, TN) • Graduate of UC Berkeley’s Policy, Organizations, Measurement & Evaluation doctoral program • Principal Investigator for the National Comprehensive Center for Teacher Quality • Research Scientist in the Performance Research Group at ETS 2

The National Comprehensive Center for Teacher Quality

• A federally-funded partnership whose mission is to help states carry out the teacher quality mandates of ESEA • • Vanderbilt University Learning Point Associates, an affiliate of American Institutes for Research • Educational Testing Service 3

Today’s presentation available online

• To download a copy of this presentation or look at it on your iPad, smart phone or laptop, go to www.lauragoe.com

  Go to Publications and Presentations page Today’s presentation is at the bottom of the page 4

The goal of teacher evaluation

The ultimate goal of all teacher evaluation should be…

TO IMPROVE TEACHING AND LEARNING

Reform, not just compliance

• Much of the conversation around implementing rigorous teacher and principal evaluation systems focuses on compliance and meeting requirements • When discussing implementation, the focus should be less on compliance and more on genuine reform —

how to ensure that the work we’re doing will actually contribute to improving teaching and learning

A simple definition of teacher effectiveness

Anderson (1991) stated that “… an effective teacher is one who quite consistently achieves goals which either directly or indirectly focus on the learning of their students” (p. 18).

Race to the Top definition of student growth

•

Student growth

means the change in student achievement (as defined in this notice) for an individual student between two or more points in time. A State may also include other measures that are rigorous and comparable across classrooms. (pg 11)

8 8

Measures: The right choice depends on what you want to measure

Measures and models: Definitions

• Measures are the instruments, assessments, protocols, rubrics, and tools that are used in determining teacher effectiveness • Models are the state or district systems of teacher evaluation including all of the inputs and decision points (measures, instruments, processes, training, and scoring, etc.) that result in determinations about individual teachers’ effectiveness 10

Multiple measures of teacher effectiveness

• • •

Evidence of growth in student learning and

competency

 Standardized tests, pre/post tests in untested subjects    Student performance (art, music, etc.) Curriculum-based tests given in a standardized manner Classroom-based tests such as DIBELS

Evidence of instructional quality

 Classroom observations    Lesson plans, assignments, and student work Student surveys such as Harvard’s Tripod Electronic portfolios/evidence binders

Evidence of professional responsibility

 Administrator/supervisor reports, parent surveys  Teacher reflection and self-reports, records of contributions 11

Validity

• Measures don’t “have” validity—depends on how the measure is used • There is little research-based support for the validity of using ANY measures, including student growth measures, for teacher evaluation  Multiple measures may serve as a means of triangulating teacher effectiveness results • Herman et al. (2011) state, “Validity is a matter of degree (based on the extent to which an evidence-based argument justifies the use of an assessment for a specific purpose).” (pg. 1) 12

Recommendation from NBPTS Task Force (Linn et al., 2011)

Recommendation 2: Employ measures of student learning explicitly aligned with the elements of curriculum for which the teachers are responsible. This recommendation emphasizes the importance of ensuring that teachers are evaluated for what they are teaching.

Measuring teachers’ contributions to student learning growth: A summary of current models Model

Student learning objectives

Description

Teachers assess students at beginning of year and set objectives then assesses again at end of year; principal or designee works with teacher, determines success Subject & grade alike team models (“Ask a Teacher”) Content Collaboratives Pre-and post-tests model Teachers meet in grade-specific and/or subject-specific teams to consider and agree on appropriate measures that they will all use to determine their individual contributions to student learning growth Content experts (external) identify measures and groups of content teachers consider the measures from the perspective of classroom use; may not include pre- and post measures Identify or create pre- and post-tests for every grade and subject School-wide value-added Teachers in tested subjects & grades receive their own value-added score; all other teachers get the school-wide average 15

1 0 3 2 8 5 4 7 6

School-wide VAM illustration

Obs/Surv VAM 16

Differentiating among teachers

• “It is nearly impossible to discover and act on performance differences among teachers when documented records show them all to be the same .” (Glazerman et al., 2011, pg 1) 17

NYSUT Model

• • Standardized test will be used as part of teachers’ scores in appropriate grades/subjects but 20% of a teachers’ evaluation score will be based on locally determined measures of student learning growth.

“Group alike” teachers, meeting with facilitators, determine which assessments, rubrics, processes can be used in their subjects/grades (multiple measures) • Assessments must focus on standards, be given in a “standardized” way, i.e., giving pre-test on same day, for same length of time, with same preparation • Assessments are recommended to the district which then considers and approves measures • Consortium of districts share measures, post online 18

Tripod Survey

•

Tripod Survey – the 7 C’s

– Caring about students (nurturing productive relationships); – – Controlling behavior (promoting cooperation and peer support); Clarifying ideas and lessons (making success seem feasible); – Challenging students to work hard and think hard (pressing for effort and rigor); – – – Captivating students (making learning interesting and relevant); Conferring (eliciting students’ feedback and respecting their ideas); Consolidating (connecting and integrating ideas to support learning) 19

What assessments are teachers and schools going to use?

• • • • • • • Existing measures      Curriculum-based assessments (come with packaged curriculum) Classroom-based individual testing (DRA, DIBELS) Formative assessments such as NWEA Progress monitoring tools (for Response to Intervention) National tests, certifications tests Rigorous new measures (may be teacher created) The 4 Ps: Portfolios/products/performance/projects School-wide or team-based growth Pro-rated scores in co-teaching situations Student learning objectives

Any measure that demonstrates students’ growth towards proficiency in appropriate standards

Using evidence of student learning growth in teacher evaluation

• Teacher preparation for measuring student learning growth is limited or non-existent • Most principals, support providers, instructional managers, and coaches are poorly prepared to make judgments about teachers’ contribution to student learning growth • They need to know how to  Evaluate the appropriateness of various measures of student learning for use in teacher evaluation Work closely with teachers to select appropriate student growth measures and ensure that they are using them correctly and consistently 22

The 4 Ps (Projects, Performances, Products, Portfolios)

• Yes, they can be used to demonstrate teachers’ contributions to student learning growth • Here’s the basic approach  Use a high-quality rubric to judge initial knowledge and skills required for mastery of the standard(s)  Use the same rubric to judge knowledge and skills at the end of a specific time period (unit, grading period, semester, year, etc.) 23

Collect evidence in a standardized way (to the extent possible)

• Evidence of student learning growth  Locate or develop rubrics with explicit instructions and clear indicators of proficiency for each level of the rubric  Establish time for teachers to collectively examine student work and come to a consensus on performance at each level Identify “anchor” papers or examples  Provide training for teachers to determine how and when assessments should be given, and how to record results in specific formats 24

Assessing Musical Behaviors: The type of assessment must match the knowledge or skill 4 types of musical behaviors: Types of assessment

Responding Creating Performing Listening 1.

Rubrics Playing tests Written tests Practice sheets Teacher Observation Portfolios Peer and Self Assessment Slide used with permission of authors Carla Maltas, Ph.D. and Steve Williams, M.Ed. See reference list for details.

Score combining and weighting

• There is no research to help us know what how to combine and weight scores from multiple measures • States are all over the map in terms of how they’re approaching the scoring and combining • Consider that weights may have an impact on instructional decisions 26

Tennessee approved assessments for non-tested subjects & grades

Tennessee overall score calculator

•Overall Observation Score x 50 •Growth Score x 35 •Achievement Measure Score x 15 •Overall Effectiveness Rating 1 = Less than 200 2 = 200+ 3 = 275+ 4 = 350+ 5 = 425-500 28

Washington DC IMPACT: Instructions for teachers in non-tested subjects/grades

“In the fall, you will meet with your administrator to decide which assessment(s) you will use to evaluate your students’ achievement. If you are using multiple assessments, you will decide how to weight them. Finally, you will also decide on your specific student learning targets for the year. Please note that your administrator must approve your choice of assessments, the weights you assign to them, and your achievement targets. Please also note that your administrator may choose to meet with groups of teachers from similar content areas rather than with each teacher individually.”

Washington DC IMPACT: Rubric for Determining Success (for teachers in non tested subjects/grades)

DC Impact: Score comparison for Groups 1-3

Group 1 (tested subjects) Group 2 (non tested subjects Group 3 (special education)

Teacher value-added (based on test scores) Teacher-assessed student achievement (based on non-VAM assessments) Teacher and Learning Framework (observations) 50% 0% 35% 0% 10% 75% 0% 10% 55% 32

Rhode Island’s SLO language

• “Student Learning Objectives are not set by educators in isolation; rather, they are developed by teams of administrators, grade level teams or groups of content-alike teachers and, are aligned to district and school priorities, wherever possible.” (pg 12) From Rhode Island’s “

Guide to Measures of Student Learning for Administrators and Teachers 2011 2012” http://www.ride.ri.gov/educatorquality/educator evaluation/Docs/GuideSLO.pdf

Rhode Island student learning objectives: Administration & scoring

• Administration & Scoring:  The teacher should explain how the evidence used to assess the objective will be collected and reviewed. The teacher should include detail about how assessments will be administered and scored. The teacher and evaluator should determine the most accurate, fair, and objective scoring process possible. 34

Rhode Island student learning objectives: Scoring

• Scoring  The evaluator should review all the available evidence related to Student Learning Objectives, noting the degree to which the objective was met on the form. Evaluators will informally rate each objective as Not Met, Met, or Exceeded. The evaluator may provide additional comments about the scoring. These informal ratings will serve as the basis for the holistic scoring. Using the Student Learning Objective scoring guidelines, evaluators will look at the whole body of evidence across all objectives and assign an overall Student Learning Objective rating. 35

Rhode Island student learning objectives: Results

• Results  At the end of the interval of instruction, the teacher should explain the results of all sources of evidence used to assess the objective. The results should be expressed numerically and in relation to the previously set targets. If any official score reports are available for the sources of evidence used (especially for common assessments) they should be submitted to the evaluator prior to the End-of-Year Conference.

Rhode Island student learning objectives: Rubric for final score

Description Score

Exceptional Attainment (5) Full Attainment (4) Considerable Attainment (3) Evidence across all Student Learning Objectives indicates exceptional student mastery or progress. All objectives are exceeded. This category is reserved for the educator who has surpassed expectations described in their Student Learning Objectives and/or demonstrated and outstanding impact on student learning.

Evidence across all Student Learning Objectives indicates superior student mastery or progress. This category applies to the educator who overall has nearly met the majority of the expectations described in their Student Learning Objectives and/or who has demonstrated a notable impact on student learning.

Evidence across all Student Learning Objectives indicates significant student mastery or progress. If an objective was not met, evidence indicates that is was nearly met. This category applies to the educator who overall has nearly met the majority of the expectations described in their Student Learning Objectives and/or who has demonstrated a considerable impact on student learning.

Partial Attainment (2) Evidence across all Student Learning Objectives indicates some student mastery or progress. Educator may have met or exceeded some objectives and no met other objectives. Educator may have nearly met all objectives. This category applies to the educator who has demonstrated an impact on student learning, but overall has not met the expectations described in their Student Learning Objectives.

Minimal or No Attainment (1) Evidence across all Student Learning Objectives indicates little student mastery or progress. This category applies to the educator who has not met the expectations described in their Student Learning Objectives and has not demonstrated a sufficient impact on student learning. This category also applies when evidence of objectives is missing, incomplete, or unreliable or when the educator has not engaged in their in the process of setting and gathering evidence for Student Learning Objectives.

Teacher and principal evaluation in isolated and/or low-capacity districts

• External evaluators may need to be brought in for very small, isolated districts   For example, a district where the superintendent is also principal, history teacher, and bus driver May also be needed when evaluators’ objectivity is impacted by factors such as fear of losing teachers or damaging long-term relationships in the community • Evaluators could be “exchanged” across districts within a specific region (“you evaluate mine, and I’ll evaluate yours”) or regional evaluators could serve a set of districts 38

Questions to consider in deciding on weighting strategies

• What are the consequences (to teachers, schools, and districts), both good and bad, of a specific weighting strategy?

• How will the weighting strategy impact schools use of resources (evaluator and teacher time and training) • How will the weighting strategy impact teacher morale, school culture, recruitment and retention?

Considerations for implementing measurement system

• • • Consider whether human resources and capacity are sufficient to ensure fidelity of implementation  Poor implementation threatens validity of results • Establish a plan to evaluate measures to determine if they can effectively differentiate among teacher performance   Need to identify potential “widget effects” in measures If measure is not differentiating among teachers, may be faulty training or poor implementation, not the measure itself Examine correlations among results from measures Evaluate processes and data each year and make needed adjustments 40

Before you implement, ask yourself…

• How will this component of the teacher and principal evaluation system impact teaching and learning in classrooms and schools?

• How will this component look different in low capacity vs. high-capacity schools? • How will reporting on this component be done (to provide actionable information to teachers, principals, schools, districts, teacher preparation programs, and the state)?

• How will we know if this component is working as we intended?

Resources and links

• • • • • • Harvard’s Tripod Survey http://www.tripodproject.org/index.php/index/ National Response to Intervention Center Progress Monitoring Tools http://www.rti4success.org/chart/progressMonitoring/progressmonitoring toolschart.htm

Colorado Content Collaboratives http://www.cde.state.co.us/ContentCollaboratives/index.asp

New York State Evaluation http://engageny.org/administrators/ Rhode Island Department of Education Teacher Evaluation – Student Learning Objectives http://www.ride.ri.gov/educatorquality/educatorevaluation/SLO.aspx

Tennessee Teacher Evaluation http://team-tn.org/ 42

References

Anderson, L. (1991).

Increasing teacher effectiveness. Paris: UNESCO, International Institute for Educational Planning.

Glazerman, S., D. Goldhaber, et al. (2011). Passing muster: Evaluating evaluation systems. Washington, DC, Brown Center on Education Policy at Brookings.

http://www.brookings.edu/reports/2010/1117_evaluating_teachers.aspx

Herman, J. L., Heritage, M., & Goldschmidt, P. (2011).

Developing and selecting measures of student growth for use in teacher evaluation. Los Angeles, CA: University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).

http://www.aacompcenter.org/cs/aacc/view/rs/26719 Linn, R., Bond, L., Darling-Hammond, L., Harris, D., Hess, F., & Shulman, L. (2011).

Student learning, student achievement: How do teachers measure up? Arlington, VA: National Board for Professional Teaching Standards.

http://www.nbpts.org/index.cfm?t=downloader.cfm&id=1305 Malta, C., and Williams, S. (January 27, 2010). Meaningful assessment in the music classroom. Presented at Missouri Music Educators Association Conference, Jefferson City, MO.

http://dese.mo.gov/divimprove/curriculum/fa/AssessmentintheMusicClassroom.pptx

National Response to Intervention Center Progress Monitoring Tools http://www.rti4success.org/chart/progressMonitoring/progressmonitoringtoolschart.htm

Questions?