Principal Evaluation: The National Story

CCSSO April 2011 Joseph Murphy Vanderbilt University

I. SETTING THE STAGE: EVALUATION IN CONTEXT

Standards

Conditions of Work

preparation internship accreditation/program approval teacher leadership preparation l c i e n s u r e evaluation governance incentives professional development Career Line induction (residency) professional development relicensure mentoring

Education

Standards

II. PRINCIPAL EVALUATION: THE NATIONAL STORY

PROBLEMS WITH CURRENT EVALUATION SYSTEMS

Content

• • Little evidence that systems evaluate what is important, i.e. not valid Insufficient attention to leadership for learning, especially curriculum and instruction

Process

• • Limited architecture – focus on one approach (e.g. goals) Perfunctory – not a deep process

Impact

• • • Principals not receiving useful feedback Not promoting professional growth of principals Not promoting organizational improvement

III. BUILDING A NEW PRINCIPAL EVALUATION SYSTEM

A. Building: Developing Guiding Principles of the Evaluation System

• •

Foundation

– Based on ISLLC Standards

Process

– Be valid, reliable, and equitable – Be transparent – Have objectivity, evidenced through multiple measures and supported by collection, analysis and application of appropriate data and evidence

– Underscore both formative and summative components – Be comprehensive yet manageable – Allow for flexibility at local level – Include multiple indicators of academic, social, and emotional growth for all students – Promote collaboration between supervisor and evaluatee – Define and prioritize expectations and areas of emphasis – Provide frequent and specific feedback – Insure due process

•

Outcomes

– Promote personal accountability – Promote continuous professional growth – Promote school improvement that results in student achievement – Provide a process for the district to connect principals to resources and growth opportunities

B. Building a Principal Evaluation System: An Overview

•

Step 1: Anchor the System

– Create the system around two elements: • research on principal impact on student learning – recommendation: ISLLC Standards • organizational outcomes – recommendation: measures of student learning

•

Step 2: Select Components

– Choose components that will tap into or shed light on the two elements noted above – Recommendation: more than one component, no more than five components – Recommendation: include one component on student learning and one on 360 degree feedback on performance

•

Step 3: Value the Components

– Provide weights to the components – Example #1: • organizational goals • • student achievement 360º assessment (VAL ED) • professional developmental goal • total – Example #2: • • 360º assessment (VAL ED) student achievement • total 50% 50% 15% 40% 30% 15% 100% 100%

•

Step 4: Set Performance Levels

– Create performance levels for each component – Recommendation: use four levels • Below basic (inadequate) • • Basic (satisfactory) Proficient • Distinguished

•

Step 5: Set Parameters

– Develop borders for each performance level – Example—student achievement • Below basic ↓.9 years growth • • Basic Proficient .9 - 1.1 years growth 1.2 - 1.3 years growth • Distinguished ↑.1.3 years growth – Example—360º (VAL ED) • Below basic 1:00 - 3:28 • • Basic Proficient 3:29 - 3:59 3:60 - 3:99 • Distinguished 4:00 - 5:00

•

Step 6: Assign Scores

– Provide an assessment score for each component and a total score – Example—components and raw total score • organizational goals (15%) basic (2) 30/60 • student achievement (40%) basic (2) 80/160 • 360º assessment (VAL ED) (30%) proficient (3) 90/120 • professional dev. goal (15%) proficient (3) 45/60 – – 245/400 ( ÷ 4 = 61/100) example—total score: » below basic » » » basic distinguished 25-50 51-65 proficient 66-84 85-100

•

Step 7: Determine Consequences

– Ensure that results are used for: • shaping professional development • retention and promotion • contract length • salary/bonuses – Example—used for contracts: » Distinguished: 3 year contract » Proficient: 2 year contract » » Basic: Below basic: 1 year contract 1 year contract and improvement program

•

Example—used for bonus payments

– Rules in example: • Pool of money = $15,000 per principal (20 principals x 15,000 = $300,000 pool); i.e., maximum bonus = $15,000 • Set bonus guideline so no bonus money for less than “proficient” work • • Bonus allocated by component Three component system in use

Principal Williams

Component 360 Degree Assessment (VAL ED)

Weight 1 Below basic 2 Basic 3 Proficient 4 Distinguished 40% 0 0 4000 6000

Student Learning

40% 0 0 4000 6000 Bonus: 8000

Organizational Goals

20% 0 0 2000 3000

Principal Smith

Component 360 Degree Assessment (VAL ED)

Weight 1 Below basic 2 Basic 3 Proficient 4 Distinguished 40% 0 0 4000 6000

Student Learning

40% 0 0 4000 6000 Bonus: 2000

Organizational Goals

20% 0 0 2000 3000

C. Principal Evaluation Work: Detailed Explanation

Components States

PERFORMANCE GOAL

•

Source of Goal

– National, state, and local assessment – Improvement plans – Standards – Principal self reflection – District goals – School audits – Surveys

•

Essential Characteristics of Goals

– Be linked to ISLLC Standards – Be organizationally grounded and emphasize the direct contributions of the leader – organizationally grounded goal(s) may include district, superintendent, and/or community-based goals and priorities – Be anchored in an analysis of multiple sources around relevant data – sources of data may reflect cognitive or non-cognitive measures

− Be specific, measurable, attainable, realistic, timely, and challenging − − Be mutually determined through collaborative dialogue − Have a longitudinal focus – evaluation goals should address the hard work that will be required over time to promote change and improvement Be collaboratively reviewed with frequent and specific feedback

STUDENT ACHIEVEMENT

• • Source of Goal – – Norm-referenced (state/district) Proficiency – End of course – SAT/ACT – AP – Locally developed Focus of Goal – Level – Equity – Value-added

306 DEGREE ASSESSMENT

• • •

Anchored on ISLLC standards Valid and reliable Helpful for individual and school

PROFESSIONAL GROWTH GOAL

• • Source of goal – – Previous evaluations School improvement plans – Demographic information – District goals Essential characteristics – Directly linked to Standards and indicators for School Improvement and ISLLC Standards – The principal should engage in personal and collegial reflective practice that promotes continuous professional growth

– The principal and supervisor will determine how the acquisition of knowledge and skills will be demonstrated – The principal’s professional growth goal should be differentiated and based on the need determined by collaborative discussion around formative and/or summative evidence – Supervisors shall provide specific and timely feedback, mentoring, and coaching through regularly scheduled collaborative meetings and ongoing communication – The supervisor should provide resources that will aid in this process – It is the responsibility of the principal to collect and maintain data that evidences the impact of personal growth and school improvement

PARENT SATISFACTION

Drill down on ISLLC Standard

IV. VAL-ED: Assessing Learning-Centered Leadership

Acknowledgments

• • The VAL-ED instrument was published and distributed by Discovery Education Assessment starting in July 2008.

We are authors of the VAL-Ed, and while we have made every effort to be objective and data-based in my statements about this instrument in this presentation, readers should judge the facts and related information materials for themselves and make independent decisions regarding the use of the instrument. 2008

Outline of Presentation

• Background on VAL-ED • The VAL-ED instrument • Psychometric properties • Score reports • VAL-ED and professional development 2008

Background on VAL-ED

The Case for Leadership Assessment

• • • Most school leadership improvement focuses on professional development, mentoring, licensing policies, and standards.

Minimal attention has been paid to assessment, feedback, and subsequent action.

Leadership assessment and feedback is an important missing link to improving and strengthening school leadership.

2008

Learning-Centered Leadership

• • • Leaders should be assessed on leadership behaviors associated with student learning.

Learning-centered leadership is leadership for student performance.

Learning-centered leadership is the framework for our leadership assessment system.

2008

Our Conceptual Model

2008

Leadership Behavior Framework

2008

• • • • • •

Definitions of Core Components

High Standards for Student Learning —There are individual, team, and school goals for rigorous student academic and social learning.

Rigorous Curriculum (content) —There is ambitious academic content provided to all students in core academic subjects.

Quality Instruction (pedagogy) —There are effective instructional practices that maximize student academic and social learning.

Culture of Learning & Professional Behavior —There are integrated communities of professional practice in the service of student academic and social learning. There is a healthy school environment in which student learning is the central focus.

Connections to External Communities —There are linkages to family and/or other people and institutions in the community that advance academic and social learning.

Performance Accountability — Leadership holds itself and others responsible for realizing high standards of performance for student academic and social learning. There is individual and collective responsibility among the professional staff and students.

2008

Definitions of Key Processes

• • • • • • Planning—Articulate shared direction and coherent policies, practices, and procedures for realizing high standards of student performance.

Implementing—Engage people, ideas, and resources to put into practice the activities necessary to realize high standards for student performance.

Supporting—Create enabling conditions; secure and use the financial, political, technological, and human resources necessary to promote academic and social learning.

Advocating—Promotes the diverse needs of students within and beyond the school.

Communicating—Develop, utilize, and maintain systems of exchange among members of the school and with its external communities.

Monitoring—Systematically collect and analyze data to make judgments that guide decisions and actions for continuous improvement.

2008

The Development of VAL-ED

• The development of VAL-ED has been supported by a 3-year grant from The Wallace Foundation.

• Three phases of our work: – Phase 1 – Leadership conceptualization and assessment system development – Phase 2 – Field testing the behavior rating scale and exploring its properties – Phase 3 – Dissemination of results and products 2008

The VAL-ED Instrument

The VAL-ED Instrument

• • • • • • The instrument consists of 72 items defining six core component subscales and six key process subscales. Principal, Teachers, & Supervisor provide a 360-degree, evidenced based assessment of leadership behaviors. Respondents rate effectiveness of 72 behaviors on scale 1=Ineffective to 5=Outstandingly effective.

Each respondent rates the principal’s effectiveness after indicating the sources of evidence on which the effectiveness is rated.

Two parallel forms of the assessment facilitate measuring growth over time.

The instrument will be available in both paper and online versions.

2008

Purpose & Uses

• The VAL-ED can be used as part of a comprehensive assessment of the effectiveness of a leader's behaviors.

• The VAL-ED reports principal performance through – – Norm-referenced scores and Criterion-reference scores. • VAL-ED can be used annually or more frequently to: – – – Facilitate a data-based performance evaluation, Measure performance growth, and Guide professional development.

2008

• • • • • •

Implementation

Identify respondents and invite participation.

Discuss use of results & confidentiality. Decide paper or online version.

Time and Timing – Average respondent requires 20 to 25 minutes.

– Schedule completion after respondents have had a reasonable time to observe/experience the leader’s work and its effects on the school.

Designate person(s) to manage collection and submission of response forms, if paper version used.

Ensure teacher confidentiality.

2008

Directions for Completing Rating Scale

2008

An Example Set of Responses

2008

Psychometric Properties

Assessing Learning-Centered Leadership: The VAL-ED vision…

A leadership assessment system that has the following properties: • • • • • • • • • Works well in a variety of settings and circumstances, Is construct valid, Is reliable, If feasible for widespread use, Provides accurate and useful reporting of results.

Is unbiased, Yields a diagnostic profile for summative and formative purposes.

Can be used to measure progress over time in the development of leadership, and Predicts important outcomes.

2008

• • • •

Psychometric Evidence

Item and response scale development

– Based on review of learning-centered leadership literature and alignment to ISLLC standards.

– Critiqued by education leaders and leadership researchers.

Item sorting study

– Established content validity by asking education leaders to sort the items into 36 cells.

Cognitive interviews of paper/pencil version

– Two rounds of cognitive interviews in three districts each.

– Three respondents evaluated the format and items.

Nine-school pilot test

– Estimated reliability of each of 12 scales.

– Established construct validity through factor analysis.

– Established face validity through questions to respondents.

2008

• • • • •

Psychometric Evidence

Cognitive interviews of online instrument including revisions based on 9 school pilot Bias review

– Submitted to urban districts to evaluate language.

11-school pilot test

– Confirmed changes made after 9-school pilot test.

300-school field test

– Conducting differential item functioning to determine biases.

– – Establishing norms.

Setting performance standards.

Proficiency standard setting

– Using bookmark approach with 24 education leaders.

2008

• • • •

Cognitive Interviews—Validity

First round – Sources of evidence cumbersome – Item stem lost – Instructions “wordy” Modifications – Instructions bulleted – – Stem included in each item “Not done” added Conclusions – VAL-ED captured all the relevant leadership behaviors – – Response scale was clear Respondents able to complete task on their own Cognitive interviews of online prototype 2008

• • • • • •

9-School Pilot Study—Validity & Reliability

Methods Confirmatory factor analysis – High Parsimonious Goodness of Fit Index (.93-.96) – First, second, and third-order factor loadings salient Responses to final survey questions – Understood items, sources of evidence Teacher and principal

Scatter of Principal ratings with Teacher Ratings

effectiveness ratings were correlated r =.47

Cronbach’s Alpha above .92 for 108-item-form scales Scores high (generally >4 on the five-point effectiveness scale) 5 4.5

4 3.5

3 2.5

2 3.5

3.7

3.9

4.1

4.3

4.5

4.7

Teacher mean ratings by school r=.47

4.9

2008

• • • • High agreement between schools – r = .79 for teacher-principal,.51 for principal supervisor, .68 for supervisor-teacher Mean effectiveness ratings lower, more variable Alpha above .89 for all 72-item-form scales 2008

2008

Respondent Feedback

Score Reports

• • •

Interpretation of Rating Scale Results

Descriptive Analysis

– Total Score – – Core Components Subscale Scores Key Process Subscale Scores

Norm-Referenced Profiles

– Principal – – Teacher Supervisor – Total respondent composite

Criterion-Referenced Profiles

– Distinguished – Proficient – Basic – Below basic 2008

• • • • –

VAL-ED Results: Performance Descriptors

Leadership behaviors of core components and key processes at levels of effectiveness that over time are virtually certain to influence teachers to bring the school to a point that results in strong value-added to student achievement and social learning for all students.

Proficient

– Leadership behaviors of core components and key processes of sufficient effectiveness that over time are likely to influence teachers to bring the school to a point that results in acceptable value-added to student achievement and social learning for all students.

Basic

– Leadership behaviors of core components and key processes of sufficient effectiveness that over time are likely to influence teachers to bring the school to a point that results in acceptable value-added to student achievement and social learning for some subgroups of students but not all.

Below basic

– Leadership behaviors of core components and key processes of insufficient effectiveness and consistency that over time are unlikely to influence teachers to bring the school to a point that results in acceptable value added to student achievement and social learning.

2008

Aggregated Effectiveness Ratings

2008

Comparisons Across Respondent Groups

2008

Comparisons Across Respondent Groups

2008

VAL-ED and Professional Development

Cell-by-Cell Feedback

2008

VAL-ED and Professional Growth

• • • Cell-by-cell feedback highlights up to 6 potential areas of growth.

Behaviors from these 6 domains are listed.

Areas of growth provide principals with information about key targets for professional development.

2008

Leadership Behaviors for Possible Improvement

Example of a potential area of growth: 2008

• • • • • •

Supporting Research & Publications

Goldring, E., Porter, A.C., Murphy, J., Elliott, S.N., & Cravens, X. (2007, March). Assessing learning-centered leadership: Connections to research, professional standards, and current practice. New York, N.Y.: Wallace Foundation.

Murphy, J., Elliott, S.N., Goldring, E., & Porter, A.C. (2007). Leadership for learning: A research-based model and taxonomy of behaviors. School Leadership & Management, 27 (2), 179-201. Murphy, J., Elliott, S.N., Goldring, E., & Porter, A.C. (in press). Leaders for productive schools. In M. Brundrett & M. Crawford (Eds.), Developing school leaders: An international perspective, London: Routledge.

Murphy, J., Elliott, S.N., Goldring, E.B., & Porter, A.C. (2006). Learning-centered leadership: A conceptual foundation. New York, NY: Wallace Foundation.

Porter, A.C., Goldring, E.B., Murphy, J., Elliott, S.N., & Cravens, X. (2006). A framework for the assessment of learning-centered leadership. New York, NY: Wallace Foundation.

Murphy, J.F., Goldring, E.B., Cravens, X.C., Elliott, S.N., Porter, A.C. (2007, August). The Vanderbilt Assessment of Leadership in Education: Measuring Learning-Centered Leadership. Journal of East China Normal University.

These and other publications are all available for download at http://www.vanderbilt.edu/lsi/valed/featured.html.

2008

Visit:

To Learn More …

http://www.val-ed.com

Contact : Discovery Education 2416 21st Avenue, South, Suite 300 Nashville, TN 37212 2008

CONTACT INFORMATION

Joseph Murphy Frank W. Mayborn Chair Box 404 GPC Vanderbilt University 230 Appleton Place Nashville, TN 37203-5721 615-322-8038 [email protected]