Application of Growth and Value-Added Models to WASL

A Summary of Issues, Developments and Plans for Washington

WERA Symposium on Achievement Growth Models

June 2, 2006 Joe Willhoft, OSPI

Overall Impressions

•

Shaw’s Rule: “For every complex problem there is a simple solution that is wrong.” -- G.B. Shaw

•

Willhoft’s Corollary: “For every complex problem there is a correct solution that can’t be understood.”

2

Issues Associated with Growth and Value-Added Models

Students must be presented with off-grade-level items. Younger students may not even have studied them; older students may not have studied them recently. Neither seems a fair representation of student performance.

If the curriculum includes blocks of content that are not taught at or before the earlier grade level but are taught at the higher grade level, then the lower grade level test has questionable validity for inferences to the domain of the trait across the two grade levels. In using the scale, performance on off-grade-level items is estimated from performance on on-grade-level items, presenting a validity concern. Growths in different regions of a vertical scale developed across several grade levels are not comparable. It is possible that students in different grades achieve the same scores. However, their educational experiences are different and therefore, appropriate achievement level descriptions differ.

3

Issues Associated with Growth and Value-Added Models

(Cont.) Students can show negative growth. Since this is possible, given enough replications, it will happen. Explanations likely will be developed that depend on the differences between the content at the two grade levels, and that begs the question of why the two tests were put on the same scale.

External achievement standards may be disordinal. For example, the cut score for “proficient” may be lower on the scale for grade five than it is for grade four. Since this can happen, given enough replications it will happen. Clearly some “heroic fudging” will be needed before the scale can be used.

Students from different grade levels with the same score will not have the same growth expectations.

4

Additional Concerns with Growth and Value-Added Models

• • • •

Record keeping systems must be more robust Missing records are usually not random Implementation of variables is inconsistent across units in VAM Modeling growth and VA is intuitively simple, but technically complex

5

All that may be true, but….

•

We will have tests in Reading and Math across grades 3 through 8

•

Parents, principals, superintendents, policymakers and the public at-large will not accept that we cannot or should not measure growth

6

Isn’t Value-Added an Improvement over Growth Models?

•

This depends on audience and purpose

–

Parents probably more interested in growth from one year to the next

–

Policymakers probably more interested in improvements conditioned on input factors (“value-added”)

•

More precise models are more complex and difficult to explain

7

How is ED Using Growth for AYP

• • • • •

“Improvement” already used as Safe Harbor States invited to apply as pilots in December 2005 (No more than 10 would be approved) In May, Sec’y Spellings approved North Carolina and Tennessee as pilot states Alaska, Arkansas, Arizona, Delaware, Florida, and Oregon will be given “early consideration” in the next round. A total of no more than 10 pilot states will be approved.

8

ED Use of Growth Models for AYP

•  

States required to meet the following: All students proficient by 2014 with annual state goals to close ach. gap; Annual ach. expectations based on grade-level proficiency, not on student background or school characteristics;

  

Schools accountable for achievement in reading/LA and mathematics; All students are included in the assessment/accountability system; Assessments in grades 3-8 and high school have been operational for more than one year, and have received approval through the NCLB peer review process for the 2005-06 school year.

  

The assessment system must also produce comparable results from grade to grade and year to year; Track student progress as part of the state data system; and Continue to include student participation rates and student achievement as separate academic indicators in the state accountability system.

9

What Are OSPI Plans?

• •

2005 Pilots in in grades 3, 5, 6, 8 and Operational forms in 4 and 7 included “vertical” forms

–

Pilot item locations contained items from preceding or following grade levels National TAC requested technical review of scaling, growth and VAM

–

OSPI contracted for development of technical treatment of the topic for NTAC (Available upon request)

10

What Are OSPI Plans?

• •

2007 tests in 3 thru 8 will include “vertical” forms

–

Pilot item locations will contain items from preceding grade levels

–

Did not use 2006 per NTAC recommendation of new test Results of vertical scaling to be presented to NTAC in January 2008

11

What Are OSPI Plans?

• • •

Hope to develop at least three-year spans:

– – –

3-4; 3-4-5; 4-5-6; 5-6-7; 6-7-8 OR A 3-4-5 scale and a 6-7-8 scale May have to settle for paired grades: 3-4, 4-5, etc.

OSPI Technology shop is developing longitudinal database for tracking student scores across time Monitor developments in other states for ED approval to use Growth Models or VAM for NCLB

12

Questions/Discussion

13