Transcript Multiple Perspectives on Computer Adaptive Testing for K

U.S. Department of Education
David J. Harmon, Ph.D.
Program Specialist
Policy implications from the federal perspective
June 20, 2010






Part 200 Final Regulations for Standards
and Assessments - Analysis of Comments
Early Implications of NCLB Assessment
System Requirements on CAT
NCLB Assessment System Requirements
Evidence of Comparability
Beyond NCLB
References
2
3

NCLB requires that all students be held to
the same achievement standards. This
means that assessments for a particular
grade and subject must elicit comparable
content knowledge and understanding,
within the framework of the standards, for
all students tested.
4



Individual level assessments (adaptive
assessments) would measure the performance
of some students at a particular grade level
against lower standards.
This would result in some schools being held
to lower standards than other schools.
Use of levels assessments would not allow all
schools and students to be held to the same
high standards required by the NCLB Act.
5
6

Early in NCLB “Blended Format”
combined



On-grade level section used for AYP and
Computer-adaptive section for instruction
Used on-grade level section for AYP to


Ensure all students and schools were being
held to the same content standards and
Same academic achievement standards
7


Oregon currently is the only State that has
received approval from the U.S.
Department of Education through the peer
review process to use CAT for AYP
designation and to meet NCLB assessment
requirements.
Some assessment requirements seem
more complex and challenging when using
CAT!!
8
9

NCLB assessment system requirements as
provided in the Standards and Assessment
Peer Review Guidance (January 2009)
apply equally to fixed form and adaptive
tests.



All forms are aligned with the State’s “grade
level” academic content and achievement
standards and yield comparable results;
All forms are equivalent to one another in terms
of content coverage, difficulty, and quality;
All assessments yield comparable results for all
subgroups;
10

If the assessment system includes various
instruments (e.g., the general assessment
in English and either a native-language
version or simplified English version of the
assessment), does the State demonstrate
comparable results and alignment with the
academic content and achievement
standards;
11


When different test forms or formats are
used, the State must ensure that the
meaning and interpretation of results are
consistent;
If the State administers both an online and
paper and pencil test, has the State
documented the comparability of the
electronic and paper forms of the test;
12


Are the assessments and the standards
aligned comprehensively, meaning that
the assessments reflect the full range of
the State’s academic content standards;
Do the assessments yield scores that
reflect the full range of achievement
implied by the State’s academic
achievement standards?
13

The following are among the requirements
that Oregon met to achieve full
assessment system approval via ED’s peer
review process:

Adaptive tests are comparable to each other
and paper/pencil versions at the achievement
levels, restricted to grade-level content, and
matched to detailed grade-level test blueprints.
14

Requirements met Oregon (cont.)
 Consistency of strand content among the
paper-and-pencil and computer-adaptive
versions documented with detailed test form
construction rules and test maps.
 Comparability of paper-and-pencil and
computer-adaptive test difficulties between
school years.
 Evidence of comparability among English and
side-by-side dual-language science test items
and forms (for both Spanish and Russian).
15
16

Content


Adaptive and paper tests are built to the same
grade level test specifications and blueprints
with weighted content allocations by strand
Consistency of strand content among the
paper-and-pencil and computer-adaptive
versions was adequately documented by
detailed test form construction rules and test
maps
17

Content

Oregon submitted a consistency of strand
content in computer administered tests study
 Each test record contains a count of operational
items by strand.
 Oregon’s adaptive test delivery algorithm was
designed to constrain the number of items by
strand to match the test blueprints.
 Study supported content comparability at the
strand level.
18

Difficulty/Scale Comparability

Research study indicated:
 There were no significant differences in the grade
levels mean scale scores across adaptive and
paper administered tests and
 The correlations were moderate-high (0.65 to
0.84) for the double tested group.
19

Difficulty/Scale Comparability

Comparability of paper and adaptive test
difficulties between school years, Oregon
submitted external pilot and follow up studies
on year-to-year comparability of Rasch
parameter estimates. Both studies show highly
consistent parameter estimates and support
comparability between years.
20
21


ED supports the development and use of a
new generation of assessments that are
aligned with college- and career-ready
standards, to better determine whether
students have acquired the skills they need
for success. (Blueprint)
New assessment systems will better capture
higher-order skills, provide more accurate
measures of student growth, and better
inform classroom instruction to respond to
academic needs. (Blueprint)
22

Race to the Top Assessment Program - The
Department published the notice inviting
applications (NIA) provides

A flavor of some of the things that will be
considered in assessments under the
reauthorization blueprint, subject to
Congressional approval
23



Adaptive testing is a means not an end.
No one knows the implications of the
ESEA reauthorization for assessment
systems or what roles CAT may play.
If subsequent ESEA assessment
requirements can be achieved using CAT, I
believe ED will be encouraging.
24




Title I--Improving the Academic Achievement of the
Disadvantaged, 34 CFR Part 200, Final Regulations for Standards
and Assessment (July 5, 2002)
Standards and Assessments Peer Review Guidance: Information
and Examples for Meeting Requirements of the No Child Left
Behind Act of 2001. (January 12, 2009)
http://www2.ed.gov/admins/lead/account/saa.html#peerreview
A Blueprint for Reform - The Reauthorization of the Elementary
and Secondary Education Act
http://www2.ed.gov/policy/elsec/leg/blueprint/blueprint.pdf
Race to the Top Assessment Program ... The Department
published the notice inviting applications (NIA) for the fiscal year
2010 Race to the Top Assessment Program competition (including
the ...
http://www2.ed.gov/legislation/FedRegister/announcements/201
0-1/012710c.html
25