Transcript Slide 1

Issues Related to Judging the
Alignment of Curriculum Standards
and Assessments
Norman L. Webb
Wisconsin Center for Education Research
University of Wisconsin-Madison
Annual Meeting of the American Educational
Research Association
Montreal, April 11, 2005
This work was supported by a subgrant from the U. S. Department of Education
(S368A030011) to the State of Oklahoma and a grant from the National Science
Foundation, (EHR 0233445) to the University of Wisconsin—Madison. Any
opinions, findings, or conclusions are those of the author and do not necessarily
reflect the view of the supporting agencies.
Webb Alignment Process
Identify Standards and Assessments
 Select 6-8 Reviewers (Content Experts)
 Train Reviewers on DOK Levels
 Part I: Code DOK Levels of the
Standards/Objectives
 Part II: Code DOK Levels and
Corresponding Objectives of Assessment
Items

Reports
Standards DOK analysis
Degree of alignment by standard
Source of Challenge
Reviewers’ Notes
General Comments Made by Reviewers
Reliability Among Reviewers
Overall Finding of the Degree of Alignment
Data Tables








Acceptable levels on four alignment criteria
Source of challenge
DOK by item and intraclass correlation
Notes
DOK level and objective code by item
Objectives by item
Items for each objective
Number of reviewers coding an item by objective
Alignment Criteria
 Categorical Concurrence
Number of items per standards
 Depth-of-Knowledge Consistency
Percent below, at, and above
 Range-of-Knowledge Correspondence
Percent of objectives with at least one item
 Balance of Representation
Index 0 to 100
What are appropriate acceptable
levels for each criterion?
Number of items for one standard
 Distribution of items by complexity
 Number of standard-objectives with at
least one item
 Distribution of items among objectives
under a standard

Alignment Levels Using the Four Criteria
Alignment
Level
Acceptable
Weak
Categorical
Concurrence
Depth of
Knowledge
Range of
Knowledge
Balance of
Representation
6 items per
standard
50%
50%
70%
---
40%-49%
40%-49%
60%-69%
Less than
40%
Less than
40%
Less than
60%
Unacceptable Fewer than 6
items per
standard
State A
Categorical Concurrence for Grade 3 Science
(N=55 items)
Standards
Title
3.1 - History/Nature
3.2 - Inquiry
Hits
Mean
S.D.
1
0
17.38
2.12
Cat. Concurr.
NO
YES
3.3 - Unifying Themes
7.5
4
YES
3.4 - Subj Matter/Conc
33.5
1.94
YES
3.5 - Design/Applic
3.6 - Personal/Social
2.12
4.75
1.27
1.09
NO
NO
66.25
5.78
Total
State B
Depth-of-Knowledge Consistency High School
Mathematics (N=51 items)
Standards
Title
# Hits
% Under % At
% Above
DOK
Consistency
M
M
M
M
10.44
83
17
0
NO
II - Geometry and Measurement
13
20
51
29
YES
III – Data Analysis and Statistics
13.44
58
40
2
WEAK
IV - Number Sense and
Numeration
2.78
25
61
14
YES
V - Numerical and Algebraic
Operations and Analytical ...
10.67
30
57
12
YES
VI - Probability and Discrete
Mathematics
6.89
42
56
2
YES
57.22
43
47
11
I - Patterns, Relationships and
Functions
Total
III
–
Mathematics Standards
V
VI
-P
-N
um
s
cr
et
e
r.
se
O
pe
/D
is
Al
g
ro
b
&
rS
en
/S
ta
t
s
s
/M
ea
is
al
ys
eo
m
/F
cn
te
rn
s
um
be
An
-N
at
a
IV
D
G
Pa
t
II
-
I-
Percent
State B DOK Consistency
100
80
% Above
60
40
% At
% Under
20
0
State B
Range of Knowledge Correspondence
High School Mathematics (N=51 items)
Standards
# Hits
# Objs Hit
% Objs Hit
Rng. of Know.
Title
Goals
#
Objs
#
Mean
Mean
Mean
I - Patterns, Relationships
and Functions
2
11
10.44
4.22
38
NO
II - Geometry and
Measurement
3
18
13
5.78
32
NO
III - Data Analysis and
Statisti
3
14
13.44
5
35
NO
IV - Number Sense and
Numeration
3
14
2.78
2.44
17
NO
V - Numerical and
Algebraic Operations
and Analytical ...
2
9
10.67
5.22
55
YES
VI - Probability and
Discrete Mathematics
2
11
6.89
3.67
33
NO
Total
15
78.11
57.22
4.39
35
III
–
Mathematics Standards
V
VI
-P
-N
um
r.
cr
et
e
O
pe
/D
is
Al
g
ro
b
&
s
se
/S
ta
t
is
s
s
/M
ea
rS
en
al
ys
eo
m
/F
cn
te
rn
s
um
be
An
-N
at
a
IV
D
G
Pa
t
II
-
I-
Percent
State B Range of Knowledge
100
80
60
40
20
0
State B
Balance of Representation
High School Language Arts (3 of 12 standards)
(N=116 items)
Balance Index
Standards
% Hits in
Std/Ttl Hits
Bal. of
Represent.
Index
Goals
#
Objs
#
Mean
S.D.
Mean
S.D.
I. - Meaning and
Communication—
Reading
1
5.11
28
8
0.57
0.12
NO
II. - Meaning and
Communication—
Writing
1
4
48
7
0.68
0.14
WEAK
VIII. - Genre and Craft
of Language
1
5
17
6
0.63
0.16
WEAK
Total
12
55.33
8
18
0.36
0.21
Title
Language Arts Standards
State B Balance of Representation
2.4
2.2
2
1.5
1.3
1.1
I.
0
50
100
150
200
250
Number of Hits
300
350
400
What considerations should be
given to different item types?
 How
to consider a multiple-point
assessment item?
 What
is the trade off between
multiple-choice items and openended items?
What are issues related to
vertical alignment?
 Appropriate
progression of
complexity across grades
 Appropriate
progression of
content across grades
Percent of DOK Levels
State A
Mathematics DOK Levels for Objectives
by Grade
100%
80%
DOK Level 4
60%
DOK Level 3
DOK Level 2
40%
DOK Level 1
20%
0%
3
4
5
6
Grade
7
8
10
State A
Reading Language Arts DOK Levels for
Objectives by Grade
Percent of DOK Levels
100%
80%
DOK Level 4
60%
DOK Level 3
DOK Level 2
40%
DOK Level 1
20%
0%
3
4
5
6
Grade
7
8
10
Vertical Alignment Questions

What level of concurrence is there between
objectives for the two grades?

To what extent do comparable objectives
increase in depth from one grade to the next?

To what extent does the range of content
increase from one grade to the next?

How does the balance of representation change
from one grade to the next?
Type of Vertical Relationships





Broader
Deeper
Prerequisite
New
Identical