Transcript Who are the Students in Alternate and Modified Achievement Standards Assessments? Jacqueline F.

Who are the Students in Alternate
and Modified Achievement
Standards Assessments?
Jacqueline F. Kearns, Ed.D., NAAC
Martha Thurlow, Ph.D., NCEO
Elizabeth Towles-Reeves, Ph.D., NAAC
OSEP Project Directors’ Conference
July 22, 2008
Part I
Alternate Achievement
Standards Assessments
Topics
1.
Alternate Achievement Standards Assessments
and the Validity Evaluation
2.
Current Research from NAAC-Who are the
Students in Alternate Achievement Standards
Assessments?
3.
Implications for the Validity Evaluation
The Assessment Triangle & Validity Evaluation
Marion & Pellegrino (2006)
OBSERVATION
Assessment System
Test Development
Administration
Scoring
INTERPRETATION
VALIDITY EVALUATION
Empirical evidence
Theory & logic (argument)
Consequential features
COGNITION
Student Population
Academic content
Theory of Learning
Reporting
Alignment
Item Analysis & DIF/Bias
Measurement error
Scaling and Equating
Standard Setting
Cognition Vertex Validity Questions
1)
2)
Is the assessment appropriate for the students
for whom it was intended?
Is the assessment being administered to the
appropriate students?
Both are important for the validity evaluation
Issues in Teaching/Assessing Students in
Alternate Achievement Standards Assessments










Varied levels of symbolic communication
Attention to salient features of stimuli
Memory
Limited motor response repertoire
Generalization
Self-Regulation
Meta-cognition
Skill Synthesis
Sensory Deficits
Special Health Care Needs

Kleinert, H., Browder, D., Towles-Reeves, E. (in press). Models of cognition for students
with significant cognitive disabilities: Implications for assessment. Review of Educational
Research.
Learner Characteristics
Demographic Variables

Learner Characteristics (all on a continuum of skills):










Expressive Language
Receptive Language
Vision
Hearing
Motor
Engagement
Health Issues/Attendance
Reading
Mathematics
Use of an Augmentative Communication System (dichotomous
variable)
Methodology
 Seven
partner states chose to participate
during the 2006-2007 school year.
 States

gathered data in the administration process for their alternate
achievement standards assessment (i.e., bubble sheet,
paper/pencil version of the LCI, etc.)
 State

1, 2, 3, 4, 5, and 6:
7:
gathered data using Zoomerang, an online survey package.
States & LCI Response Rates
State
Geography Participation Rate Sample N Response Rate
State 1
North East
0.96%
2793
100%
State 2
Mid West
1.17%
2216
100%
State 3
East
1.14%
3595
75%
State 4
North East
0.99%
722
93%
State 5
South East
0.70%
2134
87%
State 6
East
0.76%
468
91%
State 7
West
0.94%
219
47%
IDEA Categorical Distributions
2.00%
2.94% 1.88% 0.75%
1.48%
5.82%
17.73%
Autism
Mental Retardation and Multiple
Disabilities
Autism
1.00%
3.30% 1.00%
Other Health Impairment
Mental Retardation
and
Multiple Disabilities
Emotional Disability
Specific Learning Disability
66.85%
Traumatic Brain Injury
Speech Language Impairment
Orthopedic Impairment
Hearing Impairment
Deafblind
Visual Impairment
Expressive Language
80.00%
70.00%
60.00%
50.00%
Symbolic
Emerging Symbolic
Pre Symbolic
40.00%
30.00%
20.00%
10.00%
0.00%
State 1 State 2a State 3 State 4 State 5 State 6 State 7
Receptive Language
60.00%
50.00%
40.00%
Follows Directions
Requires Cues
Alerts to Input
Uncertain Response
30.00%
20.00%
10.00%
0.00%
State 1 State 2 State 3 State 4 State 5 State 6 State 7
Use of Augmentative
Communication Systems
90.00%
80.00%
70.00%
60.00%
50.00%
Percentage of Students
NOT using AAC
40.00%
30.00%
20.00%
10.00%
0.00%
State State State State State State State
1
2
3
4
5
6
7
Reading
50.00%
45.00%
40.00%
35.00%
Critical Understanding
Basic Understanding
Reads Basic Sight Words
Aware of Text
No Awareness
30.00%
25.00%
20.00%
15.00%
10.00%
5.00%
0.00%
State State State State State State State
1
2
3
4
5
6
7
Mathematics
60.00%
Applies Computational
Procedures
50.00%
Does Computational
Procedures w/wo a
Calculator
1:1 Correspondence
40.00%
30.00%
20.00%
Counts by Rote to 5
10.00%
No Awareness
0.00%
State State State State State State State
1
2
3
4
5
6
7
Expressive Language Across Grade Bands
Elementary School
12.12%
19.08%
65.13%
Middle School
7.58%
Presymbolic
Emerging
Symbolic
Symbolic
72.57%
High
8.68%
72.18%
12.02%
Presymbolic
Emerging
Symbolic
Symbolic
17.03% Presymbolic
Emerging
Symbolic
Symbolic
Reading Across Grade Bands
Elementary School
Middle School
No Awareness
11.72%
1.78%
17.22%
25.34%
No Awareness
20.84%
Aware of Text
2.75% 13.74%
17.82%
Reads Basic Sight
Words
Basic
Understanding
40.06%
40.34%
Critical
Understanding
High School
22.12%
2.15%
No Awareness
15.08%
Aware of Text
11.54%
Reads Basic Sight
Words
Basic Understanding
37.46%
Aware of Text
Critical
Understanding
Reads Basic Sight
Words
Basic
Understanding
Critical
Understanding
Mathematics Across Grade Bands
Elementary
Middle
No Awareness
No Awareness
3.98%
3.80%
Counts by rote to 5
15.28%
12.74%
7.98%
11.94%
1:1 Correspondence
21.98%
30.26%
34.24%
Does Computational
Procedures w/wo a
calculator
48.04%
High
6.40%
No Awareness
4.90%
Counts by rote to 5
17.72%
1:1 Correspondence
Does Computational
Procedures w/wo a calculator
45.38%
1:1 Correspondence
Does Computational
Procedures w/wo a
calculator
Applies Computational
Procedures
Applies Computational
Procedures
12.54%
Counts by rote to 5
Applies Computational
Procedures
Who are the Kids in Alternate Achievement
Standards Assessments?








Represent ~1% or less of the total assessed population
All disability categories were represented but primarily 3 emerge,
 Mental Retardation
 Multiple Disabilities
 Autism
Highly varied levels of expressive/receptive language use
Most students in the population use symbolic communication
Level of symbolic language use does not significantly change across gradebands
The majority of students do not use AAC
Most of the population read basic sight words and solve simple math
problems with a calculator.
Changes in skill progression in reading and math across grade bands most
likely due to identification of students rather than teaching and learning
Cognition Vertex:
Validity Evaluation Essential Questions





Who is the population being assessed?
How do we document and monitor the population?
What do we know about how they learn (theory of learning)
academic content?
What do our assessment results tell us about how the
population is learning academic content?
Are our data about the population and theory of learning
consistent with student performances on the assessment?


If not, what assumptions are challenged?
What adjustments should be made?



Participation
Theory of Learning
Student Performance
Alternate Achievement Standards Assessments

References



Kleinert, H., Browder, D., Towles-Reeves, E. (in
press). Models of cognition for students with significant
cognitive disabilities: Implications for assessment. Review of
Educational Research.
Marion, S., & Pellegrino, J. (2006). A validity framework for
evaluating the technical quality of alternate assessments.
Educational Measurement: Issues and Practices, 25(4), 47-57.
Additional Resource

Towles-Reeves, E., Kearns, J., Kleinert, H., & Kleinert, J.
(2008, May 12). An analysis of the learning characteristics of
students taking alternate assessments based on alternate
achievement standards. Journal of Special Education. Retrieved
June 2, 2008, from
http://sed.sagepub.com/cgi/rapidpdf/0022466907313451v1.
Part II
Modified Achievement
Standards Assessments
Topics
1. Research and Regulation Advice
2. Current Practice
3. GSEG Project Work
Why Start from the Student?
National Research Council – Pellegrino, Chudowsky,
& Glaser, (2001) – Knowing What Students Know
Luecht (2007) - good assessment design would apply
human factors engineering principles by developing
cognitive maps and cognitive construct models
Pellegrino (2007) - Principled Assessment Design
process that started with a clear student model as the
basis for an evidence model that would, in turn, serve
as the basis for a task model.
Why Start from the Student?
Mislevy and Haertel (2007) - central point of
agreement of these models is the necessity of
first developing a good understanding of how
people do or fail to do what is to be measured.
Then tasks can be developed that let us
observe what people do so we are able to
make inferences that are more fully supported
by clearer evidence.
Assessment as a Process of
Reasoning from Evidence
Cognition
The Assessment Triangle
– model of how students represent
knowledge & develop
observation
competence in the domain
interpretation
Observations
– tasks or situations that allow one
to observe students’ performance
Interpretation
– method for making sense of the
data
cognition
Must be coordinated!
Cognition Vertex Validity Questions
1) Is the assessment appropriate for the
students for whom it was intended?
2) Is the assessment being administered to
the appropriate students?
Both are important for the validity evaluation
April 9, 2007 Regulations Address Who
the Students Are
Preamble: The final regulations intentionally do
not prescribe which students with disabilities are
eligible to be assessed based on modified
academic achievement standards; that is the
determination of a student’s IEP Team, which
includes the student’s parents, based on criteria
developed by the State as part of the State’s
guidelines for IEP Teams
April 9, 2007 Regulations Address Who
the Students Are
The student’s progress to date in response to
appropriate instruction . . ., is such that, even
if significant growth occurs, the IEP team is
reasonably certain that the student will not
achieve grade-level proficiency within the year
covered by the student’s IEP.
Section 200.1(e)(2)(ii)
April 9, 2007 Regulations Address Who
the Students Are
Inform IEP teams that students eligible to be
assessed based on alternate or modified
academic achievement standards may be
from any of the disability categories listed in
the IDEA.
Section 200.1(e)(2)(ii)
From Cortiella (2007), Learning Opportunities for Your Child Through
Alternate Assessments – Alternate Assessments based on Modified Achievement
Standards
Current Practice
• In 2007, 5 states had assessments that they
believed to be an AA-MAS before the April,
2007 regulation release – see Lazarus,
Thurlow, Christensen, & Cormier (2007)
• Update study now being conducted – have
pulled information on eligibility for AAMAS
Eligibility Criteria in States
In 2008, 10 states had assessments that they
believed to be an AA-MAS
California
North Carolina
Connecticut
North Dakota
Kansas
Oklahoma
Louisiana
Texas
Maryland
Virginia
2008 Study
Number of States With Selected Eligibility Criteria
Format of Criteria
Descriptions (bullets, chart, written description)
# of
States
10
Flowchart/Decision Tree
4
Checklist
5
2008 Study
Number of States With Selected Eligibility Criteria
Disability Status Criteria
Student has IEP
# of
States
10
Not based on disability category label
5
Not due only to ELL designation or being on 504 plan
3
Not due to being identified as having a significant
cognitive disability
2
Student may be in any of the disability categories
6
2008 Study
Number of States With Selected Eligibility Criteria
Classroom Learning/Teaching Criteria
# of
States
Student not progressing at rate expected to reach grade
level proficiency within school year covered by IEP
8
Student is learning grade-level content
6
Not due to receiving instruction based on extended or
alternate standards or being eligible to take AA-AAS
6
Not based on attendance (extended/excessive absence)
5
Student receives specialized instruction
3
2008 Study
Number of States With Selected Eligibility Criteria
Classroom Learning/Teaching Criteria – cont.
# of
States
Student requires differentiated content for classroom
assessment
3
Student needs accommodations during classroom
instruction
2
Student’s classroom achievement and performance
significantly below grade-level peers
2
Student consistently requires instruction in prerequisite skills to the grade-level indicators
2
2008 Study
Number of States With Selected Eligibility Criteria
Classroom Learning/Teaching Criteria – cont.
# of
States
Not based on placement setting
2
Not based on amount of time in general or special
education services
2
2008 Study
Number of States With Selected Eligibility Criteria
Previous Performance Criteria
# of
States
Student passed or failed AA-AAS or other large-scale
tests
5
Student cannot demonstrate knowledge on regular
assessment even with accommodations
3
Student has been tested on multiple, valid, objective
measures over time
3
Student’s previous performance on multiple measures
is considered
3
2008 Study
Number of States With Selected Eligibility Criteria
IEP Goals & Other
# of
States
IEP includes goals that are based on grade-level
content standards
4
Not due to social, cultural, language, economic, or
environmental factors
2
Identifying Accommodations for AA-MAS
• State Approaches
• Accommodations issues for regular
assessment
• Integration of “accommodations” and
universal design principles into the regular
assessment first, then in the design of the
AA-MAS
State Approaches – Accommodations Incorporated into
AA-MAS Design
Accommodation
No. of States
Fewer items/page
3
Larger font size
3
Calculator
2
Breaks as needed
2
Key text underlined/bolded
1
AA-MAS Participation Rates (2006-07)
Rates based on # students with IEPs
State 1 (long time)
Reading = 19%
Math = 17%
State 2 (long time)
Reading = 10.5%
Math = 9.9%
State 3 (long time)
Reading = 23%
Math = 21%
State 4 (newer)
Reading = 31%
Math = 29%
AA-MAS Rates Proficient (2006-07)
Rates based on # students with IEPs
State 1 (long time)
Reading = 24%
Math = 33%
State 2 (long time)
Reading = 4.3%
Math = 2.6%
State 3 (long time)
No Data
No Data
State 4 (newer)
Reading = 52%
Math = 54%
Considerations

Alternate Achievement Standards Assessments:
Consideration of students with high reading and
math abilities
 Assessment design for a highly varied population
 Considering symbolic language use
 Skill progressions in reading and math

Considerations

Modified Achievement Standards Assessments:
Moving from student characteristics to an assessment
based on grade-level content, but with modified
achievement standards
 Clearly defining the relationships among the general
assessment, the AA-AAS, the AA-MAS, and the AAGLAS, if one exists
 Separating instructional issues from assessment issues
 Providing training and assistance for good decisions
about who needs which assessment

Contact Information
Jacqueline Kearns, Ed.D.





1 Quality Street, Suite 722
Lexington, Kentucky 40507
859-257-7672
859-323-1838
[email protected]
Elizabeth Towles-Reeves, Ph.D.





1 Quality Street, Suite 722
Lexington, Kentucky 40507
859-257-7672
859-323-1838
[email protected]
http://www.naacpartners.org/
Contact Information
Martha Thurlow, Ph.D.





207 Pattee Hall
150 Pillsbury Drive SE
Minneapolis, MN 55455
612-624-4826
[email protected]
www.nceo.info