Transcript No Slide Title

OPPORTUNITY TO LEARN AND ASSESSMENT OF
ENGLISH LANGUAGE LEARNERS
CRESST Conference, September 2005
Jamal Abedi
University of California, Davis
National Center for Research on Evaluation,
Standards, and Student Testing
September 8, 2005
Interaction between language
background factors and instruction
and assessment for ELL students
Complex linguistic structure of instructional
materials and assessment questions is
challenging for English language learners
To have a better understanding of factors
affecting instruction and assessment of ELL
students, the impact of linguistic factors must
be examined
Results of CRESST studies suggest that due
to many factors including language, ELL
students may not have the same level of
opportunity to learn as their non-ELL peers
Interaction between language background
factors and instruction and assessment for
ELL students
 Results of our studies on the assessment of
ELL students suggest that the linguistic
complexity of assessment, as a source of
construct-irrelevant variance, influence the
validity of assessment for ELL students
 Similarly, unnecessary linguistic complexity
of assessment may be a source of systematic
error of measurement in the assessment of
ELL student
 Therefore, assessments that are designed
for non-ELL students may not provide valid
outcomes for these students
Do ELL students have the same level of
opportunity to learn as non-ELL students?
CRESST Studies on OTL for ELL and nonELL student
 Two major studies:
 Study A: The level of OTL was
compared across ELL classification
(ELL/non-ELL) using a sample of over
600 grade 8 students
 Study B: OTL was compared across the
ELL subgroups and the impact of OTL
was examined on the accommodated
and non-accommodated assessments
on a group of 2,500 grade 8 students
Do ELL students have the same level of
opportunity to learn as non-ELL students?
 Methodology
 Accommodations were randomly assigned
to both ELL and non-ELL students
 Random assignment of accommodation
within intact classrooms controlled for
teacher, class, and school effects
 Additionally, a short test of English
language proficiency was administered
and was used as a covariate
 Standardized test scores were obtained
 Student’s level of opportunity to learn was
examined using questionnaire and
Do ELL students have the same
level of opportunity to learn
(OTL) as non-ELL students?
 Summary of findings:
 ELL students systematically
reported less OTL in response to
OTL questions
 Observers found major differences
between ELL and non-ELL students
with respect to OTL
 ELL students had less chance to
participate in classroom activities
Issues with LEP Assessment,
Accommodation, and AYP Reporting
1. Problems in Classification/
Reclassification of LEP Students
 LEP classification criteria varies
 LEP criteria discrepancies cause
inconsistencies nationwide
 Accuracy of AYP reporting for LEP
students suffers
Issues with LEP Assessment,
Accommodation, and AYP Reporting
2. Measurement quality of AYP instruments for LEP
students

Students’ yearly progress is measured by
performance on academic achievement tests

Academic achievement tests are constructed and
normed for native English speakers

These tests have lower reliability and validity for
LEP populations

Therefore, test results should not be interpreted
for LEP students as for non-LEP students
Issues with LEP Assessment,
Accommodation, and AYP Reporting
3. Low LEP Baseline Scores
 Schools with high numbers of LEP
students have lower baseline scores
 For those schools, year-to-year progress
goals are much more challenging and
might be considered unrealistic
 Their students may continue to struggle
with the same academic disadvantages
and limited school resources
Issues with LEP Assessment,
Accommodation, and AYP Reporting
4. Instability of LEP Subgroup

A student’s LEP status is not stable over time, it is a
moving target

When a student improves to “proficient,” the
student leaves the LEP subgroup

Those who remain are low performing

New arrivals with even lower levels of language
proficiency may join the subgroup

Even with the best resources, the LEP subgroup AYP
indicator has little chance of rising
Issues with LEP Assessment,
Accommodation, and AYP Reporting
5. Sparse LEP Population

The number of LEP students varies across the
nation

A minimum of 25 is needed for reliable AYP
reporting (Linn, Baker, & Herman, 2002)

To detect moderate changes, several hundred
subjects may be needed (Hill & DePascale, 2003)

In many states and districts, the number of LEP
students is not enough for any meaningful
analyses

This might skew some states’ accountability
and adversely affect state and federal policy
decisions
Issues with LEP Assessment,
Accommodation, and AYP Reporting
6. LEP Cutoff Points (Conjunctive vs.
Compensatory Model)

Improving America’s Schools Act (IASA), adopted
a Compensatory model

In a Compensatory model, higher scores in
content areas with less language demand
compensate for scores in areas with higher
language demand

NCLB AYP reporting is based on a conjunctive
model in which students should score at a
“proficient” level in all required content areas

This makes all the AYP requirements more
difficult for schools with many LEP students
Accommodations for LEP
Accommodations that are
appropriate for the particular
subgroup should be used
Why Should English Language
Learners be Accommodated?
Their possible English language
deficiency may interfere with their
content knowledge performance
Assessment tools may be culturally and
linguistically biased for these students
Linguistic complexity of the assessment
tools may be a source of measurement
error
Language factors may be a source of
construct irrelevant variance
Accommodations for
ELLs
Can the same accommodations used
for students with disabilities be used
for ELLs?
What is the purpose and objective of
accommodations for ELLs?
Accommodation in Stanford 9 Testing
Used by a District
SD
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
LEP
l
l
l
l
l
l
l
l
l
l
l
Accommodation
1. Extension of allotted time
2. Use of multiple shortened test periods
3. Simplification of directions
4. Reading of Questions (Math and Science only)
5. Translation of words and phrases on the spot
6. Decoding of words upon request
7. Use of calculator
8. Gestures and nonverbal expression
9. Allow students to mark answers in test booklet
10. Allow students to point to response
11. Allow student to answer orally
12. Allow student to respond on audio tape
13. Use of typewriter or personal computer
14. Use of graphic organizers & art work
15. Use of large print or magnification device
16. Testing in separate room
17. Small group setting
18. Use of study carrel
SY 2000-2001 Accommodations
Designated for ELLs Cited in
States’ Policies
There are 73 accommodations listed:
N: Not Related
R: Remotely Related
M: Moderately Related
H: Highly Related
From: Rivera (2003) State assessment policies for English language
learners. Presented at the 2003 Large-Scale Assessment Conference
SY 2000-2001 Accommodations
Designated for ELLs Cited in States’
Policies
I. Timing/Scheduling (N = 5)
N 1. Test time increased
N 2. Breaks provided
N 3. Test schedule extended
N 4. Subtests flexibly scheduled
N 5. Test administered at time of day
most beneficial to test-taker
N = not related; R = remotely related; M = moderately related; H = highly related
II.
SY 2000-2001 Accommodations
Designated for ELLs Cited in States’
Setting (N = 17)
Policies
N 1. Test individually administered
N 10. Teacher faces test-taker
N 2. Test administered in small group
N 11. Special/appropriate lighting
provided
N 3. Test administered in location with
minimal distraction
N 4. Test administered in familiar room
N 5. Test-taker in separate location (or
carrel)
N 12. Adaptive or special furniture
provided
N 13. Adaptive pencils provided
N 14. Adaptive keyboards provided
N 6. Test administered in ESL/Bilingual
classroom
N 15. Person familiar with test-taker
administers test
N 7. Individual administration provided
outside school (home, hospital,
institution, etc.)
N 16. ESL/bilingual teacher administers
test
N 8. Test-taker provided preferential seating
N 9. Increased or decreased opportunity for
movement provided
N 17. Additional one-to-one support
provided during test administration
in general education classroom (e.g.
instructional assistant, special test
administrator, LEP staff, etc.)
N = not related; R = remotely related; M = moderately related; H = highly related
III.
SY 2000-2001 Accommodations
Designated for ELLs Cited in States’
Policies
Presentation (N = 32)
R 1. Directions repeated in English
R 2. Directions read aloud
R 3. Audio-taped directions provided in
English
N 4. Key words or phrases highlighted
M 5. Directions simplified
M 6. Audio-taped directions provided in
native language
M 7. Directions translated into native
language
N 8. Cues provided to help test-taker remain
on task
M 9. Directions explained/clarified in English
M 10. Directions explained/clarified in
native language
M 11. Both oral and written directions in
English provided
M 12. Both oral and written directions in
native language provided
M 13. Test items read aloud in English
H 14. Test items read aloud in
simplified/sheltered English
N 15. Audio-taped test items provided in
English
H 16. Test items read aloud in native
language
H 17. Audio-taped test items provided in
native language
N = not related; R = remotely related; M = moderately related; H = highly related
III.
SY 2000-2001 Accommodations
Designated for ELLs Cited in States’
Policies
Presentation (N = 32)
N 18. Assistive listening devices,
amplifications, noise buffers,
appropriate acoustics provided
N 19. Key words and phrases in test
highlighted
H 20. Words on test clarified (e.g. words
defined, explained)
H 21. Bilingual word lists, customized
dictionaries (word-to-word translations)
provided
N 22. Enlarged print, magnifying equipment,
Braille provided
N 23. Memory aids, fact charts, list of
formulas and/or research sheets
provided
N 24. Templates, masks, or markers
provided
N 25. Cues (e.g. arrows and stop signs)
provided on answer form
N 26. Acetate shield for page provided
N 27. Colored stickers or highlighters for
visual cues provided
R 28. Augmentive communication systems or
strategies provided (e.g. letter boards,
picture communication devices, voice
output systems, electronic devices)
H 29. Simplified/sheltered English version of
test provided
H 30. Side-by-side bilingual versions of test
provided
H 31. Translated version of the test provided
N 32. Test interpreted for the deaf or hearing
impaired/use of sign language provided
N = not related; R = remotely related; M = moderately related; H = highly related
IV.
SY 2000-2001 Accommodations
Designated for ELLs Cited in States’
Policies
Response (N = 17)
N 1. Test-taker marks answers in test
booklet
N 2. Test administrator transfers test-taker’s
answers
N 3. Test-taker’s transferred responses
checked for accurate marking
N 4. Copying assistance provided between drafts
N 5. Test-taker types or uses a machine to
respond (e.g. typewriter/word
processor/computer)
N 6. Test-taker indicates answers by pointing
or other method
N 7. Papers secured to work area with
tape/magnets
N 8. Mounting systems, slant boards, easels
provided to change position of paper,
alter test-taker’s position
N 10. Enlarged answer sheets provided
R 11. Alternative writing systems provided
(including portable writing devices,
computers and voice-activated
technology)
R 12. Test-taker verifies understanding of
directions
R 13. Test-taker dictates or uses a scribe to
respond in English
N 14. Test-taker responds on audio tape in
English
H 15. Oral response in native language
translated into English
H 16. Written response in native language
translated into English
H 17. Spelling assistance, spelling dictionaries,
spell/grammar checker provided
N. 9. Physical assistance provided
N = not related; R = remotely related; M = moderately related; H = highly related
SY 2000-2001 Accommodations
Designated for ELLs Cited in States’
Policies
V. Other (N = 2)
N 1. Out-of-level testing provided
N 2. Special test preparation provided
Preliminary Findings: State Assessment Policies for English Language Learners, SY 2000-2001
GW/CEEE, Large-Scale Assessment Conference 2003
[email protected]
N = not related; R = remotely related; M = moderately related; H = highly related
There are 73
Accommodations Listed
47 or 64% are not related
7 or 10% are remotely related
8 or 11% are moderately related
11 or 15% are highly related
Issues with LEP Opportunity
to Learn
 The performance-gap between ELL
and non-ELL students can be partly
explained by differential level of
OTL
 Complex linguistic structure of
instructional materials may be a
factor in differential level of OTL
for ELL and non-ELL students
 To explain performance-gap
between ELL and non-ELL students,
their instructional opportunities
must be carefully examined.
Issues with LEP Assessment and
Accommodation
 Encourage and support research on the
instruction of ELL students, particularly on
opportunity to learn
 Identify instructional strategies beneficial for
ELL students; address language acquisition
needs
 Encourage and support more research on the
validity of large-scale, high-stakes assessments
 Collect longitudinal national data on the
assessment and accommodation of LEP students
that can help future curriculum and
assessment planning