Universally Designed English Proficiency and Content Area Assessments

Improving the Achievement of English Language Learners through Authentic Proficiency Assessments May, 2004

Training Goals

• • •

Determine why we need universally designed assessments – and for whom Identify and give examples of essential elements of universally designed assessments Identify and apply considerations for item review to sample test items

A Quick Definition

Universally designed assessments are built from the beginning and continually refined to be accessible and valid for the greatest number of students, including English language learners!

But, what does that really mean?

• • • •

Do we want to change the standard of performance?

NO Can we forget about accommodations if we do this?

NO Is this all figured out – for now and forever?

NO Is this something that will benefit only English language learners?

NO

Think about universal design in architecture and tool design



Curb cuts and ramps



Signage with universal symbols



Door handles rather than knobs



Special pen shapes that are easier to hold

The goal is to provide optimal standard assessment conditions for today’s diverse population of students

Remember this?

OFFICIAL BALLOT, PALM BEACH COUNTY, FLORIDA

Analysis of Florida Votes

Precincts with primarily Hispanic, African American, or elderly voters had more than 3 times as many rejected ballots as other precincts.

(New York Times, 11/12/01)

Design is important in a lot of things – including assessments!

http://go.to/funpic

Assessments Discussed in this Training

• English Language Proficiency Listening/ Speaking Test

Computer Based

• English Language Proficiency Reading and Writing Test -

Paper/Pencil

• Grade level Academic Content Area Assessment

Computer Based

• Grade level Academic Content Area Assessment

Paper/Pencil

Universally Designed Assessments Can Increase Reliability and Validity of Assessments for English Language Learners



Analyses of large data files (Abedi, 2002) found language as a source of measurement error that can negatively impact reliability of test results for ELLs.



Language has also been found to be a source of construct irrelevance that can have a negative impact on the validity of the results for ELLs (Abedi, 2002).

Universally Designed Assessments Increase Usability for the End Users – STUDENTS!

•

In the past, emphasis has been placed on developing test items without consideration of the students who are the ultimate end users

•

The students who are tested have changed and are no longer “even remotely comparable to the designer in skill set, aptitude, expectation, or in almost any attribute that is relevant to the design process” (Rubin, 1994)

Student Characteristics

Just one of many reasons that we need to be talking about universally designed assessments!

Complete these well-known phrases: 1. A penny saved is . . . 2. An idle mind is . . .

3. Don’t bite the hand that . . . 4. It’s always darkest before . . .

5. Strike while the . . .

6. Two’s company, three’s . . .

7. If at first you don’t succeed, . . .

A penny saved is . . .

. . . not much.

An idle mind is . . .

. . . the best way to relax.

Don’t bite the hand that . . .

. . . looks dirty.

It’s always darkest before . . .

. . . Daylight Savings Time.

Strike while the . . .

. . . bug is close.

Two’s company, three’s . . .

. . . the Musketeers.

If at first you don’t succeed, . . .

. . . get batteries.

What seems obvious and clear to test developers may not be all that obvious and clear to students.

It takes consistent effort and guidance to make sure that test items and tests really are accessible to all students .

This is a BIG DEAL!

Requirements for universally designed assessments are showing up in federal legislation

Title I Regulations introduced the need for universally designed assessments –

[Assessments must be] designed to be accessible and valid with respect to the widest possible range of students, including students with disabilities and students with limited English proficiency. Sec. 200.2(b)(2)

Increasingly, we need to think about students with a variety of different language and cultural background characteristics

The number of English Language Learners in schools has increased dramatically

•

Nearly 4 million LEP students in grades K-12 in 2001-2002 (71.9% increase from 1991-1992)

•

Represent about 8.4% of all public school students in the U.S.

•

Enrolled in almost half of all public schools nationwide

The number of ELLs with special needs is increasing too!

• Most recent estimate is 9% of the total ELL population is in Special Education • Estimated at about 357,000 students • More than 50% of ELLs with disabilities classified as learning disabled

The bill for the federal reauthorization of the Individuals with Disabilities Education Act (IDEA) also includes universal design – UNIVERSAL DESIGN- The State educational agency (or, in the case of a district-wide assessment, the local educational agency) shall, to the extent possible, use universal design principles in developing and administering any assessments….

Sec 612, Senate Bill 1248

So, when we think about all students , we have to think about students who have a variety of cultural, physical, sensory, and processing characteristics.

Caution

While universally designed assessments can make tests more equitable, producing results that are more valid for all students, they cannot replace instructional opportunity!

Elements of Universally Designed Assessments



Inclusive assessment population



Measures what it intends to measure



Respects the diversity of the assessment population



Clear format for text

Elements of Universally Designed Assessments



Clear pictures and graphics



Concise and readable text



Amenable to accommodations

Inclusive Assessment Population



Universally designed assessments:

 Consider all types of students in the general curriculum from the beginning  Include English language learners and students with disabilities in item tryouts and field testing

Example of Field Test Specifications

• The statewide field test will include students with disabilities and English Language Learners using the accommodations they would normally use in the large-scale assessment.

• There may be need for over sampling of these populations to ensure a valid field test.

Measures What it Intends to Measure 

Universally designed assessments reflect good measurement qualities:



Reflects the intended content standard (reviewers have information about what is being measured)



Minimize skills required beyond those being measured

Suppose a test item requires a student to read an Aesop’s fable What could this item measure?

– Decode text?

– Comprehend extended passages?

– Extent of vocabulary?

– Understand the moral or point of the fable?

– Discuss the common elements of any fable?

– Compare and contrast fables with news reports?

– Articulate the relationship between the fable and the overall culture? – Anything else? (National Center on Accessing the Curriculum, 2003)

Suggestions by Researchers . . .

• •

Popham (2001) suggests creating an assessment description for each item that concretely describes what is being tested. Chiu & Pearson (1999) suggest determining non-construct behaviors needed to complete task.

Respects the Diversity of the Assessment Population



Accessible to test takers (consider age, gender, ethnicity, disability, socio-economic level)



Avoid content that might unfairly advantage or disadvantage any student subgroup

According to the National Research Council (1999), bias arises when:

“Deficiencies in the test itself result in different meanings for scores earned by members of different identifiable subgroups.”

A.

Consider the Effect of Cultural Bias B.

If you measure from the nose to the tail, how much longer is lizard B than lizard A? Use the ruler provided in your testing kit. Give the answer in centimeters.

To raise money for a trip to the Wolfridge Environmental Learning Center, sixth graders at Johnson Middle School are selling raffle tickets . The raffle prize is an electric scooter worth more than $300. A total of 500 tickets were sold. You bought two raffle tickets, your sister bought three and your father bought one. What is the probability that someone in your family will win the prize?

Clear Format for Text



Universally designed assessments use text that can be read quickly and effortlessly

      

Standard typeface Type size at least 12 point Wide spacing High contrast Sufficient leading (Spacing) between lines of text Staggered right margins (no right justification) No background graphics behind text

Typeface: Standard typeface, with upper and lower case letters, is better than italic, small caps, or all caps

Adapted from: Making Text Legible by A. Arditi

Type Size

All type should be at least 12 point (including captions, keys, labels, and footnotes).

Remember that point size differs among typefaces.

Adapted from: Making Text Legible by A. Arditi

Spacing

: Space between letters and between words is wide

Adapted from: Making Text Legible by A. Arditi

Contrast

Text should be printed with the highest possible contrast.

Adapted from: Making Text Legible by A. Arditi

Leading: Spacing Between Lines of Text

Adapted from: Making Text Legible by A. Arditi

Staggered Right Margins Text is easier to read when it is justified on the left and unjustified on the right. This prevents pockets of spaces.

Text is easier to read when it is justified on the left and unjustified on the right. This prevents pockets of spaces.

(APH, 2002) Used with permission

Text with background graphics or other media is difficult to read.

This is a picture of a starfish in side a vortex. Of course to you it might look like a giant flower or an octopus in a whirlpool. You might even think it is op art or a computer rendering of a bad dream. However this photo is to demonstrate to you how difficult it is to read text when it is placed over a busy background.

(APH, 2002) Used with Permission

Clear Pictures and Graphics

 Clear, non-fuzzy pictures  Dark lines (minimum use of gray scale and shading)  Sufficient contrast between colors  Color is not relied on to convey important information or distinctions  Label pictures and graphics when possible



Universally designed assessments use non-text materials just as carefully as text materials



Only essential illustrations are used (referred to in text and necessary to answer question)



Illustrations for interest may draw attention away from content being assessed

Is this border distracting? Is it necessary?

Fence or Bridge or What?

fence Add context and labels to help identify pictures

Is the swimmer at the bottom of the page distracting?

Is this a better way to remind test takers to continue?

Concise and Readable Text

 Commonly used words    Vocabulary appropriate for grade level Minimum use of unnecessary words  Idioms avoided unless idiomatic speech is being measured  Technical terms and abbreviations avoided (or defined) if not related to the content being measured  Sentence complexity is appropriate for grade level Question to be answered is clearly identifiable

Clear Instructions

•

Ensure “that test takers can respond to a task in the manner that the test developer intended” (AERA, APA, NCME, 1999).

•

“While writers might think certain expectations are obvious, if they are not explicit in the item, then they are subject to honest misinterpretation in the responses” (Kopriva, 2000, p. 39).

Assessments designed to better include English language learners benefit all types of students!

• • • •

Students have the experience to understand the items Language is clear, simple and indicates precisely what is required from student (“Plain language”) Questions are amenable to supports that ELLs might use Cognitive demands are reasonable

Abedi research suggests that linguistic complexity of test items is a significant source of measurement error for English language learners (and students with disabilities) Examples of Linguistic Modifications: Familiarity or frequency of non-math vocabulary (unfamiliar or infrequent words changed): Tetris Video game Length of nominals (long nominals shortened): Last year’s class vice president The vice president (Abedi, et al., 2001)

Linguistic Modifications (continued): Question phrases (complex question phrases changed to simple question words): At which of the following times When Voice of the verb phrase (passive verb forms changed to active): If a marble is taken from the bag If you take a marble from the bag Relative clauses (relative clauses removed or recast): A report that contains 64 sheets of paper He needs 64 sheets of paper for each report

Linguistic Modifications (continued): Conditional clauses (conditionals either replaced with separate sentences or order of conditional and main clause changed): If Lee delivers x newspapers Lee delivers x newspapers Abedi found that English language learners who received a linguistically modified version of the math test performed significantly better than those receiving the original test items. They also performed significantly faster.

Over 80% of the students also told Abedi they preferred the modified math test because

: • •

It’s easier to read, and it gets to the point, so you won’t have to waste time It’s shorter and doesn’t have, like, complicated words

•

I might have a faster time completing that one ‘cuz there’s less reading’

(Abedi, et al., 2001)

Sample Readability Guidelines  Use simple, clear, commonly used words, eliminating any unnecessary words (Abedi, 2002)  Clearly define any technical terms that are used  Break compound complex sentences into several short sentences. State the most important ideas first (Rakow & Gee, 1987)  Introduce one idea, fact, or process at a time (Rakow & Gee, 1987)

Recommendations to Improve Accessibility of Text

• • • •

Simple, brief and consistent sentence structure in items Consistent and clear paragraph structure Present tense and active voice Minimal paraphrasing and rewording. If used, identify the original statement in parentheses

From Kopriva (2000)

Recommendations to Improve Accessibility of Text

• • •

Minimal use of pronouns. Follow a pronoun with the term it refers to in parentheses High frequency words Avoid words with double meanings and colloquialisms. If used, define them in the text.

From Kopriva (2000)

Example Reading Passage Newspaper Excerpt

Para. 1 “When Nicole Zachor, Laura Swanson and Carol Hinz started work on a project for history class a few months ago, the White Bear Lake sophomores had no idea that it would be displayed at the Smithsonian Institution’s National Museum of American History…” Para. 12 “ This year

junior and senior high students

started work on their projects in January or February. A project can be a research paper but it can also be a group of individual media presentation, display presentation or performance”

Side-by-Side Translation

Which of the following is a condition for a student to participate in National History Day?

A.

The student must be a junior or senior school. in high B.

The student must be able to go to Washington, D.C.

C. The student must do a project related to the national topic.

D. The student must do the project on his or her own by himself or herself.

¿Cuál de las siguientes es una condicíon para que un estudiante participe en el Día Nacional de Historia?

A. Los estudiantes deben estar en grados once o doce de High School. B. El estudiante debe estar en posibilidad de ir a Washington, D.C

C. El estudiante debe hacer un proyecto relacionado a un tema nacional.

D. El estudiante debe hacer el proyecto por sí solo

.

Ordering Pizza (Original Item) The cafeteria manager surveyed the students in a middle school to find out if they would buy Brand X pizza on Friday if the manager sold it. She made a circle graph to display the results of her survey.

NO YES

Based on the results of the survey, answer the following questions: 1. What fraction of students would buy Brand X pizza on Friday?

2. What percent of students would buy Brand X pizza on Friday?

3. There are 1200 students in this school. How many students will buy Brand X pizza on Friday if the manager’s survey is accurate?

Ordering Pizza (Revised Item) Maria surveyed the students in her school to find out if they would like pizza on Friday. She made a circle graph to display the results of her survey.

YES NO

1. What fraction of students said “yes”?

2. What percent of students said “yes”?

3.

There are 1200 students in Maria’s school. How many students said “yes”?

(From Hanson, et al., 1998)

Cognitive Demands

• • • • • • •

Amount of text not relevant to items Length of text Number of long texts Timing (may be unspoken) Number of unfamiliar words Placement of definitions (in text, to side, separate) – if allowed Location of native language text – if provided

Amenable to Accommodations

 Universally designed assessments allow needed accommodations to be used  Plan for students who continue to need accommodations  Facilitate the use of accommodations such as bilingual dictionaries or translations (when appropriate), assistive technology, oral presentation, large print, Braille

What accommodations might be difficult with this item?

Could this item be revised without knowing what it is intended to measure?

Here is a revised graphic that would clarify the item.

Items and revision used by permission from the state of NH and DRC



Universally designed assessments consider the design of the response venue as well as the assessment itself



Large bubbles that avoid most challenges of low vision or difficulty with fine motor skills (Grise, Beattie, Algozzine, 1982)



Consideration of age of students in selecting format (avoid separate answer sheets for younger students) (Tindal and others suggest 4th grade as transition point).

Clear Response Grid

Hard to see Gray columns changed to yellow.

Larger bubbles.

Easier to see.

(APH, 2003) Used with permission

Computer-Based Testing All of the Elements of Universal Design Plus Some Additional Opportunities and Challenges

Opportunities

• Efficient administration • Preferred by students • Improved writing performance • Built-in accommodations • Immediate results • Efficient item development • Increased authenticity

Challenges

• Use of technology cannot take the place of content mastery • Issues of equity and skill in computer use • Added challenges for some students • Technological challenges • Security of online data • Lack of expertise in designing accessible Web pages • Prohibitive development cost

Example: Kentucky Online Assessment

• Web-based, individualized assessment : – Students with IEP or 504 Plan that specifies need for "reader" as an instructional and assessment accommodation; – Students who require and routinely use text-reader or screen-reader technologies to access printed material in classroom instruction and assessment; – Students who have accessed and used the CATS Online Practice Area.

• Based upon success of pilot studies, 16 districts, 31 Schools, & 204 students participated in “live” CATS Online in the spring of 2003

CATS Online

How to Develop Universally Designed Assessments

• • •

Develop frameworks for alignment of items with content Include elements of universally designed assessments in test specifications Develop guidelines for item developers to follow

How to Develop Universally Designed Assessments

Flag items with bias and design issues

: • Consult with end users – students representing all subgroups • • Consult with bias or sensitivity review teams – representing all subgroups Analyze field test data and statewide test results by item and subgroup

What Reviewers Need to Know

• Purpose of the test • Standard tested by each item • Description of test takers (e.g. age, geographic region) • Test instructions • Overall test and response formats • Use of technology • Field test results by item and subgroup (when completed)

Review Team Members

• Grade-level content experts • Representatives of major cultural groups • Representatives of major disability groups • Second language acquisition experts • Who else?

Remember!

• This is a work in progress • Considerations put important issues on the table • Ongoing input is important!

Item Review Materials

• • • •

Considerations and notes Research base for considerations References Considerations form for item reviewers

•

Sample revised items

Visit: education.umn.edu/nceo

or Search for

NCEO Click on “Universal Design”