Transcript Purpose of Bias and Sensitivity Review

PARCC Bias and
Sensitivity Review
August 2012
Coral Gables
Confidential and Proprietary.
1
Points of Discussion
 Purpose of Bias and Sensitivity Reviews
 Identifying Bias and Stereotypes
 Sensitivity Awareness and
Detecting Passage Bias
 Guiding Questions and Sample Items
Confidential and Proprietary.
2
Purpose of Bias and
Sensitivity Review
 Review test materials for potential
sources of bias and stereotypes.
 Apply professional test development
standards to ensure materials are fair and
not insensitive or offensive.
Confidential and Proprietary.
3
What Is Bias?
 Language or content that prevents
members of a group from demonstrating
they possess the knowledge and skills
being measured.
 Language or content that advantages
members of a group in demonstrating they
possess the knowledge and skills being
measured.
Confidential and Proprietary.
4
Common Forms of Bias
and Stereotypes
 Regional and geographic bias
 Gender and age stereotypes
 Ethnic, cultural, and religious stereotypes
 Socioeconomic and occupational
stereotypes
Confidential and Proprietary.
5
Guideline #1
Avoid Cognitive Sources of
Construct-Irrelevant Variance
Confidential and Proprietary.
6
Some Background
Construct = knowledge, skill, or other
attribute (KSA) you are trying to test
Construct-relevant = related to KSA that you
are trying to test
Construct-irrelevant = KSA that is not
related to what you are trying to test
Variance = differences in test scores
Confidential and Proprietary.
7
Variance, Validity &
Fairness
 Construct-irrelevant variance lowers validity.
 Construct-irrelevant variance that has different
effects across groups lowers fairness.
 Construct-irrelevant differences across groups
decrease validity & fairness.
 Construct-relevant differences across groups are
valid and, therefore, fair.
Confidential and Proprietary.
8
Guideline #2
Avoid Affective Sources of
Construct-Irrelevant Variance
Confidential and Proprietary.
9
Sensitivity Awareness
 Any reference or language in an item or
passage that might cause a student to
have an emotional reaction during the
test administration can prevent a student
from being able to accurately
demonstrate ability.
 Note: The emotional factor is not limited to
negative emotions, but can also include a
“giggle factor” in items.
Confidential and Proprietary.
10
Guideline #3
Avoid Physical Sources of
Construct-Irrelevant Variance
Confidential and Proprietary.
11
Samples of Barriers
 Visual stimuli in the middle of paragraphs
 Decorative rather than informative illustrations
 Fonts that are hard to read
 Letters that look alike (e.g., O, Q) used as labels for dif-ferent
things in the same item/stimulus
Confidential and Proprietary.
12
How Might Item Bias
be Detected?
 Judgmental Procedure
• Bias and sensitivity review prior to field-testing
 Statistical Procedure
• DIF (Differential Item Functioning) analysis following fieldtests and operational administrations
Confidential and Proprietary.
13
Guiding Questions for
Passages

Does the passage disadvantage any population (gender,
race, ethnicity, language, religion, socioeconomic status,
disability or geographic region) for non-educationally relevant
reasons?

Does the passage contain controversial or emotionally
charged subject matter that is not supported by the Common
Core State Standards?

Is the passage potentially offensive, demeaning, insensitive,
or negative toward any population?

Does the passage depict any population in a stereotypical
manner?
Confidential and Proprietary.
14
Sample Passage #1
Why You Get Hungry
Not hungry? Don’t worry…you will be. Just a little while from now, you'll begin to feel
something just under your rib cage. It may seem like a little pressure or warmth. Some
people think it feels like hollowness. …
You’ve been getting hungry every few hours since the day you were born. Isn't it about
time to understand what’s going on?
Don’t blame an empty stomach. Early in the twentieth century, doctors noticed something
interesting about patients who had their stomachs surgically removed. The patents still
felt hunger the same old way. So doctors and scientists decided the feeling of hunger
must come from somewhere else.
It turned out to be a tiny area, about the size of a garbanzo bean, in the middle of your
brain. It’s called the hypothalamus (hi-po-THAL-a-mus). It constantly monitors what’s
going on in your bloodstream and sends reports to different parts of your brain.
Blood’s too warm? Make ‘em sweat. Blood’s too cool? Make ‘em shiver. Not enough fuel
in the blood? Make ‘em eat. How? By making ‘em hungry. It’s your brain creating that
unmistakable feeling called hunger.
Hunger is powerful, and it’s meant to be unpleasant. Once it starts, there’s really nothing
you can do to relieve it except eat. And so you eat—usually until a second part of your
hypothalamus tells your brain that the amount of fuel in your blood is just right …
Fairness Guideline 1, 2, or 3?
Confidential and Proprietary.
15
Sample Passage #2
The Dog Who Wouldn’t Be
My third class that morning happened to be French. The teacher was a desiccated
female whose spiritual home may have been Paris, but who had never actually been
farther east than Winnipeg. She was affected, humorless, and a tyrant. None of us
liked her. Yet I actually felt sorry for her when, in the midst of the declination of an
irregular verb, Wol whumped moodily in through the second-story window and slid to
an unsatisfactory halt upon the top of her hardwood desk. The exclamation with
which she greeted him was given in very old Anglo-Saxon, without even a hint of a
French accent.
I had an interview with the principal after this incident, but he was a reasonable man
and the upshot was that I escaped corporal punishment on the understanding that
my owl would stay at home in future.
I achieved this end, but only at the cost of giving the owls the free run of our house.
Some ten minutes before I was due to leave for school I would invite Weeps and
Wol into the kitchen, where they were allowed to finish off the bacon scraps left from
our breakfast. Apparently Wol accepted this as a sufficient bribe, for he made no
further attempt to follow me to school; and Weeps, who always accepted Wol’s lead
gave me no more trouble either. …
Fairness Guideline 1, 2, or 3?
Confidential and Proprietary.
16
Sample Passage #3
Tiny Houses: a tiny trend that runs against the mainstream
In 2008, Ben and Sarina Speed of Maine challenged themselves to build and live in the
smallest house possible—one that would accommodate only their most fundamental
needs along with the needs of their young children and two cats. Among the reasons for
downsizing was their conviction that smaller houses are environmentally friendly because
they consume less energy. Indeed, the couple reports that now the electric bill for their
640 square foot home, which is smaller than some garages, averages $20 per month.
The Speeds have joined an increasing number of Americans committed to shrinking their
residential footprints and purging their lives of excesses they have come to view as
burdensome, indulgent, harmful to the environment, or all of the above.
How small is small? Jay Shafer, the co-founder of the Small House Society, says
designs can range in size from 65 square feet to 874 square feet. Even the high end of
the range marks a substantial reduction from the average sized home.
Of course, not everyone approves of this philosophy. Supporters and opponents of tiny
homes—also called eco-homes, micro-homes, or mini-homes—tend to be resolute in
their positions.
That tiny houses use less energy and are therefore environmentally friendly is
indisputable; the smaller the area, the less energy required for heating and cooling…
Let’s discuss
Confidential and Proprietary.
17
Sample Passage #4
A Treasury of Western Folklore
Upon the 10th of May, 1869, the rival roads approached each other, and two lengths of
rails were left for the day’s work. At 8 A.M., spectators began to arrive; at quarter to 9
A.M., the whistle of the Central Pacific Railroad is heard, and the first train arrives,
bringing a large number of passengers. Then two additional trains arrive on the Union
Pacific Railroad, from the East. At a quarter of 11 A.M., the Chinese workmen
commenced leveling the bed of the road with picks and shovels, preparatory to placing
the ties. At a quarter past eleven the Governor's (Governor Stanford’s) train arrived. The
engine was gaily decorated with little flags and ribbons—the red, white, and blue. The
last tie is put in place—eight feet long, eight inches wide, and six inches thick. It was
made of California laurel, finely polished, and ornamented with a silver escutcheon,
bearing the following inscription:
THE LAST TIE LAID ON THE PACIFIC RAILROAD, MAY 10, 1869.
…In San Francisco, the wires were connected with the fire-alarm in the tower, where the
heavy ring of the bell might spread the news immediately over the city, as quick as the
event was completed.
Waiting for some time in impatience, at last came this message from Promontory Point,
at 2:27 P.M.
Confidential and Proprietary.
18
Sample Passage #4,
Continued
Almost ready. Hats off, prayer is being offered.
A silence for the prayer ensued; at 2:40 P.M., the bell tapped again, and the officer at
Promontory said:
We have got done praying, the spike is about to be presented. …
Let’s discuss
Questions?
Confidential and Proprietary.
20
How To Proceed
 Read through a specific set of test materials,
keeping in mind the guiding questions.
 Do not review these materials from the point of view
of a content expert, instead review them only from the
perspective of fairness.
 If possible sources of bias, stereotypes, or
sensitivity are detected, note them in your
comments, and then they will be discussed by
the group.
 The recorder will take complete and precise
notes for the group.
Confidential and Proprietary.
21
Passage Sources
Botkin, B. A. . A Treasury of Western Folklore. New York: Crown
Publishers, 1951
Haduch, Bill. Food Rules! New York: Puffin, 2001
Mowat, Farley. The Dog Who Wouldn’t Be. Ontario: Little Brown
& Co, 1957
Confidential and Proprietary.
22