Transcript Slide 1

Developing progress variables and learning progressions for the Carbon Cycle
The BEAR Assessment System
Karen Draney, Mark Wilson, Jinnie Choi, and Yongsang Lee, UC Berkeley
EVIRONMENTAL LITERACY
Outcome space
Progress variables
A progress variable is used to represent a cognitive theory of learning consistent with a developmental perspective. This is grounded in the
principle that assessments be designed with a developmental view of student learning. The underlying purpose of assessment is to determine
how students are progressing from less expert to more expert in the domain of interest, rather than limiting the use of assessment to measure
competence after learning activities have been completed.With a progress variable, we seek to describe a continuum of qualitatively different
levels of knowledge, from a relatively naïve to a more expert level.
Carbon Cycle progress variables
•Structure of Systems
•Tracing Matter
•Tracing Energy
General Level Structure
Level 7
Current Work
Level 6
Level 5
•Citizenship
•Change over Time
Future Work
Level 4
Level 3
Level 2
Level 1
Quantitative Reasoning
about Uncertainty
Quantitative Modelbased Reasoning
Qualitative Model-based
Reasoning
ÒSchool ScienceÓ
Narratives
Events Driven by
Hidden Mechanisms
Sequences of Events
Egocentric Reasoning
about Events
The “outcome space” refers to the mapping of all possible responses to an assessment item to a particular level of the progress variable. The
outcome space must be exhaustive (every student response must be mappable) and finite!
A careful study of this mapping has engaged our group for many meetings. In order to ensure the reliability of our scoring methods, and the
stability of our variable structure, we have taken sets of student responses to many of our assessment items, and had teams at both Michigan
State University and and Berkeley, score them All differences were resolved through discussion (below left). In addition, we have attempted to
select from our data an “exemplary” student response for each item at every scoring level of the variable(s) with which it is associated, and to
annotate why it is exemplary (below right).
TM Item scoring issue : BURNING A MATCH
Exem plary Response
Q : What happens to the wood of a match as the match burns?
Why does the match lose weight as it burns?
Total 81,819
It turns to ash. The heat makes it lose weight.
It loses weight because if the fire goes on it, It burns it and
turns into ashes.
Because wood burns into ash, takes out the water and sap.
It becomes ashes. Ashes don’t weigh anything.
• It could be a simple description
about visible events.
• Students might simply mention
ashes without recognizing
chemical reaction, so we should
not assume students recognize
chemical reaction with these
responses.
MSU : More than Level 3
• Ashes are macroscopic products and
students who identify ashes should be
distinguished from students who don’t.
• “burning into ash” is a chemical reaction,
so we can assume students identify
chemical process with these responses.
Each item must represent some number of levels of one or more progress variables. The design of items, and the selection of existing items, to
represent the variables, has occupied much of our time. In this process, the progress variables take form.
Grandma Johnson (GRANJOHN)
Grandma Johnson hadvery
sentimental feelings toward Johnson
Canyon, Utah, where she and her
late husband had honeymooned long
ago. Because of these feelings,
when she died she requested to be
buried under a creosote bush in the
canyon. Describe below the path of a
carbon atom from Grandma
JohnsonÕsremains, to inside the leg
muscle of a coyote. NOTE: T he
coyote does not dig up and consume
any part of Grandma JohnsonÕs
remains.
4. School science
narratives of
Decomposers break down Grandma
processes
JohnsonÕsremains, leftover nutrients are
absorbed into the rests of a creosote bush, a
bird eats the fruit from the bush, the coyote
catches and eats the bird.
3. Causal
sequences of
events with
hidden
mechanisms
When Grandma Johnson's body began
decom posing, the carbon atom was released
into the soil under the creosote bush, and
could have gotten washed away by rain to the
nearest lake, brook, or stream.
2. Events based
narratives about
materials
Attention
MCK: ÒThe carbon in
grandma body is
decomposed into the
ground. The plants then
use the f ertile soil to use
her carbon atoms. As the
soil passes it to the plant,
the plant is ev entually
eaten by the coy ote. The
carbon atom then trav els
to its leg.Ó
Grandma
Johnson
It Identifies reactants and
gas such as oxygen related
to chemical and physical
changes.
it correctly identifies some
reactants, but faisl to
identifies gas.
It indicates Grandm a
Johnson's body is
com posed with carbon
atoms. Decom posing
(attention to hidden
mechanism)
A carbon atom from Grandm a Johnson's
remains sink into the ground and m ixes with
it sim ply describes the path
the soil. T hen when the soil is mixed and
of carbon atoms as natural
churned, it rises to the top of the ground.
phenomena
When the coyote kills something upon that dirt,
he m ay consume it and have some of them
Graphical Representation of the Item level difficulties and
person performance levels for Structure of Systems
In the graphical representation to the right, an X represents a person
performance on a collection of items related to Structure of Systems.
The numbers on the right represent scoring levels (from the general level
structure above) on individual items (e.g. 5.2 indicates a level 2
performance on item 5). Less difficult tasks and less proficient persons
are toward the bottom of the page.
Grandma
Johnson
DES: ÒGrandma
Johnson's remains decay
and decomposers use
respiration and turn it
to carbon dioxide. The
plants absorb the carbon
dioxide. Rodents eat the
plants and then the
coyote eats the rodent.Ó
The carbon is released from Grandm a
Johnson's body and travels up through the soil
and is used during photosynthesis by the plant
to make oxygen. A primary consum er would
eat the plant som e where along the food chain,
the coyote receives the carbon atom.
Measurement Model
Items design
CRD: ÒA carbon atom from
Grandma Johnson's remains
sink into the ground and
mixes with the soil. Then
when the soil is mixed and
churned , it rises to the top of
the ground. When the coyote
kills something upon that dirt,
he may consume it and have
some of the soil come with it ,
which produces him with
Grandma Johnson's carbon
atom. Ó
Characteristics of
Response
5. Model-based
accounts across
scales
Students’ actual responses
UCB : Level 2
To define the levels of the progress variables, we started with a description of the types of reasoning necessary to function at a high level of
environmental literacy.
In addition, we examined both previous literature, and written and interview-based accounts, from elementary, middle school, and high school
students, to define additional levels. The lowest levels of reasoning, generally used by middle to upper elementary school students, we referred to
as a lower anchor This includes a lack of awareness of various “invisible” mechanisms (including microscopic and atomic-molecular structure,
large-scale structure, gases, etc.); reliance on senses rather than data; and narrative rather than model-based reasoning. As an upper anchor, we
selected the highest levels of performance seen in high school students after completing relevant science units
Responses Exemplifying Levels of the Tracing Mat ter Variable
Decomposers
Soil
Plants
Herbiv ores
Coyote
Decomposers
Soil
Plants
Herbivores
Coyote
Grandma
Johnson
The use of a formal measurement model like this one to analyze our data
allows us to examine our expectations for assessment empirically. We
can make certain that items that we expect to be particularly easy or
difficult in fact are, and we can find and examine item and person
performances that are unusual or unexpected.
In this case, we expected that students would use similar levels of
reasoning on a wide variety of different assessment tasks, leading to a
“banding” effect in which the same scoring levels would occur together
across most or all of the items. In our preliminary analysis of the
Structure of Systems data, we have indeed observed this.
Decomposers
on d
Carb
e
ioxid
Plants
Herbiv ores
Coyote
Center for Curriculum Materials in Science (CCMS)
This research is supported in part by three grants from the National Science Foundation: Developing a research-based learning progression for the role of carbon in
environmental systems (REC 0529636), the Center for Curriculum Materials in Science (ESI-0227557) and Long-term Ecological Research in Row-crop Agriculture
(DEB 0423627. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the
views of the National Science Foundation.
EVIRONMENTAL LITERACY