Transcript Document 7498082

Disciplinary Research Strategies for
Assessment of Learning
Diane Ebert-May
Department of Plant Biology
Michigan State University
www.first2.org
“Consensogram” Directions
1. Take one color-coded post-it for each question,
write the question # in the corner.
2. Write a number between 0-100 on each
post-it in increments of 10.
3. Do not share responses
“Consensogram” Questions
Please respond on a scale of 0 -100 in increments of 10:
1.
2.
3.
4.
5.
What percent of your students seem to understand concepts in your course very well
during class time, but perform disappointingly “less well” on the exam?
To what degree do the assessments you use provide convincing data about student
learning?
How important is it to use multiple kinds of data to assess your students’ learning?
How often do you use data to make instructional decisions?
In my department, assessment of faculty teaching is systematic and substantive (100
agree - 0 disagree).
Question 1
Please respond on a scale of 0 - 100 in increments of 10:
How important is it to use multiple kinds of data
to assess student learning?
How important is it to use multiple kinds of
data to assess student learning?
Question 2
Please respond on a scale of 0 - 100 in increments of 10:
How often do you use data to make
instructional decisions?
How often do you use data to make
instructional decisions?
True or False?
Assessing student learning in science
parallels what scientists do as
researchers.
Assessment in Teaching
Parallels Assessment in Research







We ask questions and develop hypotheses to solve problems and make predictions
about learning.
Our questions are based on current knowledge and theories, are creative, original
and relevant to the investigator.
Research designs and methods we use to collect data are logical arguments to
answer questions.
Instruments/techniques we use are valid, repeatable measures of learning.
Assessment (results) help us understand student thinking.
Results drive our next questions or decisions about a course.
Our ideas are peer reviewed - informally or formally
What is assessment?
Data collection with the purpose of answering
questions about…
» student understanding
» students’ attitudes
» students’ skills
» instructional design and implementation
Graduate Education

Often excellent at preparing
individuals to design and carry out
disciplinary research.
Graduate Education


Often inadequate and haphazard in
preparing future faculty/professionals to
take on the increasingly complex demands
of the professoriate.
Teaching is not mentored, peer reviewed,
or based on accumulated knowledge.
Solution: IRD model

Intergenerational research teams (IRDs) in
cooperative academic environments
» Who: senior faculty, junior faculty, postdoctoral and
graduate students.
» What: scholarship of science teaching and learning is fully
integrated into the professional culture along with
discipline-based activities.

Assessment is critical to both practices.
Collaborators









Janet Batzli - Plant Biology (University of Wisconsin)
Doug Luckie - Physiology
Scott Harrison - Microbiology (graduate student)
Tammy Long - Plant Biology
Jim Smith - Zoology
Deb Linton - Plant Biology (postdoc)
Heejun Lim - Chemistry Education
Duncan Sibley - Geology
National Science Foundation
Recognizing and Rewarding
Evaluating and Improving Undergraduate
Teaching in Science, Technology,
Engineering, and Mathematics (2003)
»National Research Council
»www.nap.edu/catalog/10024.html
What Type of Learning?
Bloom (1956)
 6 major categories in the Cognitive
Domain of Educational Objectives
 Condense to 3 - realistic to work with
Cognitive Levels



Knowledge - remember
Understanding and Application - grasp
meaning, use, interpret
Critical Analysis - original thinking,
open-ended answers, whole to parts,
parts to whole, evaluation
What is assessment?
Data collection with a purpose
-- gather data about students’
learning.
--use tools like Bloom’s taxonomy to
‘calibrate’ data

What type of data do we gather?



Depends on the evidence we will accept that
students have learned what we want them to learn.
Data must be aligned with the course goals.
Measures of knowledge, attitudes, and skills.
» tests, extended responses, concept maps,
» research papers, teamwork, communication
Research Question
How can analogous assessment
questions help us understand
students’ prior understanding
and progressive thinking about
the carbon cycle over time?
Prediction

We predict that analogous, robust
questions about the carbon cycle that
are integrated into the biology
instructional design will provide the
same results about student learning
regardless of the teacher.
Theoretical Background

Conceptual change theory
»Force Concept Inventory
(David Hestenes, Physics Dept., ASU)
Carbon Cycle = Rich Problem
Why?




Integrates many biological concepts at multiple scales.
Instruction can return to elements intrinsic in the
carbon cycle - bioenergetics, metabolism.
Several documented student misconceptions
associated with the carbon cycle.
Real-world applied consequences if students continue
to misunderstand.
Some Common Misconceptions about
Photosynthesis & Respiration
Concept 1: Matter disappears during decomposition of organisms in the soil.
Concept 2: Photosynthesis as Energy: Photosynthesis provides energy for uptake of
nutrients through roots which builds biomass. No biomass built through
photosynthesis alone.
Concept 3: Thin Air: CO2 and O2 are gases therefore, do not have mass and
therefore, can not add or take away mass from an organism.
Concept 4: Plant Altruism: CO2 is converted to O2 in plant leaves so that all
organisms can ‘breathe’.
Concept 5: All Green: Plants have chloroplasts instead of mitochondria so they can
not respire.
Instructional Design


Two class meetings on carbon cycle (160 minutes)
Active, inquiry-based learning
» Cooperative groups
» Questions, group processing, large lecture sections, small discussion
sections, multi-week laboratory investigation
» Homework problems including web-based modules

Different faculty for each course
» One graduate/8-10 undergraduate TAs per course
Experimental Design
Two introductory courses for majors:
» Bio 1 - organismal/population biology (faculty A)
» Bio 2 - cell and molecular biology (faculty B)
Three cohorts:
» Cohort 1
Bio 1
» Cohort 2
Bio1/Bio2
» Cohort 3
Other/Bio2
Assessment Design



Multiple iterations/versions of the carbon cycle
problem
Pretest, midterm, final with additional formative
assessments during class
Administered during instruction
» Semester 1 - pretest, midterm, final exam
» Semester 2 - final exam
Multiple choice question (pre-post)
The majority of actual weight (dry biomass) gained by plants as they
progress from seed to adult plant comes from which one of the
following substances?
a. Particle substances in soil that are take up by plant roots. (15%).
b. Molecules in the air that enter through holes in the plant leaves
(4%).
c. Substances dissolved in water taken up directly by plant roots.
(28%).
d. Energy from the sun (29%).
N=138
What are central questions about
learning?
1. What do we want our students to know
and be able to do?
2. What knowledge or misconceptions do our
students bring to the course?
3. What evidence will we accept that
students know and can do?
4. How does our instruction help learning?
Radish Problem (formative)



Experimental Setup:
Weighed out 3 batches of radish seeds each weighing 1.5 g.
Experimental treatments:
» 1. Seeds placed on moistened paper towels in LIGHT
» 2. Seeds placed on moistened paper
towels in DARK
» 3. Seeds not moistened (left DRY) placed in light
Radish problem (2)


After 1 week, all plant material was dried in an
oven overnight (no water left) and plant biomass
was measured in grams.
Predict the biomass of the plant material in the
various treatments.
» Water, light
» Water, dark
» No water, light
Results: Weight of Radish Seedlings
1.46 g
1.63 g
1.20 g
Write an explanation about the results.
Assessment - depends on purpose




Reports from groups, formative
Peer evaluation
Individual evaluation by instructor
Score - 5 points
Whale Problem (midterm Bio 1)
Two fundamental concepts in ecology
are “energy flows” and “matter cycles”.
In an Antarctic ecosystem with the
food web given above, how could a
carbon atom in the blubber of the
Minke whale become part of a
crabeater seal? Note: crabeater seals
do not eat Minke whales. In your
response include a drawing with arrows
showing the movement of the C atom.
In addition to your drawing, provide a written description of the steps the
carbon atom must take through each component of the ecosystem
Describe which biological processes are involved in the carbon cycle.
Grandma Johnson Problem
(final, Bio 1)
Hypothetical scenario: Grandma Johnson had very
sentimental feelings toward Johnson Canyon, Utah, where
she and her late husband had honeymooned long ago. Her
feelings toward this spot were such that upon her death
she requested to be buried under a creosote bush
overlooking the canyon. Trace the path of a carbon atom
from Grandma Johnson’s remains to where it could become
part of a coyote. NOTE: the coyote will not dig up
Grandma Johnson and consume any of her remains.
Spider Monkey Problem (final, Bio 2)
Deep within a remote forest of Guatemala, the remains of a spider
monkey have been buried under an enormous mahogany tree.
Although rare, jaguars have been spotted in this forest by local
farmers. Use coherently written sentences and clearly labeled
drawings to explain how a carbon atom in glucose contained within
muscle cells of the spider monkey might become part of a cell within
the stomach lining of a jaguar. (Note:The jaguar does not dig up the
monkey and eat the remains!) Include in your answer descriptions of
the key features (not complete biochemical pathways!) of the
organismal and cellular processes that explain how the carbon atom of
the monkey’s corpse could become a part of the jaguar’s body.
Analysis of Responses
Used same scoring rubric for all three problems - calibrated
by adding additional criteria when necessary, rescoring:
Examined two major concepts:
Concept 1: Decomposers respire CO2
Concept 2: Plants uptake of CO2
Explanations categorized into two groups:
Organisms (trophic levels)
Processes (metabolic)
Trace Carbon from Whale to Seal
(Bio1 students, n=141)
100
Organism
Process
80
%
60
40
20
Concept 1
Concept 2
Decomposers respire CO2
Plants uptake CO2
Photosynthesis
Glucose
Through Root
Through Air
Primary produces
Release CO2
Respiration
Decomposers
0
Cellular Respiration by Decomposers
(Bio1/Bio2 students, n=63)
100
80
%
60
40
20
0
Q1 Whale
Q2 Grandma J
Q3 Jaguar
Concept 1: Decomposers respire CO2
2(2) = 20.16, p < 0.01
Pathway of Carbon into Primary Producer
(Bio1/Bio2 students, n=63)
100
Air
Root
80
60
%
40
20
0
Q1 Whale
Q2 Grandma J
Q3 Jaguar
Concept 2: Plants uptake CO2
2(2) = 4.778, p = .092
Trace Carbon from Spider Monkey to Jaguar
100
Respiration
NA
80
60
%
40
20
0
Bio1/Bio2 (n=63)
0ther + Bio2 (n=40)
Concept 1: Decomposers respire CO2
2(1) = 14.59, p < .01
Pathway of Carbon into Primary Producer
100
Air
Root
NA
80
60
%
40
20
0
Bio1/Bio2 (n=63)
0ther + Bio2 (n=40)
Concept 2: Plants uptake CO2
2(1) = 8.89, p < 0.05
So What?
Problem sets about major concepts:
» Diagnostic re: what students
understand/misconceptions
» Methods; parallel to process in disciplinary research
» Learn what prior knowledge students brought to
course
» Learned what knowledge students’ gained
» Unveil new misconceptions
» Influenced what we taught, how we taught it
So What? (2)
Curricular changes:
»
»
»
»

Bacteria/Archaea metabolism - often omitted
Primary production - models in lab
Source/Sink and carbon flux
‘Spiral’ major concepts - over/over/over
Use of technology: CTOOLS (concept mapping java
applet ctools.msu.edu)
Assessment Gradient
low
Potential for Assessment of Learning
high
Multiple Choice … … Concept Maps … … Essay … … Interview
high
Ease of Assessment
Theoretical Framework
•Ausubel 1968; meaningful learning
•Novak 1998; visual representations
•King and Kitchner 1994; reflective judgement
•National Research Council 1999; theoretical
frameworks for assessment
low
Goal: explain evolution by natural selection
Individual Problem

Explain the phenotypic changes in the
tree and the animal. Use your
understanding of evolution by natural
selection.
How do we develop rubrics?





Describe the goals for the activity, problem, task
Select the assessment tasks aligned with goals
Develop performance standards
Differentiate levels of responses based on clearly
described criteria
Rate (assign value) the categories
Scoring Rubric for Quizzes and Homework
Level of Achieve ment
Exe mplary
(5 pts)
General Approach
• Addresses the
question.
• States a relevant,
justifiable answer.
• Presents arguments in
a logical order.
• Uses acceptable style
and grammar (no
errors).
Comp rehe nsion
• Demonstrates an accurate and
complete understanding of the
question.
• Backs conclusions with data
and warrants.
• Uses 2 or more ideas,
examples and/or arguments that
support the answer.
Ade quate
(3 pts)
• Does not address the
question explicitly,
although does so
tangentially.
• States a relevant and
justifiable answer.
• Presents arguments in
a logical order.
• Uses acceptable style
and grammar (one
error).
• Demonstrates accurate but only
adequate understanding of
question because does not back
conclusions with w arrants and
data.
• Uses only one idea to support
the answer.
• Less thorough than above.
Needs Improveme nt
(1 pt)
• Does not address the
question.
• States no relevant
answers
• indicates
misconceptions.
• Is n ot clearly or
logically organized.
• Fails to use acceptable
style and grammar (two
or more errors).
• Does not demonstrate accurate
understanding of the question.
• Does not provide evidence to
support their answer to the
question.
No Answer (0 pts)
Advantages of Scoring Rubrics





Improve the reliability of scoring written assignments and oral
presentations
Convey goals and performance expectations of students in an unambiguous
way
Convey “grading standards” or “point values” and relate them to
performance goals
Engage students in critical evaluation of their own performance
Save time but spend it well
Limitations of Scoring Rubrics

Problem of criteria
Problem of practice and regular use

Scoring Rubric website:

» http://www.wcer.wisc.edu/nise/cl1/flag/




Sample Rubrics for Organismal Biology
http://www.msu.edu/course/lbs/144/f01
www.first2.org
www.msu.edu/~ebertmay/isb202/home.html
The Grandma Johnson Problem
Hypothetical Scenario: Grandma Johnson had very
sentimental feelings toward Johnson Canyon, Utah where she
and her late husband had honeymooned long ago. Her feelings
toward this spot were such that upon her death she
requested to be buried under a creosote bush overlooking the
canyon. She loved the idea that she'd become part of the
wonderful wilderness and live on through the wildlife that
lived there. Think to yourself and begin to trace the path of a
carbon atom from Grandma Johnson's (GJ) remains to where
it could become part of a coyote (NOTE: the coyote WILL
NOT dig up Grandma and consume any of her remains). What
fundamental pathways and processes of biology will be
involved in the transit of GJ's carbon atoms to that of the
wild coyote in Utah?
Task: Create a concept map that illustrates your understanding of the relationship
between these 10 concepts in the context of the Grandma Johnson problem. You may add
up to 5 extra concepts if you need them to explain the problem more clearly.
photosynthesis
respiration
carbon cycle
decomposers
primary producers
consumers
carbon dioxide
glucose
energy
oxygen
1. Work on the problem individually first, save it in CTOOLS, and print a hard copy.
2. Work on the problem with a partner. Both of you can retrieve your concept maps,
discuss, revise and produce the best final map to which both of you have contributed.
3. Submit all three maps - yours, your partner's and the FINAL MAP you completed
together - Please put the final map on top, with both of your names. Staple them together
(5 pts off if not stapled).
C-TOOLS Research
Question:
Is there a correlation between students’ concept map and
their written explanation of a problem?
Methods:
1. Develop a diagnostic problem
2. Build into instructional design
3. Student complete as homework
4. Develop coding scheme and analyze
The Spider Monkey Problem
Deep within the remote forest of Guatemala, the
remains of a spider monkey were buried under an
enormous mahogany tree. Although rare, jaguars have
been spotted in this forest by local farmers. Use
coherently written sentences and clearly labeled
drawings to explain how a carbon atom in glucose
contained within the muscle cells of the spider monkey
might become part of a cell within the stomach lining
of a jaguar. Provide a written description of the
processes AND organisms the carbon atom must go
through to cycle through the ecosystem. Include a
clearly labeled drawing of the system. Note: the
jaguar does not dig up the monkey and eat the
remains!)
Task: Create a concept map that illustrates your
understanding of the hierarchy and relationships
between these 10 concepts:
photosynthesis
respiration
carbon cycle
decomposers
primary producers
consumers
carbon dioxide
glucose
energy
oxygen
Key Concepts - Spider Monkey Problem
Concept in Extended Response
Links on Concept Map
1. Decomposers perform respiration.
decomposers / respiration
2. Respiration releases CO2.
respiration / CO2
decomposers / CO2
3. Primary producers perform
photosynthesis.
primary producers / photosynthesis
4. Photosynthesis requires intake of CO2.
photosynthesis / CO2
primary producers / CO2
5. Photosynthesis produces glucose.
photosynthesis / glucose
primary producers / glucose
6. Carbon moves through food chain to
consmers (e.g. Jaguar).
consumer / primary producer
consumer / glucose
Coding Criteria
Each key concept coded
Each key concept counted as a






correct [C]
incorrect [I]
non-informative [N]
absent [A]
for both concept map
and extended
response.

Match:same coding on map
and writing or
No match: different coding
on map and writing
Matches Between Concept Map & Extended Response
Spider Monkey Problem
ISB 202
40.3%
Match
59.7%
No Match
n=31
Concept Present: Both C-Map & Extended Response
Spider Monkey Problem
ISB 202
Concept Map
Extended Response
100.0
% of Students Including
90.0
80.0
70.0
60.0
50.0
40.0
30.0
20.0
10.0
0.0
decomposers
perform cellular
respiration
release of CO2
primary producers
perform
photosynthesis
intake of CO2
production of
glucose
movement through
food chain
Concept
n=31
Non-matches: Concept Included on C-map or
Extended Response Only
Spider Monkey Problem
Concept Map Only
48.6%
51.4%
Extended Response
Only
The Plant Adaptation Problem
In this course, we have
claimed that land plants were the
first truly terrestrial organisms.
However, most biologists contend
that the saturated soils on land
were undoubtedly teaming with
bacteria, archaea, and protists
long before land plants evolved.
In light of this, what does the
phrase "truly terrestrial” mean?
To answer this question follow
these instructions:
Task: Make a concept map using the following concepts:
adaptation
dispersal
fitness
flowers
fruit
leaves
photosynthesis
reproduction
roots
seeds
vascular tissue
2. PRINT one copy for yourself then SUBMIT a copy electronically.
3. Then, using your concept map, write a short response to answer the question by
explaining the problems plants had to overcome to live on land and explain the adaptations
that allowed plants to overcome those problems.
4. After you finish you short response, print it out and use a highlighter to highlight the
statements (propositions) you used directly from your concept map. You should have
elaborated (explained) further upon these statements in your written response.
5. Hand in hard copy of (a) concept map and (b) written, highlighted answer.
Key Concepts - Plant Adaptation Problem
Key Concepts in Extended Response
Links on Concept Map
1. Adaptation increases fitness.
adaptation / fitness
2. Fitness “is” reproduction.
fitness / reproduction
3. Vascular tissue is an adaptation for
getting water on land.
vascular tissue / roots
4. Leaves are and adaptation for
photosynthesis on land.
leaves / photosynthesis
5. Flowers are an adaptation for
reproduction on land.
flowers / reproduction
6. Seeds are an adaptation for dispersal on
land.
seeds / dispersal
Matches Between Concept Map & Extended Response
Plant Adaptation Problem
Upper 25% of Students
40.9%
Match
59.1%
No Match
n=44
Concept Present: Both C-Map & Extended Response
Plant Adaptation Problem
Upper 25% of Students
Concept Map
Extended Response
100.0
% of Students Including
90.0
80.0
70.0
60.0
50.0
40.0
30.0
20.0
10.0
0.0
adaptation
increases fitness
fitness is
reproduction
vascular tissue for
water
leaves for
photosynthesis
flowers for
reproduction
seeds for dispersal
Concept
n=44
Non-matches: Concept Included on C-map or
Extended Response Only
Plant Adaptation Problem
Concept Map Only
40.4%
59.6%
Extended Response
Only
So Who?

Scientists in the disciplines -- use the
process they know best to gather data
about student learning to guide the
direction of undergraduate education.