Transcript Effective methods for the use, creation, analysis, and

Effective methods for the use, creation,
analysis, and interpretation of shortanswer student conceptual evaluations.
Ronald Thornton
Professor of Physics and Education
Center for Science & Math Teaching
Tufts University
R.K. Thornton
What was I thinking?

I’ll paint your house and walk your dog as
well.
R.K. Thornton
In Defense of Thoughtful
Multiple Choice Conceptual
Assessment
Ronald Thornton
Professor of Physics and Education
Center for Science & Math Teaching
Tufts University
R.K. Thornton
Modest Suggestions from a
Chemically Illiterate Physicist.
Ronald Thornton
Professor of Physics and Education
Center for Science & Math Teaching
Tufts University
R.K. Thornton
Center for Science and Math Teaching
Tufts University
Curriculum
Development
Educational
Research
Computer Tool
Development
Teacher & Professor
Education
R.K. Thornton
Funding



NSF
National Science Foundation
FIPSE
Fund for the Improvement of Post
Secondary Education
US Department of Education
R.K. Thornton
Wouldn’t it be nice if teachers
could understand what students
know from a simple conceptual
evaluation?
and they knew what to do to help the
student learn
R.K. Thornton
What use might this talk be?
If you intend to develop a chemistry concept
inventory these suggestions may help you
make it more useful.
 If you intend to use a chemistry concept
inventory these ideas should help you pick a
useful one.

R.K. Thornton
We have spent years


Creating effective learning environments for
introductory science(physics) courses (curricula,
tools, pedagogical methods, group structures)
And developing methods of conceptual evaluation to
measure student learning and guide our progress.
R.K. Thornton
Why Multiple Choice?






More easily administered to large numbers of
students.
Evaluation takes less time.
Student responses can be reliably evaluated even
by the inexperienced.
Can be designed to guide instruction.
With proper construction, student views can be
evaluated from the pattern of answers, changes
over time can be seen, frequency of student views
can be measured.
Multiple choice combined with open response can
help the teacher/researcher explicate the students
response.
R.K. Thornton
Why not?
Every “good” educator knows multiple
choice questions are no good.
 Badly constructed multiple choice can give
misleading results.
 Unless very carefully constructed, multiple
choice will not identify student thinking.
 The choices may be inappropriate when
used with different audiences

R.K. Thornton
First steps
Why do you want to make (use) a
conceptual evaluation?
 In what conceptual area do you want to
know how students think?

R.K. Thornton
Why?

There are pre-requisite areas of conceptual
knowledge that students need to know to
actually understand chemistry.
R.K. Thornton
What? Three modest suggestions.



Explore student beliefs in the atomic nature of
matter. (students may say atoms exist but few
believe it in any functional matter)
Explore student beliefs the dynamic nature of
equilibrium. (Most students seem to have a static
model)
Explore student beliefs about the difference
between heat energy and temperature. (Most
students do not clearly make this distinction.)
R.K. Thornton
Our research has shown.






Student conceptual responses can be context dependent.
Student domains of applicability can be different from those
of a scientist.
Students (and scientists) can hold apparently inconsistent
views simultaneously. (and it doesn’t mean they are stupid.)
Conceptual transitions are not instantaneous.
There is statistical evidence of a hierarchy of student
conceptual views.
You can do more with large-scale conceptual evaluation than
just generating a single number.
R.K. Thornton
Good Practice for the Construction
of Conceptual Multiple Choice
All answers, "right or wrong," should help evaluate
student views.
 Derive the choices in the questions from from
student answers to free response questions and from
student interviews.
 Check to see students almost always find an answer
that they are satisfied with. Random answers should
be few.
 Ask similar questions in different representations.
 Check results with different student populations.
(more)

R.K. Thornton
Good Practice (continued)





Look at correlations among questions and use
patterns to understand student thinking.
Understand the implications of “correct” and
“incorrect” answers to their performance on other
tasks.
Check for gender differences
Identify circumstances for “false positive” answers
If at all possible, construct the evaluation so it is
useful to guide instruction.
R.K. Thornton
Multiple Choice Conceptual Evaluation

Conceptual evaluation for
 kinematics
(description of motion) and
 dynamics (force and motion which is well characterized
by Newton’s Laws).
Force & Motion Conceptual Evaluation (FMCE)
 Conceptual evaluation for heat energy and
temperature
The Heat and Temperature Conceptual Evaluation
(HTCE)
Both developed by the Center for Science and Math Teaching
at Tufts
R.K. Thornton
Using the FMCE as an example


Student answers correlate well (well above 90%)
with written short answers in which students
explain the reason for their choices
Almost all students pick choices that we can
associate with a relatively small number of student
models. (Conceptual Dynamics, R.K. Thornton in ICUPE
proceedings edited by Redish)

Testing with smaller student samples shows that
those who can pick the “correct” graph under these
circumstances are almost equally successful at
drawing the graph correctly without being
presented with choices.
R.K. Thornton
FMCE as example


Because we are able to identify statistically most
student views from the pattern of answers (and
because there are very few random answers), we
are also able to identify students with less common
beliefs about motion and follow up with
opportunities for interviews or open-ended
responses to help us understand student thinking.
The use of an easily administered and robust
multiple choice test has also allowed us and others
to track changes in student views of dynamics and
to separate the effects of various curricular
changes on student learning.
R.K. Thornton
FMCE as example

Use multiple representations
 The
Force Graph questions require explicit
knowledge of coordinate systems and graphs
but require little reading.
 The Force Sled questions use natural language
and make no explicit reference to a coordinate
system or graphs.
R.K. Thornton
A sled on ice moves in the ways described in ques t ions 1-7 below. Friction is so sm all
that it can be ignored. A person wearing spiked shoes standing on the ice can apply a
force to t he sled and push it along the ice. Choose the one force (A t hrough G) which
would ke ep the sled m ovi ng as described in each statement below.
You may use a choice more than once or not at all but choose only one answer for each
blank. If you think that none is correct, answer choice J.
Direction of Force
A. The force is toward the right and is
increasing in strength (magnitude).
B. The force is toward the right and is of
constant strength (magnitude).
C. The force is toward the right and is
decreasing in strength (magnitude).
D. No applied force is needed
Direction of Force
E. The force is toward the left and is
decreasing in strength (magnitude).
F. The force is toward the left and is of
constant strength (magnitude).
G. The force is toward the left and is
increasing in strength (magnitude).
1. Which force would keep the sled moving toward the right and speeding up at
a steady rate (constant accelerat ion)?
2. Which force would keep the sled moving toward t he right at a steady
(constant ) velocit y?
3. The sled is moving toward the right. Which force would slow it down at a
st eady rate (constant accelerat ion)?
4. Which force would keep the sled moving toward t he left and speeding up at a
st eady rate (constant accelerat ion)?
5. The sled was started from rest and pushed until it reached a steady (constant )
velocity t oward the right. Which force would keep the sled moving at this
velocity?
6. The sled is slowing down at a steady rate and has an accelerat ion to the right.
Which force would account for this mot ion?
7. The sled is moving toward the left. Which force would slow it down at a
st eady rate (constant accelerat ion)?
R.K. Thornton
Questions 14-21 refer to a toy car which
can move to the right or left along a
horizontal line (the positive part of the
distance axis).
+
0
Assume that friction is so small that it
can be ignored.
A force is applied to the car. Choose the
one force graph ( A through H
) for each
statement below which could allow the
described motion of the car to continue.
You may use a choice more than once
or not at all. If you think that none is
correct, answer choice .J
__14. The car moves toward the right
(away from the origin) with a
steady (constant) velocity.
__15. The car is at rest.
__16. The car moves toward the right
and is speeding up at a steady rate
(constant acceleration).
__17. The car moves toward the left
(toward the origin) with a steady
(constant) velocity.
__18. The car moves toward the right
and is slowing down at a steady rate
(constant acceleration).
A
B
C
D
E
F
__19. The car moves toward the left and
is speeding up at a steady rate
(constant acceleration).
__20. The car moves toward the right,
speeds up and then slows down.
__21. The car was pushed toward the
right and then released. Which
graph describes the force after
the car is released.
G
H
J
+
F
o
r 0
c
e
+
F
o
r 0
c
e
+
F
o
r 0
c
e
+
F
o
r 0
c
e
+
F
o
r 0
c
e
+
F
o
r 0
c
e
+
F
o
r 0
c
e
+
F
o
r 0
c
e
-
Time
Time
Time
Time
Time
Time
Time
Time
None of t hese graphs is correct .
R.K. Thornton
Comparison with short answer
As with all the questions on the test students
who answered correctly were also able to
describe in words why they picked the
answers they did.
 Statistically one of the last questions to be
answered in a Newtonian manner is the
force on a cart rolling up a ramp as it
reverses direction at the top (question 9).

R.K. Thornton
Back to best practices. Consider




All answers, "right or wrong," should help
evaluate student views.
Derive the choices in the questions from from
student answers to free response questions and
from student interviews.
Check to see students almost always find an
answer that they are satisfied with. Random
answers should be few.
Look at correlations among questions and use
patterns to understand student thinking.
R.K. Thornton
An example from the H&T
Conceptual Evaluation

Distinguishes different student models for
the relationship between heat and
temperature.
R.K. Thornton
Questions 1 through4 refer to two cups, A and B, which containdifferent amountsof
water. T he water in each cup is heatedas described. In questions1 through3 the cups
are in aroom where the temperature is 25 °C. In question 4 the cups are in different
environments.For each question chooseone of the four answers A throughD.
A) Cup A had more heat energy transferred
B) Cup B had more heat energy t ransferred
C ) Both cups had t eh same amount of heat energy t ransferred
D) not enough informat ionis given t o det ermine the answer
____1. Cup A cont ains 100 grams
of wat er and
A
cup B cont ains t wice as much wat.erT he
Room
wat er in both cups was init ially at room
100 g
Temperature
25°C
t emperat ure. Cup A was heatedto 75°C
75°C
and cup B was heat ed t o 50°C
. Which cup
had more heat energ
y t ransferred t o it ?
•
____2.
____3.
____4.
Cup A cont ains 100 grams
of wat er and
cup B cont ains 50 grams
of wat er. T he
wat er in both cups was init ially at room
t emperat ure. Cup A was t hen heat edt o
45°C and cup B was heat ed t o 90°
C.
Which cup had mor
e heat energy
t ransferred ot it ?
Cup A cont ains 100 grams
of wat er and
cup B cont ains 80 grams
of wat er. T he
wat er in both cups was init ially at room
t emperat ure. Cup A was t hen heatedto
45°C and cup B was heat ed t o 50°
C.
Which cup had mor
e heat energy
t ransferred ot it ?
Cup A cont ains 100 grams
of wat er and is
init ially at 10°
C in a refrigerat or. Cup A is
heated unt il it s t emperat
ure is 20°C.
Cup B cont ains 50grams of wat er
init ially at 70°
C in an oven. Cup B is
heated unt il it s t emperat
ure is 90°C.
Which cup had mor
e heat energy
t ransferred ot it ?
A
100 g
Room
Temperature
25°C
45°C
A
100 g
B
200 g
50°C
B
50 g
90°C
Room
Temperature
25°C
45°C
B
80 g
50°C
A
B
Refrigerator
Temperature
100 g
50 g
10°C
20°C
90°C
Oven
Temperature
70°C
Results by category
100
natural language
90
Before Instruction
After RealTime Physics
graphical
Average % of Students
Understanding
80
70
60
50
40
30
20
10
0
Acceleration
1st & 2nd(n)
1st & 2nd(g)
Coin Toss
Cart on Ramp
3rd Collisions
Force and Motion Conceptual Evaluation
R.K. Thornton
What about 1 number results
Not my favorite, but useful in some
situations
 Let’s compare the performance of 350 RPI
students in the beginning physics course on
the FMCE and the FCI

R.K. Thornton
Comparison of FMCE Gains
Ore gon Traditional Alge bra 1988-1989 (N=236)
SUNY Albany Traditional Calculus F1998 (N=73)
Sydne y Traditional Calculus 1995 (N=472)
RPI Studio Phys ics S1998 (N=145)
Sydne y Calculus + ILDs 1999 (N=60)
M t. Ararat H.S. ILDs S1998 (N=33)
RPI Studio Phys ics + ILDs S1999 (N=311)
M uhle nbe rg Col. Calculus + ILDs F1997 (N=87)
CU Calc +Pe e r & UW Tutorial S2004 (N=391)
Jolie t Junior Colle ge Calculus RTP labs 1997-2003 (N=199)
Dick ins on Work s hop Phys ics F1999-2000 (N=104)
Ore gon Alge bra + ILDs F1991, Pre from 1989 (N=79)
Ore gon Alge bra RTP labs F1991-94, Pre from 1989 (N=613)
Tufts Age bra + ILDs 1994, 1996, 1997 (N=325)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
R.K. Thornton
<g> (% Norm alize d Gain)
.
100%
Still one number

Let’s compare the performance of 350 RPI
students in the beginning physics course on
the FMCE and the FCI
R.K. Thornton
Correlation Coefficient 0.791
R.K. Thornton
Correlation Coefficient 0.8309
R.K. Thornton
Are the evaluations the same?






Yes? Very high correlations (about 0.8 pre and
post with different instructional methods)
Yes? A high score on one implies a high score on
the other.
No? FCI fractional scores are almost always
higher than FMCE scores
No? Evaluations are measuring different things
No? A low score on the FMCE (non-Newtonian
student) does not imply a low score on the FCI
Lets look at a group of non-Newtonian students
R.K. Thornton
FMCE Conceptual Categories for Low FMCE/High FCI Students
FMCE<0.4, FCI•
0.6 (N=54)
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Velocity
Acceleration
Coin Acc
1st & 2nd (nl)
1st & 2nd (g)
Coin Toss
Cart Ramp
3rd Contact
3rd Collision
R.K. Thornton
The conceptual threshold effect
(looking at pre-post correlations)
R.K. Thornton
Pre/Post Evaluation--The Threshold Effect
Tufts University Calculus-based Physics (N=181)
FMCE Post vs. Pre
1.00
0.80
0.60
0.40
Spring 1994 (N=48)
Spring 1995 (N=37)
Spring 1997 (N=43)
Spring 1998 (N=53)
0.20
0.00
0.00
0.20
0.40
0.60
Before Instruction
0.80
1.00
R.K. Thornton
University Physics Courses
Before Instruction
Average College and University Results
Force
Before Instruction
Acceleration
Velocity
0
20
40
60
80
100
% of Students Understanding Concepts
R.K. Thornton
University Physics Courses
After Normal Instruction
Average College and University Results
After Traditional Instruction
Force
Before Instruction
Acceleration
Velocity
0
20
40
60
80
100
% of Students Understanding Concepts
R.K. Thornton
Physics & Science Courses
Using New Methods
We have evidence of substantial, persistent learning
of such physical concepts by a large number of
students in varied contexts in courses and
laboratories that use methods I am about to
describe.
Such methods also work for students who have
traditionally had less success in physics and
science courses: women and girls, minority
students, and those who are badly prepared.
R.K. Thornton
University Physics Courses
After New Methods
Average College and University Results
After New Methods
Force
After Traditional Instruc.
Before Instruction
Acceleration
Velocity
0
20
40
60
80
100
% of Students Understanding Concepts
R.K. Thornton
Our Instructional and Assessment Philosophy
“I still don’t have all of the
answers, but I’m beginning to ask
the right questions.”
R.K. Thornton