Transcript Scientific Reasoning Competency Assessment in Higher Education in Virginia Harold Geller

Scientific Reasoning
Competency Assessment in
Higher Education in Virginia
Harold Geller
[email protected]
Department of Physics and Astronomy
George Mason University
What I’m Going to Talk About
• SCHEV Mandate for Core Competencies
• Scientific Reasoning Competency
• Basis of GMU Response
– Definition
– Standards
– Methodology
– Analysis and Presentation
• Commentary
SCHEV Mandate for Core
Competencies
• The State Council of Higher Education for
Virginia (SCHEV)
• Meeting on core competencies in
November 2001
– See state issued guidelines
• Original deadlines modified in 2002
– Responses required were delayed to 2003
Scientific Reasoning Competency
• Not easily defined
– One approach
• Hazen and Trefil (1991)
– Ability to comprehend science article in
newspaper
» Does this really define scientific reasoning?
» Can this be left to the media?
» Is this definition an end-all?
» What grade level does this represent?
– GMU approach
• See GMU Assessment Proposal
Basis of GMU Response
• Issues hashed out in committee
– Definition
– Implementation
• Which students take exam
• How many students take it
• How administered
– Ability to measure across disciplines
• Lowest common denominator
• Content independence
GMU Definition of Scientific Literacy
[by committee]
• Based upon 8 defined learning goals
–
–
–
–
–
–
–
–
Identify different scientific ways of knowing
Develop and test hypotheses
Interpret data
Recognize primary and secondary sources
Utilize qualitative and quantitative techniques
Recognize power and limitations of science process
Communicate among scientists and general public
Recognize role of science in decision making
GMU Standards
[by committee]
• To be set and established by
committee consisting of
– All faculty that teach general education
science courses
– Members of the Scientific Reasoning
Competency Assessment Committee
GMU Methodology
[by committee]
• Develop common test items
– Reflect goals in definition
• Pilot questions
– Use in final examinations of sample of general
education science courses
– Count as part of grade in class
• Analyze results of pilot testing
• Develop standards based upon results in
sample of classes
As Part of Methodology
(a syllabus addendum)
[by committee]
• Sample supplement to all general education
science classes’ syllabi
– “One of the goals of the general education program at
George Mason University is to develop students
competent in scientific reasoning. This course will
help you achieve that goal. In order to judge
university success in reaching this goal, a test will be
given in late March or early April that will focus on
scientific reasoning. This required multiple choice test
will take approximately half an hour or so to complete
and will count toward your grade.”
Sample Question (vetoed)
[by Geller]
• Johannes Kepler (1571-1630) derived his three
Laws of Planetary Motion solely from the
observations of Tycho Brahe (1546-1601). This
represents which of the following ways of
scientific knowing?
• A inductive
• B deductive
• C empirical
• D theoretical
• E none of the above
Sample Question (vetoed)
[by Geller]
•
•
•
•
•
•
The above graphs represent a phase diagram for water and carbon
dioxide. Using the phase diagram for carbon dioxide, what phase would
carbon dioxide be found at a pressure of 0.5 atmosphere and a
temperature of -78.5o C?
A
liquid
B
gas
C
solid
D
any of the above is possible
E
none of the above
Sample Question (vetoed)
[by Geller]
Time
Time
Time
Time
• The above graphs all represent experimental values observed over
a period of time. The graph that best represents a value that is
increasing exponentially over time is which one of the above
graphs?
• A
Graph A
• B
Graph B
• C
Graph C
• D
Graph D
• E
None of the above
Sample Question (vetoed)
[by Geller]
• Stefan-Boltzmann's Law relates the energy output of an
object to its temperature. The energy is proportional to
the temperature (in Kelvin) to the fourth power.
Therefore, if the temperature (in Kelvin) of an object is
doubled, its energy output will increase by how many
times?
• A 2
• B 4
• C 8
• D 16
• E 32
Sample Questions (affirmed)
[Article from Washington Post]
Tumors Prevalent in Anacostia's Fish
Rate Rivals Highest Recorded for U.S. River; Vehicle
Pollution Blamed
By Monte Reel
Fish in the Anacostia River have cancerous tumor rates
that are as high as ever documented in an American
river, and a U.S. government-led study to be published
next month links the tumors to pollution caused by
vehicle emissions and runoff.
Fifty to 68 percent of mature brown bullhead catfish
collected in 2001 from three parts of the river in the city
had liver tumors, most of which were cancerous,
according to the study led by the U.S. Fish and Wildlife
Service. In addition to the liver tumors, 13 to 23 percent
of the bullheads had skin tumors, scientists found…
Sample Questions (affirmed)
[based upon article from Washington Post]
70
60
50
40
30
20
10
0
1990
liver tumors
skin tumors
1995
2000
Tumors on Bullhead Catfish in the
Anacostia River
% fish with these
tumors
% fish with these
tumors
Tumors on Bullhead Catfish in the
Anacostia River
2005
70
60
50
40
30
20
10
0
1990
liver tumors
skin tumors
1995
Year
70
60
50
40
skin tumors
liver tumors
1995
2000
year
1.
2005
2005
Tumors on Bullhead Catfish in the
Anacostia River
% fish with tumors
% fish with these tumors
Tumors on Bullhead Catfish in the
Anacostia River
30
20
10
0
1990
2000
Year
60
50
40
30
20
10
0
1990
liver tumors
skin tumors
1995
2000
2005
Year
Which one of the following graphs is consistent with the data
presented in the article on tumors of Bullhead Catfish at a
specific point in the Anacostia river? (for competency 3)
Sample Questions (affirmed)
[based upon article from Washington Post]
•
Which of the following hypotheses are justified
given the data in the article? You expect that
fish who eat insects in the same river will:
(competency 2)
–
–
–
–
Have the same level of tumors as brown bullhead
catfish because they are living in the same
environment.
Have more tumors because the insects also pick up
the air pollution.
Have fewer tumors than bullhead catfish because
they do not eat from the river bottom.
Can’t be predicted because what the fish eat does
not matter.
GMU Analysis and Presentation
[by committee]
• Present aggregate data to SCHEV
– Percentages of high proficiency
– Percentages of acceptable proficiency
– Percentages of low proficiency
• Adjust as needed
– Determination to be made in pilot testing
Commentary
[by Geller]
• How best to test scientific literacy (a.k.a. assessment)
– Essays, multiple choice, short answer, mixes
• Problems with need to apply across disciplines
– Lowest common denominator
– Content independence
• How important is content?
• Comparison with state examinations
– Standards of Learning (Virginia)
• Comparison with reading comprehension
– New York Times is aimed to be 9th grade reading level
– Washington Post aims at 8th grade reading level
Commentary
[by Geller]
• Value of tests
– Measure of higher education or lower education?
– Added burden to all lower level science classes
• What if students take more than one of these introductory
science courses?
– Some students take astronomy as general elective.
» This can skew statistics.
• Unfunded mandates
– More attention to something other than the course
itself?
• Why is this a college/university requirement?
– Failure of “lower education”?
Acknowledgements
• GMU Office of Institutional Assessment
– Karen Gentemann
– Ying Zhou
• Center for Teaching Excellence
– Laurie Fathe
• GMU Scientific Literacy Competency Committee
– Department representatives
• Faculty volunteers
• Department Colleagues
– Robert Ehrlich, Robert Ellsworth, Maria Dworzecka,
Greg Foster (Chemistry), Joe Weingartner