Transcript Assessing Case-Based Instruction Mary Lundeberg Professor and Chair Teacher Education Michigan State University October, 2005 Buffalo, New York.

Assessing Case-Based
Instruction
Mary Lundeberg
Professor and Chair
Teacher Education
Michigan State University
October, 2005
Buffalo, New York
Overview
• Why do investigations in classrooms?
– Research questions;
– Student learning, motivation, challenges
• Rules of thumb for designing studies
• How and what should we measure?
– Priorities, tasks, data interpretation
• Discussion of research questions,
design and measures
What evidence might you
collect?
Research Question
Method
Assessment Materials
Why do investigations?
• Scientific inquiry includes:
– Significant questions that can be
empirically investigated
– Research linked to theory
– Methods appropriate to the question
– Replication and generalization across
studies
– Professional scrutiny and critique
» Shavelson & Towne, 2002
Research questions
• Description
– What is happening?
• Cause
– Is there a systematic effect?
• Process or mechanism
– Why or how is it happening?
» Shavelson & Towne, 2002
Description
What is happening?
• Survey of case teaching in science
• (23 states + Canada)
– 2004-2005 Faculty perceptions (n=101)
Contexts of case use; benefits and challenges
– 2005-2006 Student perceptions
benefits and challenges
Student Learning Perceptions: Critical Thinking
Students are better able to
view an issue from multiple
perspectives
91.3
1.3
90.1
Students develop a deeper
understanding of concepts
1.3
88.8
Students demonstrate
stronger critical thinking skills
2.5
82.6
Students make connections
across multiple content areas 0
61.3
Students have increased their
discussion of ethical issues
8.8
Students have difficulty
making connections across
multiple content areas (-)
17.5
62.6
0
20
40
60
Percent
Disagree
Agree
80
100
Student Learning Perceptions: Learning (Pos)
91.3
Students have a better grasp
of the practical application of
core course concepts
78.8
Students strengthen
communication skills
5
18.8
More content is covered in my
classroom
47.6
0
20
40
60
Percent
Disagree
Agree
80
100
Student Motivation and Participation
Students take a more active
part in the learning process
when they use case studies
95.1
80.1
Students develop positive
peer-to-peer relationships
1.3
27.5
Attendance has increased on
the days when cases are used
17.5
16.3
Students have difficulty
working in small groups (-)
65.1
0
20
40
60
Percent
Disagree
Agree
80
100
Student Motivation
Motivation:
93.8
Students are more engaged in
class
Other:
45
Student evaluate my teaching
more positively
8.8
0
20
40
60
Percent
Disagree
Agree
80
100
Obstacles Faculty Encounter: Top Five
Cause:
Is there a systematic effect?
• Does case-based learning lead to greater
increases in competency, knowledge and/or
attitudes about genetics or infectious
diseases than lectures?
• Does case-based teaching increase ability to
interpret data in biology?
• Does case-based teaching allow transfer of
concepts across the curriculum?
Research Design:
Two Instructors and Two Classes
Unit A: Infectious Unit B: Genetics
Disease
Class 1
Use of caseUse of traditional
based instruction lecture method
or other control
Class 2
Use of traditional Use of caselecture method
based instruction
or other control
Case It! Research Design
Solomon Four Group Design
OX O
O O
X O
O
Pre/Post with case
Pre/Post without case (control)
Posttest only with case
Posttest only without case
Pre/Post Test Results:
Data Interpretation
14
Test scores
12
10
8
Pre
Post
6
4
2
0
Exp
Control
Process or mechanism
Why or how is it happening?
• Do cases enrich student understanding of
core concepts and if so, how and for whom?
• What misconceptions do students bring to
instruction and do cases increase or dispel
misconceptions?
• When in the instructional process are cases
most promising? How much knowledge is
needed?
• Does discussing cases with different global
audiences influence student understanding,
awareness or attitudes?
NSF Grant 2003-2008
Set stage for inquiry about infectious diseases:
Quic kT ime™ and a MPEG-4 Video dec ompress or are needed to see this pic ture.
QuickTime™ and a
MPEG-4 Video decompressor
are needed to see this picture.
ELISA Results
HIV Case studies
•
•
•
Run an ELISA and a Western blot:
• Anna
• Anna’s Boyfriend
• Anna, first trimester
• Anna, second trimester
• Anna’s baby
• Positive and negative controls
What is the status of each person tested? How would you explain these
results to Anna? What recommendations would you give Anna as she
cares for herself and her baby?
Anna begins taking antiretroviral medications. A viral load test is
performed one, three, and six months after she begins this drug
treatment. After running the PCR analysis on these samples, what
would you conclude about the effectiveness of the treatment?
Case It! Project
Disease transmission testing 2-3 people for HIV with ELISA
and Western Blot (person, partner, baby)
include PCR testing for viral load variations
Testing issues
How reliable are the tests? What is the latency period for developing AIDS?
Social, personal, cultural and ethical issues
Should people engage in unprotected sex to have children? What are some global
differences in treatment of HIV in Africa and the US?
Health and biological issues
What are the symptoms of AIDS? How has the disease affected this person’s health?
How have the medicines affected this person’s life/health? Why do side effects vary?
How does viral load vary depending on the medicine, exercise and nutrition used by the
person infected?
http://www.uwrf.edu/caseit/caseit.html
LaunchPad: Discussion
Focus Group Questions
1)
From this experience, have any of your perspectives and/or
understandings about HIV/AIDS epidemic changed? If so,
please explain.

2)
62% or 31 of the participants said their perspectives have
changed - more confident - more concerned - more
understanding
How relevant do you feel the topic of HIV/AIDS is to your life?
Why do you feel this way?

66% or 33 of the participants said it was relevant either personally
or globally
Focus Group Data
• “It made me a lot more aware of the disease…talk a lot more
about how there are parentless families over there, like 1/3
children have parents missing because of the virus. It just
makes me wonder like what our country is doing to help, have a
more global perspective.”
• “It was definitely a cultural difference they were not as open
about the disease with family members. One of them responded
and told me that its not really accepted by parents down there
and down here parents support us in situations like that.”
Focus Group Data:
Accessibility/Availability of
Treatment
• “I think I definitely learned more…it was amazing. When they
[Zimbabwe students] would ask questions like what if I can’t
afford medical attention or what if I don’t have a doctor near by
that has the medication for me…we don’t think about it and we
always have the money somehow or we will get it to get help.
But they don’t necessarily have the ability.”
• “…one of the African students asked me if suicide would be the
best answer and that kind of really hit home compared to our
questions that were more surface-y…I thought they were more
personal, it was good.”
Student self-report
Student surveys
Student focus groups after course concludes.
Retrospective open ended reflections
most powerful when students examine data
(e.g., portfolios, case-tests, journals)
What evidence might you
collect?
Research Question
Method: Class A
Unit 1 (cases)
Unit 2 (lecture)
Assessment Materials
Class B
(lecture)
(cases)
How and what
should we measure?
Framework for PISA 2006 Science
Assessment
Context
Competencies
Knowledge
What you know:
• about the natural
world
(knowledge of
science); and
• about science itself
(knowledge about
science)
• Identify scientific
questions;
Life situations
that involve
science and
technology.
Require you
to:
• Explain phenomena
scientifically; and
• Use scientific evidence.
How you
do so is
influenced
by:
Attitudinal Responses
How you respond to science
issues (interest, support for
scientific enquiry, responsibility)
PISA 2006 Knowledge of Science Categories
Physical Systems
structure and properties of matter (e.g., thermal and electrical conductivity)
physical changes of matter (e.g., states of matter, elements, bonds)
chemical changes of matter (e.g., reactions, energy transfer, acids/bases)
motions and forces (e.g., velocity, friction)
energy and its transformation (e.g., conservation, dissipation, chemical
reactions)
interactions of energy and matter (e.g., light and radio waves, sound and seismic
waves)
Living Systems
 cells (e.g., structures and function, DNA, plant and animal)
humans (e.g., health, nutrition, subsystems [i.e. digestion, respiration,
circulation, excretion, and their relationship], disease, reproduction)
populations (e.g., species, evolution, biodiversity, genetic variation)
ecosystems (e.g., food chains, matter and energy flow)
biosphere (e.g., ecosystem services, sustainability)
Earth and Space Systems
structures of the Earth systems (e.g., lithosphere, atmosphere, hydrosphere)
energy in the Earth systems (e.g., sources, global climate)
change in Earth systems (e.g., plate tectonics, geochemical cycles, constructive
and destructive forces)
Earth’s history (e.g., fossils, origin and evolution)
Earth in space (e.g., gravity, solar systems)
PISA 2006 Knowledge About Science
Categories
Scientific Enquiry
origin (scientific questions).
purpose (e.g., to produce evidence that helps answer scientific questions, current
ideas/models/theories guide enquiries).
observations and experiments (e.g., different questions suggest different scientific investigations,
current scientific knowledge).
data (e.g., quantitative [measurements], qualitative [observations])
measurement (e.g., inherent uncertainty, replicability, variation, accuracy/precision in equipment
and procedures).
characteristics of results (e.g., empirical, tentative, testable, falsifiable, self-correcting).
Scientific Explanations
types (e.g., hypothesis, theory, model, law).
formation (e.g., extant knowledge and new evidence, creativity and imagination, logic).
rules (e.g., logically consistent, based on evidence, based on historical and current knowledge).
outcomes (e.g., new knowledge, new methods, new technologies, new investigations).
PISA 2006 Knowledge About Science Categories
(Continued)
Science and Technology in Society
role of science (e.g., understand the natural world, answers questions) and role of science-based
technology (e.g., attempts to solve human problems, develop artifacts, design processes, human
adaptation [non-biological]
relationships between science and technology (e.g., science often advances due to new
technologies, advances in scientific knowledge can advance technology).
risks (e.g., may create new problems, knowledge is often not public, benefits versus costs,
unintended consequences).
influence (e.g., science and technology influence society through their knowledge, procedures,
products, and world views).
challenges (e.g., societal issues and aspirations often inspire questions for scientific research and
problems for technological innovations).
limits (e.g., science cannot answer all questions and technology cannot solve all societal problems
or meet all human aspirations).
Recommended Contexts for the PISA 2006
Science Assessment
Personal
Health (e.g., maintenance of health, accidents, nutrition).
Resources (e.g., personal consumption of materials and energy).
Environment (e.g., environmentally friendly behavior, use and disposal of materials).
Hazards (e.g., natural and human-induced, decisions about housing).
Frontiers (e.g., interest in science’s explanations of natural phenomena, science-based hobbies, sport and
leisure, music and personal technology).
Social
Health (e.g., control of disease, social transmission, food choices, community health).
Resources (e.g., maintain human populations, quality of life, security, production and distribution of food,
energy supply).
Environment (e.g., population distribution, disposal of waste, environmental impact, local weather).
Hazards (e.g., rapid changes [ severe weather], slow and progressive changes [ erosion, sedimentation], risk
assessment).
Frontiers (e.g., new materials, devices and processes, genetic modification, weapons technology, transport).
Global
Health (e.g., epidemics, spread of infectious diseases).
Resources (e.g., renewable and non-renewable, natural systems, population growth, sustainable use of
species).
Environment (e.g., biodiversity, ecological sustainability, control of pollution, production and loss of soil).
Hazards (e.g., climate change, impact of modern warfare).
Frontiers (e.g., extinction of species, exploration of space, origin and structure of the universe).
PISA 2006 Areas for Assessment of
Attitudinal Responses
Interest in Science
Show curiosity in science and science-related issues and endeavors.
Demonstrate willingness to acquire additional scientific knowledge and skills, using
a variety of resources and methods.
Demonstrate willingness to seek information and have an ongoing interest in
science, including consideration of science-related careers.
Support for Scientific Enquiry
Support the importance of considering different scientific perspectives and
arguments.
Support the use of factual information and rational explanations.
Support the need for logical and careful processes in drawing conclusions.
Responsibility for Sustainable Development
Show a sense of personal responsibility for achieving a healthy population and safe
environments.
Demonstrate awareness of the environmental consequences of individual actions.
Demonstrate willingness to take action to maintain natural resources.
New directions for assessment
of student understanding
• priorities for assessment,
• representations of student knowledge
and competence,
• selection of tasks to assess
performance, and
• procedures for data interpretation and
analysis.
Ways to assess
understanding
• Transfer of knowledge using a task to
show how concepts
– are applied to different,
– real world problems, and
– explained using literacy fitting the context.
What tasks or situations
allow for observation of
student performance?






Course exams
Case assessments--paper and/or video
Direct questioning/surveys of students
Examination of group products; interaction
Student analysis of portfolios; journals
Minute papers
 Muddiest or most significant point
Meta-analysis of
Main Effects of PBL
(Dochy, Segers, Van den Bossche, & Gijbels, 2003)
What evidence might you
collect?
Research Question
Method: Class A
Unit 1 (cases)
Unit 2 (lecture)
Assessment Materials
Class B
(lecture)
(cases)
Resources for CBI
(http://sciencecases.educ.msu.edu/references/)
http://www.wcer.wisc.edu/salgains/instructor
http://www.indiana.edu/~nsse
Interpretation of Data
Not all that counts can be counted and
not all that is counted, really counts.
Einstein
We do not see things as they are,
We see things as we are.
The Talmud