Transcript Assessing Learning in Complex Domains

Storytelling as an
Instructional Method
Generating and Assessing Stories
in Support of Instruction
J. Michael Spector
Florida State University, Tallahassee, FL USA
[email protected]
Nov 7-8, 2006
Mesa, Arizona
Learning Systems Institute, Florida State University, Suite 4600-C University Center, Tallahassee, FL 32306-2540
(PHONE): (850) 644-2570 (FAX): (850) 644-4952 URL: WWW.LSI.FSU.EDU
Outline:
Twice Told Tales – 2 Stories & 2 Ideas
•
•
•
•
•
•
Tell me a story
Why
When
How
Generating stories automatically
Assessing learning with stories
Learning Systems Institute, Florida State University, Suite 4600-C University Center, Tallahassee, FL 32306-2540
(PHONE): (850) 644-2570 (FAX): (850) 644-4952 URL: WWW.LSI.FSU.EDU
A Tale of Two Stories
• A Game of India – H. A. Nielsen (Michigan
Quarterly Review, 1978)
– Experience first … then understanding
– Humility … we know less than we are inclined
to believe
• A Confession – Lev Tolstoy, 1884
– Learning as disturbance
– Quine & Ullian’s Web of Belief
– The interconnectedness of experience
Learning Systems Institute, Florida State University, Suite 4600-C University Center, Tallahassee, FL 32306-2540
(PHONE): (850) 644-2570 (FAX): (850) 644-4952 URL: WWW.LSI.FSU.EDU
Why
• Natural inclination
– It is what people do
• Memory
– efficiency of episodic memory (Anderson,
1983; Schacter, 1996)
– efficient encoding, decoding and recall
• Complexity
– Confronting complexity indirectly – tacit
knowledge
Learning Systems Institute, Florida State University, Suite 4600-C University Center, Tallahassee, FL 32306-2540
(PHONE): (850) 644-2570 (FAX): (850) 644-4952 URL: WWW.LSI.FSU.EDU
Story-based Approaches
Goal-based
Scenarios
Storytelling
Case Studies Dynamic
Stories
Structure
Case analyses –
developed over
time; static
Verbal or
written; static
Case
documentation;
static
Generated
situations,
problems,
solutions
Form
Case, scenario
story
case
Problem
scenario
Learning
Approach
Experiential
learning
Reflective
learning
Learning from
examples
Multi-faceted
Aim
Acquire specific
knowledge and
skills
Enhanced
understanding
Acquiring case
experience
Deep insights
into complex
problems
Application
Education,
training
Organizational
learning,
leadership, etc.
Medical and
Wide variety
business educ.,
engineering
Learning Systems Institute, Florida State University, Suite 4600-C University Center, Tallahassee, FL 32306-2540
(PHONE): (850) 644-2570 (FAX): (850) 644-4952 URL: WWW.LSI.FSU.EDU
When
• With younger learners – widespread
acceptance
• With millennials … growing expectations
• With what types of instructional goals and
tasks?
• With non-recurrent, ill-structured,
complex, dynamic problem solving tasks!
Learning Systems Institute, Florida State University, Suite 4600-C University Center, Tallahassee, FL 32306-2540
(PHONE): (850) 644-2570 (FAX): (850) 644-4952 URL: WWW.LSI.FSU.EDU
How
•
•
•
•
•
•
•
Personal delivery … pacing, intonation, …
Computer-based delivery
Provide an overview – scenarios
Generate interest
Motivate inquiry
Reflective exercises
Computers generating stories
Learning Systems Institute, Florida State University, Suite 4600-C University Center, Tallahassee, FL 32306-2540
(PHONE): (850) 644-2570 (FAX): (850) 644-4952 URL: WWW.LSI.FSU.EDU
Automatic Generation of Stories
People tell stories. They tell stories to fit a
situation or need. The use of stories in
that sense is dynamic.
Can an instructional computing system tell
a story based on an underlying
mathematical model and problem scenario
or need?
This possibility exists – an important
research agenda worth pursuing.
Learning Systems Institute, Florida State University, Suite 4600-C University Center, Tallahassee, FL 32306-2540
(PHONE): (850) 644-2570 (FAX): (850) 644-4952 URL: WWW.LSI.FSU.EDU
A Visual Representation
Human Resource Al l ocati on
Tra nsf er
Out
Persons Available
Hiri ng
In
Persons
R elea se
Learning Systems Institute, Florida State University, Suite 4600-C University Center, Tallahassee, FL 32306-2540
Persons
A ll oca ti o n
Persons Allocated
(PHONE): (850) 644-2570 (FAX): (850) 644-4952 URL: WWW.LSI.FSU.EDU
Generating a Problem
• Create and validate a system dynamics
simulation model for a complex, dynamic
system – a non-trivial task but many such
simulation models already exist
• Ensure that each variable, stock and
constant are well documented
• Given the current state of the simulation
model and the learner’s experience,
generate a problem situation from the
model itself
Learning Systems Institute, Florida State University, Suite 4600-C University Center, Tallahassee, FL 32306-2540
(PHONE): (850) 644-2570 (FAX): (850) 644-4952 URL: WWW.LSI.FSU.EDU
This depicts a system dynamics
model – the highlighted flow rate is
opened. There is a documentation
tab in PowerSim that allows each
component to be described. This
can be used to generate a task for
someone interacting with the
simulation model.
Learning Systems Institute, Florida State University, Suite 4600-C University Center, Tallahassee, FL 32306-2540
(PHONE): (850) 644-2570 (FAX): (850) 644-4952 URL: WWW.LSI.FSU.EDU
Interaction and Feedback
Learning Systems Institute, Florida State University, Suite 4600-C University Center, Tallahassee, FL 32306-2540
(PHONE): (850) 644-2570 (FAX): (850) 644-4952 URL: WWW.LSI.FSU.EDU
Assessing Learning with Stories
How well do stories support learning in and
about complex, ill-structured domains?
How to determine? - standard problems
with standard solutions are not available
A developmental pathway - inexperienced
problem solving and decision making
towards expert-like problem solving and
decision making
Learning Systems Institute, Florida State University, Suite 4600-C University Center, Tallahassee, FL 32306-2540
(PHONE): (850) 644-2570 (FAX): (850) 644-4952 URL: WWW.LSI.FSU.EDU
DEEP – An Assessment Tool
http://deep.lsi.fsu.edu/DMVS/jsp/index.htm
• Problem – determine progress of
learning in complex domains
• Approach – identify and annotate key
influence factors
• Strategy – compare responses to those
of known experts and track
development
• Tactic – minimize extraneous cognitive
load on respondents
Learning Systems Institute, Florida State University, Suite 4600-C University Center, Tallahassee, FL 32306-2540
(PHONE): (850) 644-2570 (FAX): (850) 644-4952 URL: WWW.LSI.FSU.EDU
Capturing Problem Conceptualizations
Present a problem situation or scenario, ask
respondents to:
1. indicate factors (name and briefly describe) they
believe critical to resolving the situation
2. indicate how these factors are interrelated (draw
links and describe relations)
3. identify the assumptions involved thus far
4. describe additional information that would be
required to resolve the situation or solve the
problem
Learning Systems Institute, Florida State University, Suite 4600-C University Center, Tallahassee, FL 32306-2540
(PHONE): (850) 644-2570 (FAX): (850) 644-4952 URL: WWW.LSI.FSU.EDU
Learning Systems Institute, Florida State University, Suite 4600-C University Center, Tallahassee, FL 32306-2540
(PHONE): (850) 644-2570 (FAX): (850) 644-4952 URL: WWW.LSI.FSU.EDU
Problem Conceptualization & Capture
Learning Systems Institute, Florida State University, Suite 4600-C University Center, Tallahassee, FL 32306-2540
(PHONE): (850) 644-2570 (FAX): (850) 644-4952 URL: WWW.LSI.FSU.EDU
DEEP in USE
Learning Systems Institute, Florida State University, Suite 4600-C University Center, Tallahassee, FL 32306-2540
(PHONE): (850) 644-2570 (FAX): (850) 644-4952 URL: WWW.LSI.FSU.EDU
Medical Summary Data
5 Experts
Scenario 1
14 Novices
Percent
cause/effect
Percent
73
68.9%
example
0
0.0%
correlation
8
7.5%
25
23.6%
106
100%
process
TOTAL Links
cause/effect
185
58.2%
example
24
7.5%
correlation
84
26.4%
process
25
7.9%
TOTAL Links
5 Experts
318
14 Novices
Percent
Scenario 2
cause/effect
Percent
cause/effect
28
41.2%
0
0.0%
correlation
10
14.7%
correlation
process
30
44.1%
process
68
100%
example
TOTAL Links
100%
Learning Systems Institute, Florida State University, Suite 4600-C University Center, Tallahassee, FL 32306-2540
example
TOTAL Links
137
46.6%
27
9.2%
123
41.8%
7
2.4%
294
100%
(PHONE): (850) 644-2570 (FAX): (850) 644-4952 URL: WWW.LSI.FSU.EDU
Node-Link Clusters
Learning Systems Institute, Florida State University, Suite 4600-C University Center, Tallahassee, FL 32306-2540
(PHONE): (850) 644-2570 (FAX): (850) 644-4952 URL: WWW.LSI.FSU.EDU
Comparing Experts & Novices
Links
Biology Experts – Scenario 2 (N = 5; Links = 128)
From / To
Percentage
Nodes
1
Analyze the mercury
concentrations
16
9/7
12.5%
2
Possible solutions
14
7/7
10.93%
3
Food chain
12
7/5
9.37%
3
Source of mercury
contamination
12
8/4
9.37%
4
Limit consumption
11
7/4
8.59%
Biology Novices – Scenario 2 (N = 16; Links = 147)
Links
From / To
Percentage
Nodes
1
Biological effects of mercury on
fish
41
21 / 20
27.89%
1
Source of mercury
contamination
41
24 / 17
27.89%
2
Human interaction
29
12 / 17
19.72%
3
Social awareness
28
13 / 15
19.04%
4
Analyze the mercury
concentrations
24
17 / 7
16.32%
Learning Systems Institute, Florida State University, Suite 4600-C University Center, Tallahassee, FL 32306-2540
(PHONE): (850) 644-2570 (FAX): (850) 644-4952 URL: WWW.LSI.FSU.EDU
Additional Issues & Measures
• Separating structural and semantic analysis
• Structural analysis
– Central nodes
– Terminal nodes (all links in same
direction)
– Feedback and systemic measures
• Similarity metrics
– Graph theory – diameter, density, path
analysis
– Tversky similarity metric
Learning Systems Institute, Florida State University, Suite 4600-C University Center, Tallahassee, FL 32306-2540
(PHONE): (850) 644-2570 (FAX): (850) 644-4952 URL: WWW.LSI.FSU.EDU
First Pass at a Systemic Metric
• Hypothesis: experts will tend to think
more systemically than non-experts
• Indicators of systemic thinking:
– Internal feedback (links back to other
parts of the system; two-way links)
– One possible measure – ratio of
unreachable pairs to all possible
ordered pairs of nodes in the problem
conceptualization
Learning Systems Institute, Florida State University, Suite 4600-C University Center, Tallahassee, FL 32306-2540
(PHONE): (850) 644-2570 (FAX): (850) 644-4952 URL: WWW.LSI.FSU.EDU
7 nodes – possible ordered pairs = 2,520
lots of internal feedback depicted
no unreachable pairs
No terminal nodes
Learning Systems Institute, Florida State University, Suite 4600-C University Center, Tallahassee, FL 32306-2540
(PHONE): (850) 644-2570 (FAX): (850) 644-4952 URL: WWW.LSI.FSU.EDU
7 nodes – possible pairs
much internal feedback
6 unreachable pairs
1 terminal node
Learning Systems Institute, Florida State University, Suite 4600-C University Center, Tallahassee, FL 32306-2540
(PHONE): (850) 644-2570 (FAX): (850) 644-4952 URL: WWW.LSI.FSU.EDU
7 nodes
little internal feedback
6 terminal nodes
38 unreachable pairs
Learning Systems Institute, Florida State University, Suite 4600-C University Center, Tallahassee, FL 32306-2540
(PHONE): (850) 644-2570 (FAX): (850) 644-4952 URL: WWW.LSI.FSU.EDU
An Ending
“What we cannot speak about, we must
pass over in silence”
(Wittgenstein, #7, Tractatus Logico-Philosophicus)
“The Moral to this story, the moral to this
song, is simply that one should never be
where one does not belong “
(Bob Dylan, The Ballad of Frankie Lee and Judas Priest)
May the Force be with you …
Learning Systems Institute, Florida State University, Suite 4600-C University Center, Tallahassee, FL 32306-2540
(PHONE): (850) 644-2570 (FAX): (850) 644-4952 URL: WWW.LSI.FSU.EDU
References (1/4)
•
•
•
•
•
•
•
•
Anderson, J. R., Bothell, D., Byrne, M. M., Douglass, S., Leviere, C., & Qin, Y. (2004). An integrated theory of the mind.
Psychological Review, 111(4), 1036-1060.
Andre, T. (1986). Problem-solving in education. In G. D. Phye & T. Andre (Eds.). Cognitive classroom learning (pp. 169–
204). New York: Academic Press.
Andrews, G. & Halford, G. S. (2002). A cognitive complexity metric applied to cognitive development. Cognitive
Psychology, 45(2), 153-219.
Bransford, J. D., Brown, A. L., Cocking, R. R. (Eds.) (2000). How people learn: Brain mind experience and school.
Washington, DC: National Academy Press.
Chi, M.T.H., Feltovich, P.J., & Glaser, R. (1981). Categorization and representation of physics problems by experts and
novices. Cognitive Science, 5, 121-152.
Christensen, D. L., Spector, J. M., Sioutine, A., & McCormack, D. (2000, August). Evaluating the impact of system
dynamics based learning environments: A preliminary study. Paper presented at the International System Dynamics
Society Conference, Bergen, Norway, August, 2000.
Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the crafts of reading, writing, and
mathematics. In L. B. Resnick (Ed.), Knowing, learning, and instruction: Essays in honor of Robert Glaser (pp. 453-494).
Hillsdale, NJ: Lawrence Erlbaum.
Dijkstra, E. (1972). The humble programmer. Communications of the ACM, 15(10), 859-866.
•
Dörner, D. (1996). The logic of failure: Why things go wrong and what we can do to make them right (R. Kimber & R.
Kimber, Trans.). New York: Metropolitan Books. (Original work published in 1989).
•
Dummer, P., & Ifenthaler, D. (2005, March). Planning and assessing navigation in model-centered learning
environments: Why learners often do not follow the path laid out for them. Presentation at the International
Conference for Methods and Technologies for Teaching, Palermo, Italy, March 10, 2005.
Ericsson, K. A, (1993). The role of deliberate practice in the acquisition of expert performance. Psychological Review,
100(3), 363-406.
Ericsson, K. A. (2001). Attaining excellence through deliberate practice: Insights from the study of expert performance.
In M. Ferrari (Ed.), The pursuit of excellence in education (pp. 21-55). Mahwah, NJ: Erlbaum.
•
•
Learning Systems Institute, Florida State University, Suite 4600-C University Center, Tallahassee, FL 32306-2540
(PHONE): (850) 644-2570 (FAX): (850) 644-4952 URL: WWW.LSI.FSU.EDU
References (2/4)
•
•
•
•
•
•
•
•
•
•
•
•
•
Gardner, H. (1999). Multiple approaches to understanding. In C. M. Reigeluth (Ed.), Instructional design
theories and models: A new paradigm of instructional theory (Vol. II) (pp. 69–89). Mahwah, NJ: Lawrence
Erlbaum Associates.
Herl, H. E., O’Neil, H. F., Jr., Chung, G. L. W. K., Bianchi, C., Wang, S., Mayer, R., Lee, C. Y., Choi, A., Suen, T., &
Tu, A.(1999). Final report for validation of problem solving measures. CSE Technical Report 501. Los Angeles:
CRESST.
Johnson-Laird, P. (1983). Mental models. Cambridge: Cambridge University Press.
Jonassen, D. (1999) Designing constructivist learning environments. In C. M. Reigeluth (Ed.), Instructional
design theories and models: A new paradigm of instructional theory (Vol. II) (pp. 215–239). Mahwah, NJ:
Lawrence Erlbaum Associates.
Jonassen, D. H. & Hernandez-Serrano, J.(2002). Case-based reasoning and instructional design: using
stories to support problem solving. Educational Technology Research and Development, 50(2), 65-77.
Jonassen, D. H. (2006). Modeling with technology: Mindtools for conceptual change (3rd ed.). Columbus, OH:
Pearson/Prentice Hall.
Kintsch, E. (1993). Principles of instruction from research on human cognition. In J. M. Spector, M. C. Polson,
& D. J. Muraida, Automating instructional design: Concepts and issues (pp. 22-42). Englewood Cliffs, NJ:
Educational Technology Publications.
Lowyck, J., & Elen, J. (2004). Linking ICT, knowledge domains, and learning support for the design of
learning environments. In N. M. Seel & S. Dijkstra (Eds.), Curriculum, plans and process in instructional
design: International perspectives (pp. 239-256). Mahwah, NJ: Erlbaum.
Lowyck, J., Pöysä, J., & van Merriënboer, J. (2003). Conditions of ICT-based design for learning communities.
Technology, Instruction, Cognition, and Learning, 1(2), 153-182.
McCarthy, B. (1996). About learning. Barrington, IL: Excell Inc.
Merrill, M. D. (2002). First principles of instruction. Educational Technology Research and Development, 50(3),
43-59.
Miller, G. A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for
processing information. Psychological Review, 63, 81-97.
Milrad, M., Spector, J. M., & Davidsen, P. I. (2002). Model facilitated learning. In S. Naidu (Ed.), Learning and
teaching with technology: Principles and practices (pp. 13-27). London: Kogan Page.
Learning Systems Institute, Florida State University, Suite 4600-C University Center, Tallahassee, FL 32306-2540
(PHONE): (850) 644-2570 (FAX): (850) 644-4952 URL: WWW.LSI.FSU.EDU
References (3/4)
•
•
•
•
•
•
•
•
•
•
•
Moreno, R. Mayer, R., Spires, H., Lester, J. (2001) The Case for social agency in computer-based teaching: Do
students learn more deeply when they interact with animated pedagogical agents? Cognition and Instruction,
19(2), 177-213.
Nelson, L. M. (1999). Collaborative problem solving. In C. M. Reigeluth (Ed.), Instructional design theories and
models: A new paradigm of instructional theory (Vol. II) (pp.241– 267). Mahwah, NJ: Erlbaum.
Novak, J. D. (1998). Learning, creating and using knowledge: Concept maps™ as facilitative tools in schools
and corporations. Mahwah, NJ: Erlbaum.
Paas, F., Renkl, A., & Sweller, J. (Eds.). (2003). Cognitive load theory. Educational Psychologist, 63(1), Whole
Issue.
Paivio, A. (1986). Mental representations: A dual coding approach. Oxford, England: Oxford University Press.
Polson, M. C. (1993). Cognitive theory as a basis for instructional design. In J. M. Spector, M. C. Polson, & D.
J. Muraida, Automating instructional design: Concepts and issues (pp. 5-22). Englewood Cliffs, NJ:
Educational Technology Publications.
Salas, E., & Fiore, S. (Eds.) (2004). Team cognition: Understanding the factors that drive process and
performance. Washington, DC: American Psychology Association.
Schank, R. C., Berman, T. R. & Macperson, K. A. (1999). Learning by doing. In C. M. Reigeluth (Ed.),
Instructional design theories and models: A new paradigm of instructional theory (Vol. II) (pp. 161–181).
Mahwah, NJ: Lawrence Erlbaum Associates.
Schvaneveldt, R. W. (Ed.) (1990). Pathfinder associative networks: Studies in knowledge organization.
Norwood, NJ: Ablex.
Schwartz, D., Lin, X., Brophy, S., & Bransford, J. D. (1999). Toward the development of flexibly adaptive
instructional designs. In C. M. Reigeluth (Ed.), Instructional design theories and models: A new paradigm of
instructional theory (Vol. II) (pp. 183–213). Mahwah, NJ: Lawrence Erlbaum Associates.
Seel, N. M. (2001). Epistemology, situated cognition, and mental models: Like a bridge over troubled water.
Instructional Science, 29(4-5), 403-427.
Learning Systems Institute, Florida State University, Suite 4600-C University Center, Tallahassee, FL 32306-2540
(PHONE): (850) 644-2570 (FAX): (850) 644-4952 URL: WWW.LSI.FSU.EDU
References (4/4)
•
•
•
•
•
•
•
•
•
•
•
•
•
Seel, N. M. (2003). Model-centered learning and instruction. Technology, Instruction, Cognition, and Learning,
1(1), 59-86.
Seel, N., Al-Diban, S., & Blumschein, P. (2000). Mental models and instructional planning. In J. M. Spector & T.
M. Anderson (Eds.), Integrated & Holistic Perspectives on Learning, Instruction & Technolog (pp. 129-158).
Dordrecht, The Netherlands: Kluwer.
Spector, J. M. (2000, Fall). Trends and issues in educational technology: How far we have not come. Update
Semiannual Bulletin 21(2).
Spector, J. M. (2001). A philosophy of instructional design for the 21st century? Journal of Structural learning
and Intelligent Systems 14(4), 307-318.
Spector, J. M., & Anderson, T. M. (2000). Integrated and holistic perspectives on learning, instruction and
technology: Understanding complexity. Dordrecht: Kluwer.
Spector, J. M., & Koszalka, T. A. (2004). The DEEP methodology for assessing learning in complex domains
(Final report the National Science Foundation Evaluative Research and Evaluation Capacity Building).
Syracuse, NY: Syracuse University.
Spector, J. M., Dennen, V. P., & Koszalka, T. A. (2005, July). Causal maps, mental models and assessing
acquisition of expertise. Proceedings of the 11th Annual Human Computer Interaction International
Conference, Las Vegas, Nevada, July, 2005.
Spiro, R. J., Feltovich, P. J., Jacobson, M. J., & Coulson, R. L. (1991, May). Cognitive flexibility, constructivism,
and hypertext: Random access instruction for advanced knowledge acquisition in ill-structured domains.
Educational Technology, 31 (5), 24-33.
Stoyanova, N., & Kommers, P. (2002). Concept mapping as a medium of shared cognition in computersupported collaborative problem solving. Journal of Interactive Learning Research, 13(1/2), 111-133.
Suppes, P. (1978). Impact of research on education: Some case studies. Washington, DC: National Academy
of Education.
Sweller, J. (2003). Evolution of human cognitive architecture. The psychology of learning and motivation, 43,
215–266.
Taricani, E. M., & Clariana, R. B. (2006). A technique for automatically scoring open-ended concept maps.
Educational Technology Research &Development, 54(1), 65-82.
van Merriënboer, J. J. G. (1997). Training complex cognitive skills: A four-component instructional design
model for technical training. Englewood Cliffs, NJ: Educational Technology Publications.
Learning Systems Institute, Florida State University, Suite 4600-C University Center, Tallahassee, FL 32306-2540
(PHONE): (850) 644-2570 (FAX): (850) 644-4952 URL: WWW.LSI.FSU.EDU