Transcript Knowledge Elicitation Techniques

Uncovering the Problem-Solving Process:
Cued Retrospective Reporting,
Eye Tracking, and Expertise Differences
Tamara van Gog, Fred Paas, & Jeroen J. G. van Merriënboer
I3CLEPS Workshop/Mini-conference,
August 29, 2005
Overview
Experiment:
- Theory
- Design
- Comparison of 3 verbal methods
- The 3 methods & expertise differences
- Uncovering expertise-related performance
differences through eye movement data
- Present limitations and future research
- Discussion
Theory
Use of process-tracing techniques to uncover
problem-solving processes in order to advance /
inform:
- Psychological theory
- Expert systems
- User-system interaction,
But also
- Instructional design
e.g., design of process-oriented worked examples
Theory
From the literature (Kuusela & Paul, 2000; Taylor & Dionne, 2000):
+ of concurrent reporting (“think aloud”):
more information on actions taken
+ of retrospective reporting:
more information on rationale for actions taken and
strategies that control the process
Needed: A method that combines + & + :
Cued retrospective reporting based on a record of eye
movements & mouse/keyboard operations?
Design
Within-subjects, 26 participants, electrical circuits
troubleshooting tasks:
Seq.
1
2
3
4
CR 1+2
CRE 3+4
RR 5+6
CRR 7+8
Condition + Tasks
CRE 3+4 RR 5+6
CRR 7+8 CR 1+2
CR 1+2
CRR 7+8
RR 5+6
CRE 3+4
CRR 7+8
RR 5+6
CRE 3+4
CR 1+2
CR = concurrent reporting; CRE = concurrent reporting with eye
tracking; RR = retrospective reporting; CRR = cued retrospective
reporting.
Comparison of 3 Methods: Hypotheses
1. Concurrent reporting (CR):
more ‘action’ info than RR
2. Retrospective reporting (RR):
more ‘why’, ‘how’, & ‘metacognitive’ info than CR
3. Cued retrospective reporting (CRR):
-> more ‘action’ than RR
-> more ‘why’, ‘how’, & ‘metacognitive’ than CR
Comparison of 3 Methods: Analyses
Segmentation based on speech  sentences /
utterances (preceded & followed by a pause)
Coding scheme task-oriented main categories:
‘action’
‘why’
‘how’
‘metacognitive’
20% of protocols scored by 2 raters: kappa = .79
 good; proceeded with 1 rater
Analyses on nr. of codes on main categories,
obtained by summing codes on subcategories
Comparison of 3 Methods: Results
Friedman Tests with Conover (1999) comparisons
CR vs RR:
as hypothesized: ‘action’  CR >RR
however: ‘why’ and ‘how’  CR > RR, and
‘metacognitive’ CR = RR
CRR vs RR:
as hypothesized: ‘action’  CRR >RR
‘why’: CRR = RR
‘how’ and ‘metacognitive’: CRR > RR
Expertise Differences: Explorative
5 “highest” and 5 “lowest” expertise participants
(from 26). Determined by performance efficiency:
“highest”: higher performance, lower mental effort,
lower time-on-task
“lowest”: lower performance, higher mental effort,
higher time-on-task
- Differences in elicited information?
- Differences in preferences/experiences?
(open-ended debriefing questions)
Expertise Differences: Elicited Information
Differences in elicited information?
(Mann-Whitney U Tests)
CR:
‘how’ and ‘metacognitive’ info: “lowest” > “highest”
RR:
‘why’ info: “highest”> “lowest”
‘how’ info: “lowest” > “highest”
CRR:
‘action’ and ‘metacognitive’ info: “lowest” > “highest”
Expertise Differences: Experience
Differences in preferences/experiences?
“lowest”:
experience: CR 
(4/5)
preference: CRR > CR & RR
(4/5)
“highest”:
no differential experiences/preferences
Mediating factors mentioned re. experience /
preference, by both “lowest” and “highest”:
- Time-on-task
- Cue
Studying Expertise-Related Performance
Differences: Eye Movement Data 1
Eye fixation data provide insight in the allocation of
attention, and hence differ with expertise
Research use: provide information about the
problem-solving process at a finer grained level
than verbal protocols?
(Ultimate) educational use: guiding novices’
attention?
1
Data from Van Gog, Paas, & Van Merriënboer (2005),
Applied Cognitive Psychology
Eye Movement Data: Participants & Procedure
Same 5 “lowest” and 5 “highest” expertise participants
Data collected in first 3 phases of the process:
1. Problem orientation (until pushing switch to
observe circuit behavior)
2. Problem formulation and action decision
3. Action evaluation and next action decision
% time spent on phase, mean fixation duration (MFD),
and in 1st phase fix. related to faults
Task
Short-circuit
Only 3 Volt
Eye Movement Data: Results
Phase 1: problem orientation
(Mann-Whitney U Tests, 2-tailed, α = .10)
% of time:
MFD:
% fixations on battery:
Gaze switches short-circuit:
“highest” > “lowest”
“lowest” > “highest”
“highest” > “lowest”
“highest” > “lowest”
(NB: only trend)
Eye Movement Data: Results
Phase 2: problem formulation & action decision
(Mann-Whitney U Tests)
% of time:
MFD:
“highest” = “lowest”
“highest” = “lowest”
MFD First ½: “highest” > “lowest”
MFD Second ½: “highest” = “lowest”
Eye Movement Data: Results
Phase 3: action evaluation & next action decision
(Mann-Whitney U Tests)
% of time:
MFD:
“highest” > “lowest”
“highest” = “lowest”
MFD First ½: “highest” = “lowest”
MFD Second ½: “highest” = “lowest”
Eye Movement Data: Results
Mean Median Fixation Duration (ms)
MFD over phases (Friedman + Nemenyi post-hoc):
n.s. for “lowest”; “highest” 1 < 2.1., 2.2., 3.2 & 2.1 >3.1
400
350
300
250
lower expertise
200
150
higher expertise
100
50
0
1
2.1
2.2
Phases
3.1
3.2
Limitations
- CRR and fabrication?
- Cue: combination of eye movements AND
mouse/keyboard operations
- Only quantitative analyses of protocols
- Eye movement data: distinction of phases
- Performance efficiency measure:
very relative distinction (lowest and highest within
this group of participants)
- Small nr of participants in analyses related to
expertise differences
Future Research
- Qualitative differences between CRR and RR?
- Cue: different effects with only eye movements OR
mouse/keyboard operations?
- Cue: technical optimization?
- (RR/)CRR: effects of other prompts?
- Further study of performance efficiency measure to
distinguish expertise levels
- Replications with larger N
Thank you for your attention!
[email protected]