Transcript DickClark - Pittsburgh Science of Learning Center

Cognitive Task Analysis
Dick Clark
Center for Cognitive Technology
Rossier School of Education
Keck School of Medicine
University of Southern California
[email protected] - www.cogtech.usc.edu
PSLC October 15, 2013
Center for Cognitive Technology
Topics
1.
Why the interest in Cognitive Task Analysis (CTA)?
2.
What evidence supports CTA’s use in education?
3.
How is it implemented? Examples? Exceptions?
4.
Next steps in research.
2
Why Cognitive Task Analysis?
• Methods for identifying the cognitive strategies used by experts and
novices to perform complex tasks.
• Supports decisions on WHAT to teach - not how.
• Important because of evidence that +/- 70% of expert decisions and
many actions are implicit – automated and nonconscious -- in order to
circumvent limits on WM.
• When CTA used to design instruction, 1σ increase in learning and
.5σ decrease in time to learn.
• Preliminary evidence of increases in task self-efficacy and
persistence (decreased dropout) in higher education courses.
3
Brief History of CTA
• Recent developments in long history of Task Analysis
• Gilbreth’s 1890 – 1930 QUERTY keyboard, 3X bricklaying
• Crandall & Gretchell-Leiter (1993) identified 30% more indicators of
distress in premature babies with Klein’s CTA (Crandall, Klein &
Hoffman, 2006).
• Chao & Salvendy (1994) examined four different methods of
capturing the strategies experts use for three debugging tasks.
• Average of 40% procedural steps and 30% explanations
• Increased to average of 80% of steps after interviewing 6 experts
• Cost-benefit diminishes beyond 4 to 6 experts
• Why do experts recall different IF – THEN steps?
4
Chao & Salvende, (1994)
5
Decision step recall increase with more experts
Chao & Salvende, (1997) Figure 4
6
PhD Students (intermediates) vs. Psychology Faculty
Feldon (2010)
7
70% Decisions Missing and 4 to 6 Experts to Remedy
• Other studies, including partial replications of Chao & Salvende
• Trauma Surgeons (Campbell, 2010; Crispen 2010; Sullivan et al,
2011; Velmahos et al, 2006)
• Psych faculty teaching experimental design (Feldon, 2010)
•
Expert instructors consistently describe 30% of decisions but but about
60% of actions when teaching.
• With CTA the decisions identified reached 90 to 100% with four to
six experts.
•
Most of our studies focused on surgical procedures because of disputes
about “expertise” and surgeon’s legally required to report mistakes.
8
Variation in SME Action and Decision Steps
(Crispen, 2010 – Cricothyrotomy procedure)
9
Percent of decisions identified with each new SME
Crispen, 2010; Figure 6
10
Expert Knowledge Provided During Teaching
Sullivan, Yates, Clark, Green, Tang, Cestero, Plurad, Lam & Inaba (In Press)
11
Unexpected Result: Controversial CVC Procedure
Figure 4: Yates, Sullivan & Clark (2011)
12
Exception: Two CTA studies of catheter procedure
Clark, 2014)
13
CTA in Instructional Design
14
CTA in Instructional Design
Gucev (2012) randomized double blind experiment on CTA in
Ultrasound Guided Regional Anesthesia
15
Gucev CTA Study Design and Results
• Both experimental and control groups:
• Same tasks and conceptual knowledge required by the American
and European Societies of Regional Anesthesia.
• Same instructional methods (conceptual knowledge first then
demonstration and practice).
• Participants were second and third year medical students.
• Experimental group received CTA content for Societies tasks and the
control group received the approved Societies content and tasks.
• Results – benefits of CTA on learning and performance over controls:
• Declarative knowledge effect size d = 1.43 (42%)
• Procedural knowledge effect size d = 1.65 (45%)
• Effect size for the time for task performance was d = -1.12 (-37%)
16
Benefit of Cognitive Task Analysis?
Instruction based on CTA is consistently more effective than
Behavioral Task Analysis or “self report”.
• Hoffman (1998) 38% better with CTA – changed textbooks on
prenatal infections.
• Velmahos et al (2002) 35% better surgical decisions, improved
transfer, 25% quicker, no important errors.
• Tofel-Grehl & Feldon (2013) meta analysis (57 comparisons).
• Hedges g = .88 (31%) overall but g =1.56 (44%) for PARI-type
CTA methods and g = .39 (16%) for Klein’s CDM method.
• Biology lab course significantly better performance and lower
dropout (Feldon et al, 2010; Feldon & Stowe, 2009).
17
CTA vs. Traditional Instruction - Biology Lab Reports
Universal Lab Report Rubric Criteria
Treatment
Mean
(SD)
Control
Mean
(SD)
F
p-value
0.90 (.50)
0.77
(0.48)
4.378
.037*
0.43
(0.52)
0.28
(0.44)
6.171
.014*
0.70
(0.63)
0.54
(0.57)
4.703
.031*
0.31
(0.46)
0.21
(0.40)
3.463
.064
2.34 (1.49)
1.78
(1.37)
9.501
.002**
Discussion: Conclusions based on data
Conclusion is clearly and logically drawn from data provided. A
logical chain of reasoning from hypothesis to data to conclusions is
clearly and persuasively explained.
Discussion: Alternative explanations
Alternative explanations are considered and clearly eliminated by
data in a persuasive discussion.
Discussion: Limitations of design
Limitations of the data and/or experimental design and
corresponding implications discussed.
Discussion: Implications of research
Paper gives a clear indication of the implications and direction of
the research in the future.
Discussion: Total Score
Feldon et al. (2010); Feldon & Stowe (2009)
18
Biology 101 Attrition (Withdraw Rates)
16
14
12
10
8
6
4
2
0
CTA
Control
Condition Condition
Treatment
Control
Fisher’s
Exact
(2-sided)
2-week
Enrollment
142
172
-
Final
Enrollment
140
158
-
Overall
Dropouts
2
14
p=.005**
Biology
Majors
1
3
p=.334
Non-Majors
1
11
p=.010**
Women
1
8
p=.041*
Men
1
6
p=.072
Feldon et al. (2010); Feldon & Stowe (2009)
19
CTA with Online Faculty at Kaplan University
CTA with four of the most effective online faculty teaching intro courses.
Plan:
1. Identify the strategies reported by most of the experts interviewed.
2. Translate them into a Likert-type values survey that would be offered
to a large random sample of 280 online instructors in different fields.
“How likely are you to advise a new instructor to use ……?”
3. Correlate the rankings of the items by individual faculty with their
student’s learning and retention data.
4. Use the items that predicted the greatest success to help hire new
faculty, train existing faculty and evaluate the results.
20
Results of Kaplan U Survey Based on CTA
• DROPOUT: With every .5 increase in survey ranking of items, student
dropout decreased 1.6% (a low score of 1 predicts a dropout rate of
41% whereas a score of 5 predicts a significantly lower rate of 29.4%)
• GPA: With every .5 increase in survey ranking of items, GPA increased
about .15 points. A score of 3.0 on the survey would predict a GPA of
2.1 whereas a score of 5.0 on the survey would predict a GPA of 2.5.
• RETENTION: Faculty who valued making themselves available by
phone, calling students who were not actively participating and who
tried to help students recover from problems had an 81% chance of
higher retention rates in academic programs
21
Example: Kaplan Career Services Advisors
• Content based on a CTA of
career service advisors with
highest placements
• “Kaplan Way” design and
delivery
• Randomized controlled
study (treatment n: 63;
control n: 67)
• 15% improvement in
performance (key metric:
job placements)
22
Cost of CTA?
Taken from Clark, 2014
23
What is Cognitive Task Analysis?
•
100 + strategies for capturing the implicit and explicit strategies
experts use to perform complex tasks based on Newell & Simon’s
“Human Problem Solving” (1972).
•
•
Goal is to enhance human or machine learning and performance.
Four types of CTA processes (Marsha Lovett’s 2x2):
Descriptive
Empirical
Think-aloud
of novice.
Difficulty
Factors
Assessment.
Think-aloud
of expert.
Prescriptive
DFA
Theoretical
Cognitive
modeling of
errors, informal
strategies.
Cognitive
modeling of
success,
normative
strategies.
24
What is Cognitive Task Analysis?
• Yates (2007) sorts prescriptive CTA methods by outcome: Those that
capture declarative (what) and/or procedural (how) and/or Strategic
(when) expert knowledge.
• Our emphasis is on a blending of the three varieties of CTA methods
that capture all three types of knowledge identified by Tofel-Grehl &
Feldon (2013) meta analysis as the most productive:
1. CDM (Critical Decision Method; Klein et al, 1989).
2. PARI (Precursor, Action, Result, Interpretation; Hall et al, 1995).
3. CPP (Concept, Process, Principle, Procedure; Clark, 2014).
25
What is Cognitive Task Analysis?
•
Three to six experts selected because they are consistently and
recently successful (not simply “experienced”) and NOT instructors.
•
Evidence that each expert has different implicit knowledge
about same tasks and that instructors invent “superstitious”
steps.
•
Results of interviews corrected by experts and edited into one “gold
standard” approach for novices based on maximum efficiency and
accuracy.
•
Range of problem examples and performance scenarios are also
collected from experts for use in instruction.
•
Goal is to develop a succinct and accurate procedure (when and
how) to perform as basis for demonstrations and practice exercises.
•
Emphasis on IF – THEN decisions.
26
What is Cognitive Task Analysis? Six Tasks
Interview experts with recent, consistently successful experience who
are NOT full time instructors.
Task 1. Outline sequence of tasks “as performed on the job”
–
–
–
If no necessary sequence, teach easier tasks before
more difficult tasks.
Place prerequisite knowledge first.
If safety is an issue – “Safety first”.
“In about 30 seconds, describe the actions and decisions
you implement to achieve the goal of this task.
27
Surgery Task Sequence
Task 1
Task 2
Task 3
Task 4
Task 5
Select catheter & choose insertion site
Immobilize patient, prepare site
and insert catheter needle
Introduce guide wire and
incise skin around wire insertion
Introduce intravenous dilator and catheter
Prepare lumens and secure line with
non-absorbable sutures
28
Example Course Outline: Examining patent applications
Preparing search
Performing substantive
examinations
reports
Analyzing
applications
Performing
searches
Comparing
documents with
invention
Writing further communication(s)
or refusal
Evaluating
search
results
Selecting relevant
documents
Finding lack of unity
Determining claimed invention
Determining described invention
Determining Using
search
search
strategies tools
Identifying
relevant EPC
requirements
Re-examining
applications
Discussing with applicant
Determining
Determining
claimed subject
novelty &
matter
inventive steps
Issuing communications
or votes (including preexamination results)
Examining amendments
Determining Classifying
mean features applications
of invention
Writing preexamination results
29
What is Cognitive Task Analysis? Six Tasks
Interview 2-3 experts with recent, successful experience
Task 2) For each task, describe clearly enough so
trainees can read and apply
• Context (Where, When)
• Condition or Cue (What Starts the task)
• Sequence of Actions and Decisions (How)
that
Finish this step before going on to step 3 –
• Tasks or task sequence may change when you see
performance steps
• Can estimate time required to train at this point
30
Task 2: Actions and Decisions
• Explain each action in the sequence you perform them
• Things people do (start with action verbs)
• Explain each decision
• Describe as “IF” and “THEN” sentences
MOST IMPORTANT: Write steps clearly enough so that a trainee
could read and then do what you are describing.
31
Catheter Placement Steps -Decision Procedure
Start by deciding among three sites for catheter placement.
1. IF the neck is accessible and can be moved, and the head and neck are
free of excessive equipment, THEN select jugular placement.
1. IF neck is inaccessible or cannot be moved, THEN select subclavian.
1. IF the subclavian veins are thrombosed and there is no injury to the IVC,
THEN select femoral vein placement.
32
Catheter Placement Steps
Dilator and catheter
insertion for Triple Lumen
catheters:
• Step 13A: Thread the guide
wire into the tip of the dilator.
• Direct the dilator down the
wire slowly and through
subcutaneous tissue (3 – 4
cm).
33
33
Patent Examination Procedure Example
start
start
Build first impression, by looking at drawings, main
claims, and the first page
Read application
If you read an application, then
• start with studying the drawings and put them next
to the text.
• start with reading the main claims.
Specify classes,
keywords,
databases
Search with
program to retrieve
documents
Perform
examination
yes
Get a grip on an application by reading the whole
thing minus claims
no
Are there
major
defects?
Write draft
communication
Write draft vote
Write search report
stop
If you read an application, then
• highlight/underline passages that refer to prior
art, technical effects, and formal defects.
• first study any independent claims if reference is
made to such claims in the text.
• focus attention on detailed descriptions related to
drawings.
• use any references that are made to prior art as
input or starting point for your search.
• only study dependent claims after you understand
the application.
stop
34
What is Cognitive Task Analysis? Six Tasks
Task 3) Collect task-related information about:
• Supplies and equipment (and location)
• Performance standards (speed, quality)
• Common novice performance errors
• Reasons (Personal Benefits and Personal Risks)
35
What is Cognitive Task Analysis? Six Tasks
Task 4) Identify conceptual knowledge related to procedure:
•
•
•
•
Facts (required statements about anything)
Concepts (define new terms – get examples)
Processes (how things work)
Principles (what causes things to happen)
Conceptual knowledge is important IF people must remember
something to tell someone else about it – or IF they must apply it
to adjust a procedure to solve an unexpected or novel problem
36
Supportive Conceptual Knowledge
Knowledge
Types
Presentation During Instruction
Type of
Information
Example
Practice and Assessment During Instruction
Objective is to
Remember
Demonstration of
when and how to
perform
Recall when to
use; Recall
action and
decision steps
Proxy for Remember
Procedure
Fact
Statement of
fact
Statement of fact
Recall fact
Recognize fact when
presented with
distractors
List of
defining
attributes
Examples and Non—
examples of concept
List defining
attributes
verbally or in
writing
Recognize defining
attributes when
presented with
distractors
Identify or generate
examples and nonexamples
Critique someone else’s
identification or
generation of examples
List of phases,
events and
causes at each
phase
Examples;
simulations of
phases, events, and
causes
Recall phases,
events, and
causes
Recognize phases,
events, and causes;
Recall missing phases,
events, and causes
Identify causes of faults in a
process;
Predict events in a process
Critique someone else’s
description of causes or
prediction of events in a
process
Statement of
cause and
effect
relationship
Examples,
demonstration,
simulation of cause
and effect
relationship
Decide if principle applies;
Predict an effect;
Apply the principle to
solve a problem, explain a
phenomenon or make a
decision
Critique someone else’s
application of the
principle to solve a
problem, explain a
phenomenon or make a
decision
(Terms with
definitions and
example)
Process
(How something
works)
Principle
(Cause and
effect
relationship)
Knowledge
Integration
Knowledge
Transfer
Explain the interconnections among
conceptual knowledge components, or
the conceptual foundation of procedures,
or the procedural implementation of
conceptual knowledge components
Multiple and varied contexts for
examples
Recall the
principle
Recognize the
principle;
Recall missing
elements of the
principle
Decide when to use;
Perform the steps
(actions and decisions)
Proxy for Use if
application is
impossible **
When to use;
List of action
and decision
steps
Concepts
Reorder steps;
Recall next or missing
steps
Objective is to Use or
Apply**
Critique performance
or output of actions
and decisions
Recall fact in task context
Opportunities (including instructions, templates, rubrics) to self-explain, discuss, present, describe or select their
reasoning about interconnections among knowledge components, for example the principle(s) that justify the
application of a procedure.
Multiple and varied contexts for practice and assessment.
Opportunities for students to explain how they would use the knowledge in other contexts
© 2011 Atlantic Training Inc.
37
What is Cognitive Task Analysis? Six Tasks
Task 5: Collect five authentic problems trainees will learn to solve
•
•
•
•
One for demonstration during training
One for practice and feedback
One for progress check
Two for competency tests
38
What is Cognitive Task Analysis? Six Tasks
Task 6) Give CTA document from SME A to SME B, C, D, E, etc.) to
“correct”.
• Flynn (2013) found reviews of one CTA interview by 3 SMEs
more efficient and effective than 4 complete interviews.
• Develop a “gold standard” CTA for training and/or job aid
development – use language novices will understand.
• Pull CTA into training design that includes:
• Performance objectives and reasons
• References to prior knowledge (analogies, examples)
• Conceptual knowledge underlying procedure
• Demonstration of procedure (worked example)
• Part and whole task practice with feedback
39
CTA Problems and Exceptions
•
Cannot use Expert-based CTA IF:
•
•
•
•
No experts available and/or
New (novel) tasks, technology, science, processes, or
If “experts” not consistently succeeding at task
Problems using CTA:
• Analyst training requires many hours of practice.
• “Clients” resist added front end expense of structured interviews
and/or have used an ineffective CTA method in the past.
• Experts sometimes hold back their “secret sauce” and/or reject the
gold standard believing it demeans their skills.
40
Next Steps in CTA Research
•
•
•
•
•
Need to focus research on most effective of the 100+ CTA methods.
Clear operational definition of CTA methods.
Data mining to extend and/or replace structured interviews.
Why are different experts aware of different tasks and steps?
Better understanding of how declarative and procedural knowledge
interact during task performance (as task elements change).
• Cost-effectiveness of different types of CTA for instruction.
• Analysis of why CTA:
• Decreases time to learn,
• Increases self-efficacy,
• Increases persistence and
• Increases transfer.
41
42
References
Evidence for most claims and references in this presentation
and a review of the research on CTA can be found at:
www.cogtech.usc.edu
Access the “Publications” tab
43