Transcript The Portal Assessment Design System
Some Implications of Expertise Research for Educational Assessment
Robert J. Mislevy, PhD Professor of Measurement & Statistics University of Maryland Keynote address at the 34th International Association for Educational Assessment (IAEA) Conference, Cambridge University, September 8, 2008.
September 8, 2008 IAEA 2008 Slide 1
Introduction
These are exciting times in assessment.
Developments in psychology and technology.
Insights from expertise research with implications for assessment design.
Let’s start with a quiz.
September 8, 2008 IAEA 2008 Slide 2
What is this a picture of?
( http://www.optillusions.com) September 8, 2008 IAEA 2008 Slide 3
Which cards need to be turned over?
Each card has a letter on one side and a number on the other. Consider the rule “If there is a vowel on one side, there is an even number on the other.” Which cards do you need to turn over to make sure the rule is not violated? (Wason,1966) September 8, 2008
E 4 T 7
IAEA 2008 Slide 4
A Little Story
I will read a little story and ask you some questions about it.
September 8, 2008 IAEA 2008 Slide 5
Was this sentence in the story?
1.
2.
3.
Mary gave Ed a stuffed hippo. Harold received ninety gifts.
Ralph gave Tina a toy car. September 8, 2008 IAEA 2008 Slide 6
There were five sentences in the story.
In order, what were the second-to-last words in each of the sentences?
September 8, 2008 IAEA 2008 Slide 7
What is this a picture of?
( http://www.optillusions.com) September 8, 2008 IAEA 2008 Slide 8
Which cards need to be turned over?
Each card has a letter on one side and a number on the other. Consider the rule “If there is a vowel on one side, there is an even number on the other.” Which cards do you need to turn over to make sure the rule is not violated? (Wason,1966) September 8, 2008
E 4 T 7
IAEA 2008 Slide 9
Whose IDs do we need to check?
Each person has an age and a beverage. Consider the rule “If you are under 21, your beverage must be non-alcoholic.” Which people do you need to check to make sure the rule is not violated? 15 Year old September 8, 2008 Water drinker IAEA 2008 30 Year old Wine drinker Slide 10
1.
2.
3.
Was this sentence in the story?
Mary gave Ed a stuffed hippo. Harold received ninety gifts.
Ralph gave Tina a toy car. YES NO NO
Ralph gave a toy car to Tina.
September 8, 2008 IAEA 2008 Slide 11
There were five sentences in the story.
In order, what were the second-to-last words in each of the sentences?
One, stuffed, to, four, holiday.
September 8, 2008 IAEA 2008 Slide 12
Limitations and Difficulties
Processing limitations Limited attention Limited working memory Knowledge limitations Not knowing what information is relevant Don’t know how to integrate information Not knowing what to expect Not knowing what to do and when to do it Lack of production proficiency September 8, 2008 IAEA 2008 Slide 13
Capabilities
Reasoning in terms of patterns Many patterns simultaneously (language) Can make perception, procedures, strategies automatic with practice Can think about our thinking (metacognition) Benefit from procedures, methods, tools, external knowledge representations
Expertise as the circumvention of human processing limitations (Salthouse, 1991)
September 8, 2008 IAEA 2008 Slide 14
A Closer Look at Cognition
Knowledge as patterns, at many levels… Assembled to understand, to interact with, and to create particular situations in the world Developed, strengthened, modified by use Associations of all kinds, including applicability, affordances, procedures, strategies, affect “The user’s knowledge of the language rules is interlocked with his knowledge of when, where, and with whom to use them” (R. Ellis, 1985) September 8, 2008 IAEA 2008 Slide 15
Text More focused research areas within cognitive psychology today differ as to their foci, methods, and levels of explanation. They include perception and attention, language and communication, development of expertise, situated and sociocultural psychology, and neurological bases of cognition.
Walter Kintsch’s CI Theory of Reading Comprehension
LTM Situation Model Text base Context Context 1 •E.g., reading tasks in Occupational English Test (OET; McNamara, 1996) call upon patterns re language, but also genre, medical knowledge, use of information in clinical settings A relevant pattern from LTM may be activated in contexts but not others (e.g., physics models, use of conditionals).
If a pattern hasn’t been learned, it won’t be activated (although it may get constructed in the interaction).
IAEA 2008 Slide 16 September 8, 2008
Text More focused research areas within cognitive psychology today differ as to their foci, methods, and levels of explanation. They include perception and attention, language and communication, development of expertise, situated and sociocultural psychology, and neurological bases of cognition.
Walter Kintsch’s CI Theory of Reading Comprehension
LTM Situation Model Action Text base Context Context 1 Context 2 September 8, 2008 IAEA 2008 Slide 17
Text More focused research areas within cognitive psychology today differ as to their foci, methods, and levels of explanation. They include perception and attention, language and communication, development of expertise, situated and sociocultural psychology, and neurological bases of cognition.
Walter Kintsch’s CI Theory of Reading Comprehension
LTM Situation Model Action Text base Context Context 1 Context 2 September 8, 2008 IAEA 2008 Slide 18
Text
Walter Kintsch’s CI Theory of Reading Comprehension
LTM Situation Model Action Text base Context More focused research areas within cognitive psychology today differ as to their foci, methods, and levels of explanation. They include perception and attention, language and communication, development of expertise, situated and sociocultural psychology, and neurological bases of cognition. Context 2 Context 3 September 8, 2008 IAEA 2008 Slide 19
Expertise Research
Cognitive task analysis (e.g., Simon & Chase) Compare experts & novices in replicable conditions What knowledge is needed? How is it represented? How is it used? What makes tasks hard? Ethnographic research (e.g., Lave) Expertise in situ “Critical incident” studies (NBME) Replication possible in simulations Flight simulators, football kick in video situations September 8, 2008 IAEA 2008 Slide 20
Expertise Research
Experts organize their knowledge effectively Perceive / understand / act in terms of fundamental principles rather than surface features (Chi, Feltovich, & Glaser) Importance of interaction with situation Cycles of “perceive / understand / act” External knowledge representations (KRs) Nexus of info-processing & sociocultural POV Supported cognition / distributed cognition September 8, 2008 IAEA 2008 Slide 21
So… How do you use this improved understanding of the nature and acquisition of expertise to design and conduct assessments?
September 8, 2008 IAEA 2008 Slide 22
Assessment Arguments
What complex of knowledge, skills, or other attributes should be assessed? What behaviors or performances should reveal those constructs?
What tasks or situations should elicit those behaviors? (Messick, 1994) September 8, 2008 IAEA 2008 Slide 23
Examples
The Architectural Registration Examination (ARE) Architectural design; CAD-like environment DISC simulator Simulations for problem-solving in dental hygiene NetPASS (Cisco) Computer network design & troubleshooting September 8, 2008 IAEA 2008 Slide 24
ARE Example (Irv Katz, ETS)
To replace 10-hour hand-drawn design problem Reflects changing of profession to CAD Premium on thinking, not drawing Planning the firestation site September 8, 2008 IAEA 2008 Slide 25
An Example of a Task Prompt for the ARE September 8, 2008 IAEA 2008 Slide 26
AB D September 8, 2008 Slide 27
September 8, 2008 Slide 28
NCARB Example (Irv Katz, ETS)
Differences between novices and experts • Success rate: 98% vs. 88% • Planning time & execution time • Patterns of revision involving rework Implications for task design:
Constraints
• Number • Variation in importance/difficulty • Degree of conflict • Implicit constraints? (c.f. writing expertise) September 8, 2008 IAEA 2008 Slide 29
Design Patterns
“Design under constraints” is common to many domains: e.g., engineering, assessment design, wedding planning, apparel design Could define a Design Pattern with these as foci What are characteristics of performances (i.e., observables) that evidence these knowledge / capabilities / attunements? What are characteristics of situations that elicit these observables?
September 8, 2008 IAEA 2008 Slide 30
Other design pattern possibilities
Troubleshooting finite systems Medical diagnosis Inquiry cycle Model-based reasoning September 8, 2008 IAEA 2008 Slide 31
DISC Example
The Dental Interactive Simulations Corporation (DISC) The DISC Simulator & Scoring Engine Cognitive Task Analysis to support design rationale September 8, 2008 IAEA 2008 Slide 32
Goals of the DISC CTA
What are the kinds of knowledge and skills that hygienists call upon to solve problems and make decisions in dental hygiene?
What do they say and do that gives you evidence about their use of this knowledge?
What kinds of situations call upon this knowledge?
How do you make re-usable schemas to evoke evidence and construct ‘stories’ around these recurring patterns?
September 8, 2008 IAEA 2008 Slide 33
Using Disparate Sources of Information Expert Novice Tries to use all sources of information all the time. Constructs a model of the patient, with each source of information an imperfect and incomplete window on some aspect of the total situation. Exhibits movement back and forth between sources (resources and personal knowledge/experience), trying to fit the partial clues together into a unified whole. Uses single information sources in isolation. Compartmentalized use of information and failure to integrate information across sources. September 8, 2008 IAEA 2008 Slide 34
Formulating Problems & Hypotheses Expert Forms problems and generates hypotheses using efficient, focused, and targeted action. Uses forward and deductive reasoning in formulating problems. Thinks strategically and functions within the problem space. Efficiently generates and prunes search trees. Novice Uses forward reasoning but scope and depth of knowledge to support forward reasoning is limited. Generates a search tree but possesses limited tools for pruning the search tree. September 8, 2008 IAEA 2008 Slide 35
Vocabulary and Language Usage Expert Novice September 8, 2008 Retrieves and uses appropriate, clear, sophisticated, accurate, and precise terminology. Uses spontaneous declaration and can retrieve without effort. Creates messages that are easily understood by the target audience. Strained, unreliable, and effortful retrieval of terminology. Hesitant, delayed, and labored responses due to time required to process and lack of knowledge. Uses terminology unsystematically. Prone to canned speeches and explanations without awareness of individual patients’ needs. IAEA 2008 Slide 36
An example with the Cisco Learning Institute: The NetPass Prototype
Create on-line performance assessment of networking skills Focus on learner feedback rather than high stakes testing Expert/Novice studies ground design September 8, 2008 IAEA 2008 Slide 37
The task starts with a scenario and description of goals
September 8, 2008 IAEA 2008 Slide 38
To capture their mental model of the network, we ask them to draw the network with a diagramming tool September 8, 2008 IAEA 2008 Slide 39
The diagram is created by dragging and dropping icons September 8, 2008 IAEA 2008 Slide 40
Configuring the devices September 8, 2008 IAEA 2008 Slide 41
When students are done, they press Submit”… September 8, 2008 IAEA 2008 Slide 42
And the graphical representation is converted into an text representation in XML format September 8, 2008 IAEA 2008 Slide 43
The text file is scored following detailed rules, which result in characterizations of the work September 8, 2008 IAEA 2008 Slide 44
And feedback is created for the student… both diagnostic and a summary profile.
September 8, 2008 IAEA 2008 Slide 45
Conclusion
Insights from expertise research can improve the practice of assessment, and support deeper learning.
Doing so requires a deeper understanding of assessment design.
» More explicit arguments and representations » Generativity, re-usability, and inter-operability “Too many notes” ? (Emperor Joseph II) Suitable conceptual frameworks, tools, and exemplars are now beginning to appear.
September 8, 2008 IAEA 2008 Slide 46