Transcript Windows Into The Mind

Windows Into The Mind
Richard J. Shavelson & Maria Araceli Ruiz-Primo
Stanford University
Invited Talk
University of Ancona
Ancona, Italy
June 26, 2000
Assessing Knowledge Structure
DENNIS THE MENACE
“GEE! I’M GLAD WE DID CONCEPT
MAPS - NOW WE BOTH KNOW
WHAT I LEARNED TODAY!!! ”
July 27, 2000
Shavelson & Ruiz-Primo: Windows
Into The Mind
2
Overview
• Cognitive theory underlying assessments of
knowledge structure
• Direct and indirect methods of assessing
“cognitive structure”
• Direct method: Concept maps
• Indirect method: Cognitive maps
• Link between concept and cognitive maps
• Concluding comments
July 27, 2000
Shavelson & Ruiz-Primo: Windows
Into The Mind
3
Toward An Achievement Framework:
Knowledge Types and Dimensions
Proficiency
Low
Declarative
Knowledge
Procedural
Knowledge
Strategic
Knowledge
(Knowing the “that”)
(Knowing the “how”)
(Knowing the “which,”
“when,” and “why”)
High
Extent
(How much?)
Domain-specific content:
• facts
(How is it organized?)
• concepts
• principles
Structure
Domain-specific
production
systems
Problem schemata/
strategies/
operation systems
Others
(Precision?
Efficiency?
Automaticity?)
Cognitive
Tools:
July 27, 2000
Planning
Monitoring
Shavelson & Ruiz-Primo: Windows
Into The Mind
4
Associative and Semantic
Models of Memory
• Nodes represent
concepts
• Lines represent
relations among
concepts
• Probe memory by
asking about concepts
(or lines)
July 27, 2000
is a
contains
is an example of
Associative
Shavelson & Ruiz-Primo: Windows
Into The Mind
Semantic
5
Concepts Maps
• Methods (Mapping Techniques)
–
–
–
–
Constructing A Map
Fill-In-The-Node Map
Fill-In-The-Line Map
Many Others
• Data structure and scoring methods depend
on the technique
July 27, 2000
Shavelson & Ruiz-Primo: Windows
Into The Mind
6
What Are Concept Maps?
A Concept Map of a Concept Map!
Concept Map
is a
graph
consist of
consist of
nodes
represent
July 27, 2000
linked by
between a pair of
labeled
lines
denote a
conceptsShavelson & Ruiz-Primo: Windows relation
Into The Mind
7
Concept Map Assessment
Components
• Task that invites a student to provide evidence
bearing on his or her declarative knowledge
structure in a domain
• Format for the student’s response
• Scoring system by which the student’s concept
map can be evaluated accurately and
consistently
July 27, 2000
Shavelson & Ruiz-Primo: Windows
Into The Mind
8
Are There Different Types of
Concept Maps?
Yes, more than we would like!
Variations in the task, the response format, and
the scoring system produce different concept
mapping techniques.
July 27, 2000
Shavelson & Ruiz-Primo: Windows
Into The Mind
9
Concept Map Assessment:
An Example
Task
The following map shows the relationship among concepts related to
genetic continuity. The concepts are arranged hierarchically and linked
each other. Please examine the map and supply, in the space provided,
a word or two for labeling each link such that association between
concepts is made clear.
Response Format
Scoring System
Genetic
continuity
Asexual
reproduction
1. through
2. producer
3. requires
4. called
5. fuse into
• One point is given for each correct relationship
Sexual
reproduction
Meiosis
Male Female
gamete gamete
• One point is given for each level up until two
levels beyond the last branching
• One point is given for the first branching where
two or more concepts are connected to the
concept above. Three points are given for any
subsequent branching
• One point is given for each cross link
Source: Tamir, P. (1995). Science Assessment. In M. Birenbaum & F. J. R. C. Dochy (Ed.). Alternatives in assessment of achievements,
learning processes, and prior knowledge. Boston: Kluwer Academic Publishers.12
July 27, 2000
Shavelson & Ruiz-Primo: Windows
Into The Mind
10
Another Example
Task
Construct a concept map that reflects what you know
about Ions, Molecules, and Compounds. The map should
have 10 concepts in it. Organize the terms in relation to
one another in any way you want.
Response Format
Scoring System
Focuses on the accuracy of the propositions:
Blank page for
constructing the map
July 27, 2000
• Four points if the proposition is outstanding
and complete
• Three points if the proposition is correct and
complete
• Two points if the proposition is incomplete
but correct
• One point if the proposition is correct but does
not show understanding between the two
concepts.
• Zero if the proposition
Shavelson & Ruiz-Primo: Windows
Into The Mind
11
On The Reliability & Validity of
Concept Maps
• Reliability: Can a student’s concept maps be consistently scored by
different persons?
• Equivalence of Mapping Techniques: Do all mapping techniques
provide the same picture of a student’s declarative knowledge structure?
• Task Demands: Do different mapping techniques impose different
cognitive demands on a student?
• Comparing Concept Maps With Traditional Assessments: Do
concept maps measure something different than multiple-choice tests?
July 27, 2000
Shavelson & Ruiz-Primo: Windows
Into The Mind
12
Study 1:
Tasks
Mapping Technique 1
No-Concepts Provided
Mapping Technique 2
Concepts Provided
• Construct a concept map about Ions,
Molecules, and Compounds.
• Construct a concept map about Ions,
Molecules, and Compounds.
• Select another 7 concepts that you
think are the most important in
explaining the topic.
• Examine the 10 concepts and construct
the map.
• Organize the concepts
• Organize the concepts
• Redraw the map so someone else can
read it.
LIST OF CONCEPTS
acids
anions ...
July 27, 2000
Shavelson & Ruiz-Primo: Windows
Into The Mind
13
Response Format
Construct-a-Map:
No Concepts
Blank page to construct the map
Construct-a-Map:
Concepts Provided
Blank page to construct the map
Concepts: Atoms, Compounds,
Ions...
July 27, 2000
Shavelson & Ruiz-Primo: Windows
Into The Mind
14
Scoring System
Based on a criterion map, scoring focuses on:
• Proposition Accuracy: Accuracy of propositions
evaluated on a five-level scale--from 0 for inaccurate to 4
for accurate excellent
• Convergence: Proportion of accurate propositions in
student’s map to the total possible accurate propositions in
the criterion map
• Salience: Proportion of valid propositions to the total
number of propositions in student’s map
July 27, 2000
Shavelson & Ruiz-Primo: Windows
Into The Mind
15
Proposition “Quality”
Quality of
Proposition
Definition
Excellent
Outstanding proposition. Complete and correct. It shows a deep
understanding of the relation between the two concepts.
acids-compounds: < that gives off H+ when dissolved in water are
4 points
Good
Complete and correct proposition. It shows a good understanding
of the relation between the two concepts.
acids-compounds: > are examples of
3 points
Poor
Incomplete but correct proposition. It shows partial understanding
of the relation between the two concepts.
acids-compounds: < form
2 points
Don’t Care
Although valid, the proposition does not show understanding
between the two concepts.
acids-compounds: > is a different concept
1 points
Invalid
Incorrect proposition.
acids-compounds: > made of
July 27, 2000
Shavelson & Ruiz-Primo: Windows
Into The Mind
0 points
16
Generalizability(Reliability) of Scores
Percent of Total Variability and G Coefficients for
Proposition Validity Total Scores
Person (p)
Rater (r)
Sample (s)
pxr
pxs
rxc
prc,e
2

2

July 27, 2000
(nr=2,ns=3)
No-Concepts,
Sample A, and B
Sample A & B
71.64
.15
0
0
22.81
.01
5.37
78.67
0
0
.79
17.64
.18
2.69
.89
.89
(nr=2,ns=2)
.88
.88
(nr=1,ns=1)
.78
.78
Shavelson & Ruiz-Primo: Windows
Into The Mind
17
Discriminant Validity
Correlation Between Multiple-Choice Test and Concept Map Scores
Type of
Score
Proposition Validity
Congruence
Salience
July 27, 2000
No-Concepts
Sample A
Sample B
.58
.64
.63
a
.66
.55
.45
.61
.50
Shavelson & Ruiz-Primo: Windows
Into The Mind
18
Study 2:
Tasks
Construct-a-Map
• Construct a map.
• Use the 20 concepts
provided to construct the
map.
• Organize the concepts
any way you want.
• Check the map.
• Redraw the map so
someone else can read
it.
July 27, 2000
Fill-in-the-Nodes
• Examine the skeleton
map, the blank nodes
and the terms provided.
• Select the term that
corresponds to each
blank node.
• Write the term inside the
circle.
• Check that all blank
nodes are filled-in.
Shavelson & Ruiz-Primo: Windows
Into The Mind
Fill-in-the-Linking Lines
• Examine the skeleton
map, the blank linking
lines and the linking
words provided.
• Select the linking words
that correspond to each
blank linking line.
• Write the linking words
on the blank linking line.
• Check that all blank lines
are filled-in.
19
Response Format
Construct-a-Map
Fill-in-the-Nodes
Elements, Atoms, and Compounds
Fill-in-the-Linking Lines
Elements, Atoms, and Compounds
atoms
atoms
lose or gain
electrons to form
composed of
Blank page to construct the map
Concepts: Elements, Atoms,
Compounds...
composed of
compounds
compounds
formed with cations
and anions are
formed from
molecules sharing
made up
of opposite
ions
formed from
molecules sharing
made up
of opposite
ionic
compounds
molecular
compounds
example of
with 3 or more
elements are
that begins
with H are
N2O4
July 27, 2000
Shavelson & Ruiz-Primo: Windows
Into The Mind
with 3 or more
elements are
N2O4
ternary
ionic
acids
20
Construct-A-Map
Construct a concept map that reflects what you know about what a concept
map is. Organize the terms in relation to one another in any way you want.
July 27, 2000
Shavelson & Ruiz-Primo: Windows
Into The Mind
21
Fill-In-The-Node
Examine the map, the blank nodes and the terms provided in the list below. Select the term
that corresponds to each node and write it down inside the circle. Use each term only once.
ELEMENTS, ATOMS, AND COMPOUNDS
A
a re or ga nize d on a
P er iodic
Table
a re com pose d of the sam e ty pe of
Atom s
a re com bine d
c he m ica lly
to form
a re com pose d
of m ore than
one ty pe of
with e qual num be r of elec trons
a nd pr otons have a ne utr al
a re ne ga tive ly
c ha rged
partic les of
Com pounds
a re form e d when
a tom s lose or gain
that ar e
form e d from
the attra ction
of c ations and
a nions a re
c alled
a re loca ted
on the le ft
side of the
a re elem e nts
loca te d in the
c enter of the
lose or gain
e lec tr ons to
form
have a
negative
have e ithe r a
positive or a
negative
Elec tr ons
c ontaining
m olec ules
a nd shar ing
e lec tr ons
a re ca lle d
a re loca ted on the
upper right side
of the
form ions
that ca n
have
diff er ent
have a
positive
have a
neutra l
Tra nsition
Metals
have a
negative
a re com pose d of two
m ade up of
opposite
c ha rged
a re com pose d of
m etals a nd
te nd to f or m
will form
c ontaining 2 or m or e
e lem e nts a re c alled
te nd to f or m
a re atom s with
m or e pr otons tha n
e lec tr ons a nd a re
c alled
P oly a tom ic
Ions
is an e xa m ple of a
N 2O 4
a re atom s with
m or e ele c trons
than protons
a nd ar e ca lle d
Cations
a re com pose d of
H+ and
c ontain one
or m ore
with 3 or m ore
e lem e nts a re
is a te rna ry ionic
c om pound that
c ontains sulf ite
which is a
with two
e lem e nts a re
a re com pose d of
c ations a nd
have to ha ve
hydroge n
that be gin
with H a re
is an e xa m ple of a
July 27, 2000
Shavelson & Ruiz-Primo: Windows
Into The Mind
22
Fill-In-The-Linking-Lines
Examine the map, the blank lines and the linking words provided on the list below.
Select the word(s) that corresponds to each line and write the number down in the line.
Use eachCnumber only once.
ELEMENTS, ATOMS, AND COMPOUNDS
Elem e nts
a re or ga nize d on a
P er iodic
Table
a re com pose d of the sam e ty pe of
Atom s
a re com bine d
c he m ica lly
to form
a re com pose d
of m ore than
one ty pe of
a re loca ted on the
upper right side
of the
with e qual num be r of elec trons
a nd pr otons have a ne utr al
a re ne ga tive ly
c ha rged
partic les of
a re loca ted
on the le ft
side of the
a re elem e nts
loca te d in the
c enter of the
Cha rge
have a
negative
Com pounds
have a
positive
have e ithe r a
positive or a
negative
Elec tr ons
a re form e d when
a tom s lose or gain
Tra nsition
Metals
have a
negative
Ions
c ontaining
m olec ules
a nd shar ing
e lec tr ons
a re ca lle d
Metals
Non-Me ta ls
a re com pose d of two
m ade up of
opposite
c ha rged
a re com pose d of
m etals a nd
will form
Ionic
Com pounds
Molec ular
Com pounds
te nd to f or m
P oly a tom ic
Ions
Cations
Anions
N 2O 4
is a te rna ry ionic
c om pound that
c ontains sulf ite
which is a
Ter na ry Ionic
Com pounds
have to ha ve
hydroge n
a re com pose d of
H+ and
c ontain one
or m ore
with two
e lem e nts a re
Acids
Bina ry I onic
Com pounds
is an e xa m ple of a
Sodium
Sulf ite
July 27, 2000
Shavelson & Ruiz-Primo: Windows
Into The Mind
23
Scoring System
Construct-a-Map
Fill-in-the-Map
•
Proposition Accuracy: Accuracy of
propositions evaluated on a five-level
scale--from 0 for inaccurate to 4 for
accurate excellent
•
Convergence: Proportion of accurate
propositions in student’s map to the total
possible accurate propositions in the
criterion map
•
Salience: Proportion of valid
propositions to the total number of
propositions in student’s map
July 27, 2000
•
Correct/Incorrect: Student’s
responses on the skeleton map were
scored as correct or incorrect.
Shavelson & Ruiz-Primo: Windows
Into The Mind
24
Are Fill-In-The-Map Scores Sensitive To The
Nodes/Linking Lines Selected To Be Filled-In?
Means and Standard Deviations by Type of Map and Sample
Type of Skeleton Map
n
Mean
S.D.
(Max = 12)
Fill-in-the-nodes
Sample 1
80
11.21
1.42
Sample 2
72
10.80
1.74
Sample 1
78
9.77
2.74
Sample 2
73
8.99
3.09
Fill-in-the-linking lines
July 27, 2000
Shavelson & Ruiz-Primo: Windows
Into The Mind
25
Are The Two Forms Of Fill-InThe-Map Equivalent?
Means and Standard Deviations by Type of Map and Sequence
Fill-in-the-nodes
Sequence
Fill-in-the-linking lines
n
Mean
S.D.
Mean
1 Nodes 1-Lines 1
43
11.09
1.52
9.72
2.84
2 Nodes 1-Lines 2
36
11.03
1.33
9.31
3.06
3 Nodes 2-Lines 1
35
10.63
1.81
9.83
2.65
4 Nodes 2-Lines 2
37
10.97
1.67
8.68
3.13
152
11.02
1.59
9.36
2.93
Total
July 27, 2000
Shavelson & Ruiz-Primo: Windows
Into The Mind
S.D.
26
Can The Two Mapping Techniques
Be Considered Equivalent?
Means and Standard Deviations
across the Three Types of Assessments
Type of Assessment
n
Max
Mean
S.D.
152
152
152
135
1
1
53.91
.50
.73
22.17
.19
.17
Fill-in-the-nodes
Fill-in-the-lines
152
151
12
12
11.02
9.39
1.59
2.93
Multiple-Choice Test
150
30
24.05
3.74
Construct-A-Map
Proposition Accuracy
Convergence
Salience
Fill-In
July 27, 2000
Shavelson & Ruiz-Primo: Windows
Into The Mind
27
Do The Mapping Techniques Provide Information
Similar To That Provided By A
Multiple-Choice Test?
Structure
C-M FI-N FI-L
Type of Assessment
Construct-A-Map (C-M)
Fill-in-the-nodes-NOD (FI-N)
Fill-in-the-lines-LIN (FI-L)
Multiple-Choice Test (MC)
July 27, 2000
Extent
MC
(.99)
.47
(.71)
.44
.40
(.85)
.44
.37
.53
Shavelson & Ruiz-Primo: Windows
Into The Mind
(.74)
28
Other Empirical Evidence
• Correlations tell you about rank ordering on
different measures--that’s only part of the story
• Evidence is needed for the cognitive claims that
different methods measure somewhat different
aspects of achievement:
–
–
–
–
Talk aloud
Focus group
Group work
Other?
July 27, 2000
Shavelson & Ruiz-Primo: Windows
Into The Mind
29
Talk Aloud with Concept Maps:
Method Variation
Correspondence Between Directedness and Inferred Activity
0.6
Proportion Scores
0.5
0.4
Low Directed
0.3
Contruct-A-Map
Fill-In-Nodes
Fill-In-Lines
0.2
High Directed
0.1
0
Explanation
Monitoring
Conceptual
Errors
No-Code
Verbal Codes
July 27, 2000
Shavelson & Ruiz-Primo: Windows
Into The Mind
30
Tentative Concept Maps Conclusions
• Students’ maps can be consistently scored by different persons even
when complex judgments, such as proposition quality, are required
• Different mapping methods provide different pictures of students’
declarative knowledge structure. The construct-a-map method
provides opportunities to reveal students’ conceptual understanding
• Different mapping techniques impose different cognitive demands on
students. Highly structured mapping techniques allow students to
respond by elimination or guessing, whereas constructed response do
not.
• Correlations between concept-map and multiple-choice scores are
positive and moderate suggesting that these two types of assessment
measure overlapping but somewhat different aspects of declarative
knowledge
July 27, 2000
Shavelson & Ruiz-Primo: Windows
Into The Mind
31
Cognitive Maps
• Methods
–
–
–
–
Word Association
Similarity Judgment
Card Sorting
Tree Building
• Data structure: Proximities or distances
• Scaling methods: Networks or hierarchical
clusters
July 27, 2000
Shavelson & Ruiz-Primo: Windows
Into The Mind
32
Example: Similarity Judgments
Rate the similarity of each pair of terms by circling one of the numbers
provided or “DK -- Don’t Know
1. Central Tendency
(Closely Related)
Mean
1
2
3
4
5
6
7
2. Hypothesis
(Closely Related)
July 27, 2000
(Unrelated)
DK
Description
1
2
3
4
5
Shavelson & Ruiz-Primo: Windows
Into The Mind
6
7
(Unrelated)
DK
33
Concept x Concept Proximity
Matrix For A Student
Cent.
Tend.
Central Tendency
0
Correlation
8
Description
1
Regression
9
ANOVA
9
Hypothesis
8
Inference
8
Mean
1
Median
1
Mode
1
Population
8
Sample
3
Statistics
1
t-test
8
Variability
7
Variance
8
July 27, 2000
Corr.
Desc.
Regr.
ANOVA
Hyp.
Inf.
0
4
3
7
7
8
6
8
8
8
5
6
8
8
7
0
8
9
8
5
2
6
4
8
2
1
7
1
3
0
1
3
2
9
8
9
3
8
4
2
9
8
0
3
3
9
9
8
3
8
7
1
7
5
0
1
9
9
9
2
6
4
4
9
8
0
8
8
8
1
2
1
2
9
7
Mean Median Mode
0
3
3
8
6
6
7
8
7
0
2
7
5
5
8
8
9
0
7
8
6
9
9
8
Shavelson & Ruiz-Primo: Windows
Into The Mind
Pop.
0
1
3
4
9
8
Sample Stat.
0
2
9
3
3
0
7
4
6
t-test
0
8
8
Variab. Variance
0
1
0
34
Network Representation
of Proximities
Variance
Variability
Statistics
Mode
Description
Sample
Median
Central Tendency
Inference
Population
Mean
Hypothesis
t-test
Regression
ANOVA
July 27, 2000
Shavelson & Ruiz-Primo: Windows
Into The Mind
Correlation
35
Small Pilot Study (N=28) Of
Similarity Judgments And:
Terms: Variability, Variance,
Standard Deviation
• Concept maps
Variability
measures
measures
Std. Dev.
Variance
• Multiple choice
is the square
root of
1. Which of the following refers
to a procedure for making
generalizations from a limited
random sample of data?
a.
b.
c.
d.
Statistical Inference
Intuition
Population Parameterization
Standardization
Source: Ed Wiley’s 2nd Year Project
July 27, 2000
Shavelson & Ruiz-Primo: Windows
Into The Mind
36
Sample Portion Of Concept Map
refers to relationships within the
Hypothesis
Sample
Linear
Regression
uses sample to test ideas in the
Inference
Population
ANOVA
July 27, 2000
Shavelson & Ruiz-Primo: Windows
Into The Mind
37
Correlation Among Methods
Type of Assessment
Multiple Choice
Concept Map
Pretest
Concept Map
Similarity Judgment
.155
.554*
.251
Posttest
Concept Map
Similarity Judgment
.330
.434*
.706*
* p < .01
July 27, 2000
Shavelson & Ruiz-Primo: Windows
Into The Mind
38
What If We Could?
• Collect cognitive structure information
indirectly by computer
• Immediately show a student a
representation of her knowledge
structure
• Permit her to modify the structure
• And label the lines?
July 27, 2000
Shavelson & Ruiz-Primo: Windows
Into The Mind
39
Conclusions
• Concept and cognitive maps appear to tap a
different aspect of declarative knowledge
than do multiple-choice tests
• Concept and cognitive maps may provide
similar representations of structure although
this has not been tested directly
• By assessing different aspects of knowledge,
we may broaden what gets taught!
July 27, 2000
Shavelson & Ruiz-Primo: Windows
Into The Mind
40