投影片 1 - HKEdCity
Download
Report
Transcript 投影片 1 - HKEdCity
Classroom-based learning
diagnostic model for teachermade tests: Program and
applications
1Tsai-Wei
Huang, 2Pei-Chen Wu
1National Chiayi University, Taiwan,
2National Ping Tung University of Education, Taiwan
2013/01/15
Why Teacher-made test is necessary?
– Confirmation of instructional objectives
– Formative assessment for learning outcomes
– Integration between instruction and learning
What students need from tests?
– Feedbacks of progressions
– Feedbacks of concept mastered or
misconceptions revealed
– Feedbacks of their cognition styles detected
What teachers need from tests?
– Feedbacks of item/test quality
– Feedbacks from students’ outcomes
– Promotion of Item construction skills
– Promotion of teaching effectiveness
– Diagnostic information on students’ learning
Aberrance index
Detecting students’ (or items’) aberrant
response patterns based on a theoretical
model, e.g., Guttman ideal response model
Person-fit index
Item-fit index
What aberrance index can do?
– Able to reflect each student’s (item’s)
aberrant responses
– provide rich diagnostic information about
misconceptions and psychological
response patterns during learning
processes.
Illustration of PERSON/ITEM Aberrant Response Patterns (N=10 persons, K=15 Items).
SP chart
Item #
14
2
3
9
15
6
7
1
5
13
11
12
8
4
10
p=
.80
.70
.70
.60
.60
.60
.50
.50
.40
.40
.30
.20
.20
.20
.10
q=
.20
.30
.30
.40
.40
.40
.50
.50
.60
.60
.70
.80
.80
.80
.90
2
3
3
4
4
4
5
5
6
6
7
8
8
8
9
Q = Nq =
ID
T
t
s
#6
12
.80
.20
1
1
1
1
1
1
1
1
0
0
1
0
1
1
1
#3
11
.73
.27
1
1
1
1
1
1
1
1
1
1
1
0
0
0
0
#9
10
.67
.33
1
1
0
1
1
1
0
1
1
1
0
1
0
0
0
#4
9
.60
.40
1
1
0
1
1
1
1
1
0
0
0
0
1
0
0
#1
8
.53
.47
1
1
1
1
0
0
1
0
1
0
1
0
0
1
0
#5
7
.47
.53
0
0
1
0
1
1
1
0
1
1
0
1
0
0
0
#2
6
.40
.60
0
1
1
1
1
1
0
1
0
0
0
0
0
0
0
#8
4
.27
.73
1
1
1
0
0
0
0
0
0
1
0
0
0
0
0
#7
2
.13
.87
1
0
1
0
0
0
0
0
0
0
0
0
0
0
0
#10
1
.07
.93
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
RATIONALE
--BW family indices
Person-facet
–
–
–
–
W (carelessness)
B (guessing)
C (capability)
M (misconception)
Ti
Wi
(1 u
j 1
u
ij
j Ti 1
*
(qij qiT
)
[(K 1) / 2]
Ti
Ci
) (qiT* qij )
[(K 1) / 2]
K
Bi
ij
u
ij
j 1
*
(qiT
qij )
[(K 1) / 2]
K
Mi
(1- u
j Ti 1
ij
) (qij qiT* )
[(K 1) / 2]
RATIONALE
--BW family indices
N
Mj
Item-facet
–W
–B
u (t t )
100
[( N 1 ) / 2 ] – C
–M
i Q j 1
ij
ij
*
Qj
(hint )
(disturbance )
(difficulty )
(misfit )
Qj
u
wj
i 1
*
(tQj
t ij )
ij
[(N 1) / 2]
N
(1 u
bj
ij
i Q j 1
[(N 1) / 2]
Qj
cj
(1- u
i 1
ij
*
) (tQj
tij )
[(N 1) / 2]
N
u
mj
*
) (tij tQj
)
i Q j 1
ij
*
(tij tQj
)
[(N 1) / 2]
BW learning diagnostic Model
1 Pi. P. j
1u ij 1 4
BW model
Pij { Pi. 1 P. j
Person
overall
ability power
Pi . Ci 1 Wi 1 1 Bi M i
Item overall
difficulty
power
P. j c j 1 wj 1 1 bj m j
uij
}
Wbstar program
Design ideas
Functions--Person aspect
Able to estimate the probabilities of
– items correctly answered
– concepts mastered
– teaching objectives reached
Able to classify students to adequate classes of
psychological cognition status for diagnosis under
a certain criterion (e.g., alpha = .05)
Functions--Item aspect
– Able to provide item information about
Difficulty
Discrimination
Option distraction
Hint
Disturbance
Misfit
– Able to examine item’s functions by classification
under a certain criterion (e.g., alpha = .05)
Program’s looks
Program’s looks
Program’s looks
S-P chart
Example 1:
Fraction and Decimal
Data
– 32 students in the 4th grade elementary mathematics class
in Taiwan.
– Teacher-made test (22 items) with five concepts of fraction
and decimal:
transforming decimal into fraction (TDF)
equivalent fraction (EF)
comparing fraction with decimal (CFD)
transforming fraction into decimal (TFD)
unit transformation (UT)
Q matrix of items on concepts judged by 2 teachers
Classification design
Item analysis
--high w value (11.36)
Irrelevant Hint:
160 4/5 >1.605!
How about
160 2/5?
Item analysis
--high b value (4.83)
ID 8 student gave an correct
answer of 68 for q2.
Sequentially, he gave q3 an
answer of 68/1000.
Concepts mastery
Aberrance patterns (H1,H3)
Aberrance patterns (A1, A1’, A2, A4 )
Aberrance patterns (L1, L2, L3, L4 )
Example 2:
Two-tier acid-base test
Examining
the effects of
classifying response
patterns by BWCM indexes
67 sixth grade students (41 boys,
61.19%; 26 girls, 38.81%) in Taiwan.
Four cognitive styles of the two tier items
REASON (KNOW-HOW)
2-TIER ITEM
ANSWER
(KNOW-WHAT)
RIGHT
WRONG
True knowing
Guessing
Careless
Unknown
RIGHT
WRONG
Questions
Q1 : How do the classified clusters
based on the BW indices be verified?
Q2 : What characteristics of the
classified clusters will reveal?
Q3 : What characteristics will change
for individuals between the know-what
test and know-how test?
RESULTS
Classification
HIT RATES
DA
MLR
Group 2
Group 3
K-mean
98.5%
91.5%
Ward
97.1%
95.6%
K-mean
100%
97.0%
Ward
100%
100%
Properties of 2 clusters
Group of low
misconception with
aberrant responses
Ward
K-mean
2
2
M
C
Group
B
M
C
Group
B
W
W
1
1
0
5
10
15
20
25
30
0
5
10
15
Group of high
20
25
30
competence
with normal
responses
3 clusters
Middle competence
Group with complex
aberrant responses
K-mean
Ward
Low
competence
Group with
guessing
aberrant
responses
3
3
M
M
C
Group 2
B
C
Group 2
B
W
W
1
1
0
5
10
15
20
25
30
35
40
0
5
10
15
40
35
30
25competence
20
High
Group with
normal responses
Change between know-what (ANSWER)
and know-how (REASON)
CASE 60
H1H1
2.5
3
2
2.5
CASE 50
H1A3
2
1.5
1.5
1
0.5
0
ANSWER
1
ANSWER
REASON
0.5
REASON
0
-0.5
-0.5
-1
-1
-1.5
-1.5
ZW
ZB
ZC
ZM
ZW
L1’H1
ZB
CASE 12
1
0.5
0
ANSWER
REASON
-0.5
-1
-1.5
ZW
ZB
ZC
ZM
ZC
ZM
Recommendations
The BW model can provide sufficient and matching
information of interpretations to an original
response data matrix in both person-and item
facets, especially in a small sample size situation
like that of students takes a teacher-made test in
class.
Teachers can use the useful information to improve
their test development skills and to have immediate
diagnostic feedbacks from their students’ response
patterns.
The BW model can provide the extent to which
students mastered in concepts and achieved to
objections so that teachers can go further for
designing remedial instructions.
Future Outlook
The BW model in this study is now only suitable
for the dichotomously scoring situations;
Polytomously scoring system is under developing...
Related articles and resources
Huang, T. W. (2008). A study of cutoffs for aberrant indices under different data structures. The Journal of
Guidance and Counseling, 30, 1-16. (in Chinese)
Huang, T. W. (2011). Establishing and examining the diagnostic space of two new developed person-fit indices:
The W* and the B* indices. Psychological Testing, 58(1), 1-27. (in Chinese)
Huang, T. W. (2011). Robustness of BW aberrance indices against test length. Knowledge Management & ELearning: An International Journal, 3, 310-318.
Huang, T. W. (2012). Aberrance Detection Powers of the BW and Person-Fit Indices. Journal of Educational
Technology & Society, 15(1), 28-37.
Huang, T. W. & Wu, P. C. (in press). A classroom-based cognitive diagnostic model for teacher-made tests: An
example with fractions and decimals. Journal of Educational Technology & Society.
Huang, T. W. (2007, July). Establishing Cutoffs for Two New Aberrance Indices: The Within- Ability-Concern
Index and the Beyond-Ability-Surprise Index. Paper presented at the annual meeting of the International
Conference on the Teaching of Psychology, Vancouver, Canada.
Huang, T. W. (2012, July).Verifying the effects of item-fit indices on gender-related differential item
functioning. Paper presented at the XXX International Congress of Psychology, Cape town, South Africa.
Huang, T. W., & Tsai, S. W. (2012, August). Examining the effect of classifying response patterns on an acidbase test by a Guttman-based person-fit index set. Paper presented at the 1st International Conference on
Education Measurement and Evaluation 2012 (ICEME2012), Manila, Philippines.
The WBStar program now is available in CHINESE version at http://140.130.46.9. English version is not
completely done, but coming soon…
THANKS FOR YOUR ATTENTIONS!