Transcript to view the PowerPoint presentation

Using Rasch Analysis to Develop an
Extended Matching Question (EMQ)
Item Bank for Undergraduate Medical
Education
Mike Horton
Bipin Bhakta
Alan Tennant
Medical student training

Miller (1990) identified that no single assessment
method can provide all the data required for judging
anything as complex as the delivery of professional
services by a competent physician
• Knowledge
• Skills
• Attitudes
Miller’s pyramid of competence
does
case presentations
log books
direct observation of
clinical activity
shows how
OSCE
knows how
EMQ, FRQ,
Essays
knows
MCQ
What are assessments used for?

Primary aim
• To identify the student who is deemed to be safe
and has achieved the minimal acceptable standard
of competence

Secondary aim
• To identify students who excel
Written Assessment







“True or False” questions
“Single, best option” multiple choice questions
Multiple true or false questions
“Short answer” open ended questions
Essays
“Key feature” questions
Extended matching questions
Free response questions


Broadly, the free-response type questions are commonly
believed to test important higher-order reasoning skills
Validity is high as examinees have to generate their own
responses, rather than selecting from a list of options
However…



Only a narrow range of subject matter can be assessed in a
given amount of time
They are administratively resource-sapping
Due to their nature, the reliability is limited.
Multiple choice questions
Multiple choice type questions have been widely used and are
popular as:


They generally have a high reliability
They can test a wide range of themes in a relatively short period
of time
However…


They only assess the knowledge of isolated facts
By giving an option list, examinees are cued to respond and the
active generation of knowledge is avoided
What are Extended Matching Questions (EMQs)?

EMQs are used as part of the undergraduate medical course
assessment programme

EMQs are used to assess factual knowledge, clinical decision
making and problem solving

They are a variant of multiple choice questions (MCQs)

EMQs are made up of 4 components
Example of Extended Matching Question (EMQ) format
(Taken from Schuwirth & van der Vleuten, 2003)
Theme:
Micro-organisms
Answer options:


A
B
C
D
E
F
G
H
Campylobacter jejuni
Candida Albicans
Clostridium difficile
Clostridium perfringens
Escherichia coli
Giardia lamblia
Helicobacter pylori
Mycobacterium tuberculosis
Instructions:


I
J
K
L
M
N
O
P
Proteus mirabilis
Pseudomonas aeruginosa
Rotavirus
Salmonella typhi
Shigella flexneri
Tropheryma whippelii
Vibrio cholerae
Yersinia enterocolitica
For each of the following cases, select (from the list above) the microorganism most likely to be responsible. Each option may be used once,
more than once or not at all.
A 48 year old man with a chronic complaint of dyspepsia suddenly develops severe
abdominal pain. On physical examination there is general tenderness to palpitation with
rigidity and rebound tenderness. Abdominal radiography shows free air under the
diaphragm.
A 45 year old woman is treated with antibiotics for recurring respiratory tract infections.
She develops a severe abdominal pain with haemorrhagic diarrhoea. Endoscopically a
pseudomembranous colitis is seen.
Item Pools

Currently, a lot of institutions formulate their tests from year to year by
selecting items from a pre-existing pool of questions.
• Questions are pre-existing
• Time and resources are saved by employing this method.
However…

It has been widely recognised that if tests are made up of items from a
pre-existing item pool, then the relative difficulty of the exam paper will
vary from year to year. [McHarg et al (2005), Muijtens et al (1998),
McManus et al (2005)]
Item Pools

If the questions have been set, used and assessed using traditional
approaches, then this will provide a certain amount of information about each
of the individual questions, however, there are also drawbacks surrounding
the traditional approach.

It has been recognised [Downing (2003)] that there are certain limitations to
Classical Measurement Theory (CMT) in that it is sample dependent.

Thus, the comparability of examination results from year to year will be
confounded by the overall difficulty of the exam and the relative ability of the
examinees, rendering a comparison invalid.

This is particularly troublesome when we wish to
• compare different cohorts of students
• maintain a consistent level of assessment difficulty over subsequent administrations.
The problem

What is the best way to ensure that all passing students are reaching the
required level of expertise??

2 forms of pass mark selection
• Criterion referenced
• Norm referenced

Criterion referenced refers to when a specific pass mark has been
designated prior to the exam as a pass/fail point.

Norm referenced refers to when a specific proportion of the sample will be
designated to pass the exam. i.e. the highest scoring 75% of students will
pass
Norm referenced or Criterion Testing?
Norm-referenced testing





Whatever the ability of the students taking the test, a fixed portion of them
will pass/fail
The standard needed to pass the test is not known in advance
The validity of the norm-referencing method relies on a homogenous
sample, which may not necessarily be the case
There is also the risk that with a less able group of students, a student
could pass the exam without reaching the desirable acceptable level of
clinical knowledge.
Norm referencing is not appropriate
Norm referenced or Criterion Testing?
Criterion testing

Relative difficulty of Exam could change depending on the items that are
in the exam, therefore a pre determined pass mark could be easier or
harder to obtain depending upon the items in the test

Although having their own disadvantages, it has been recognised [Wass
et al (2001)] that Criterion referenced tests are the only acceptable means
of assessing that a pre-defined clinical competency has been reached
Solution?

It has been identified [Muijtjens (1998)] that a criterion referenced test
could be utilised if a test was constructed by selecting items from a bank
of items of known difficulty, which would then enable measurement and
control of the test difficulty.

Difficulty estimates, as defined by classical test theory, are sample
dependent!

Item Banking
Item banking

Item Banking is a process whereby all EMQ Items that have been
used over the past few years are ‘banked’ and calibrated onto the
same metric scale

Previously used EMQ Items

Psychometrically calibrated using Rasch Analysis

Data is linked by common items that have been used between the
exams
• “Common Item Equating”
Rasch Analysis



When data fit the model, generalisability of EMQ difficulties
beyond the specific conditions under which they were observed
occurs (specific objectivity )
In other words…
Item Difficulties are not sample dependent as they are in
Classical Test Theory
Item banking
Term 1
Term 2
Term 3
Term 4
ITEM 1
ITEM 2
ITEM 3
ITEM 4

These Items cannot be directly compared as there are no
common links
Item banking
Term 1
Term 2
Term 3
Term 4
ITEM 1
ITEM 2
ITEM 3
ITEM 4
ITEM 5

These Items can be directly compared via the common link item
across all terms
Item banking

Following calibration, the set of items within the bank will define
the common domain that that they are measuring (in this case,
medical knowledge).

These items, therefore, will provide an operational definition of
this unobservable (latent) trait.

When all EMQ Items are calibrated onto the same scale, then it
will be possible to compare the performance of students across
terms, despite the fact that the EMQ exam content was made up
of different items across each term.
Sufficient Linkage?



What is classed as sufficient linkage between item sets?
There has been some variation in the literature regarding this.
Three differing viewpoints suggest that:
• linking items should be the larger of 20 items or 20% of the total number of
items [Angoff (1971)]
• 5 to 10 items are sufficient to form the common link [Wright & Bell (1984)]
• one single common item could provide a valid link in co-calibrating datasets
[Smith (1992)].

However, it has also been suggested [Smith (1992)] that the larger the
number of common items across datasets, the greater degree of precision
and stability for the item bank.
Potential Problems?

Limited Linkage
• Data overlap reduced

Potential Misfit or DIF on Link
Items
Q1
Q2
Q3
Q4
Q5
Q6
Q7
Q8
Q9
Q10
Q11
Q12
Q13
Q14
Q15
Q16
Q17
Q18
Q19
Q20
Q21
Q22
Q23
Q24
Q25
Term8
Term7
Term6
Term5
Term4
Term3
Term2
Sparse Data Matrix
Term1

Sample

Data was collected from 550 4th year medical students over 8
terms

All EMQ data was read in to a single analysis to establish the
Item Bank

RUMM2020 Rasch Analysis software was used

Over the 8 terms, 6 different test forms were used (the test
remained the same over the first 3 terms)
1 1
1 2
1 3
1 4
1 5
1 6
2 11
2 12
2 13
2 14
2 15
2 16
3 21
3 22
3 23
3 24
3 25
3 26
3 27
3 28
3 29
4 31
4 32
4 33
4 34
4 35
4 36
4 37
4 38
4 39
4 40
5 41
5 42
5 43
5 44
5 45
5 46
5 47
5 48
6 51
6 52
6 53
6 54
6 55
6 56
6 57
6 58
6 59
6 60
6 1051
7 61
7 62
7 63
8 71
8 72
8 73
8 74
8 75
9 81
9 82
9 83
9 84
9 85
9 86
10 91
10 92
10 93
10 94
10 95
10 96
2
3
4
5
6
10
10
10
11
11
11
11
12
12
12
12
13
13
13
13
14
14
14
14
15
15
15
15
15
15
16
16
16
16
17
17
17
17
17
18
18
18
18
18
18
18
19
19
19
19
19
19
20
20
20
20
20
20
21
21
21
21
21
22
22
22
22
22
23
23
23
23
23
23
23
97
98
99
101
102
103
104
111
112
113
114
121
122
123
124
131
132
133
134
141
142
143
144
145
146
151
152
153
154
161
162
163
164
165
171
172
173
174
175
176
177
181
182
183
184
185
186
191
192
193
194
195
196
201
202
203
204
205
211
212
213
214
215
221
222
223
224
225
226
227
1
2
3
4
5
6
Theme
Code
Question
Code
1
Theme
Code
Question
Code
Theme
Code
Question
Code
EMQ Item Bank
23
24
24
24
24
24
24
24
24
24
24
25
25
25
25
25
25
25
25
25
25
26
26
26
26
26
26
27
27
27
27
27
27
27
28
28
28
28
29
29
29
29
30
30
30
30
31
31
31
31
31
31
31
31
32
32
33
33
33
33
34
34
34
34
35
35
35
228
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
261
262
263
264
265
266
267
271
272
273
274
281
282
283
284
291
292
293
294
301
302
303
304
305
306
307
308
311
312
321
322
323
324
331
332
333
334
341
342
343
1
2
3
4
5
6
• Approximately
25% of the
items were changed from
exam form-to-exam form
• This provides good linkage
EMQ Item Bank
Pretty Good!!
To be expected
Low. But…
Low Person Separation Index


Person Separation Index is fairly low
We would expect this to a certain degree due to the highly focussed sample
Problematic Items

Approximately 12.5% (26/205) of Individual
Items were found to display some sort of misfit or
DIF
Misfit

What does misfit tell us?

Misfitting items are flagged up for non-inclusion and will either
be amended or removed
Differential Item Functioning
Misfit & DIF

What does misfit tell us?
Misfitting items are flagged up for non-inclusion and will either be
amended or removed

DIF could be due to:

• Curriculum changes
• Teaching Rotations
• Testwiseness

Does Exam Difficulty remain equal over
different exam forms?

Example across item set using Exam Form 1 as
the Baseline
Measurement and the pass mark
The latent trait
Increasing student ability
Minimum standard
0
The “real” assessment
100
EMQ Exam
60%
Pass mark
Equivalent standard?



Exam Form 1 was based on 98 Items
Pass Mark was set at 60%
60% of 98 = Raw Score pass mark of 58.8
Equivalent pass mark?

Raw score of 58.8 on Exam Form 1 = 0.561 logits
Equating Problems

We had to remove 2 extreme items

Exam form 1 was out of 98 anyway
Exam Form
Maximum
obtainable
score
1
98
2
99
3
100
4
100
5
100
6
99
Equivalent standard?
Exam form 1 had 98 items
60% of 98 = 58.8
58.8 score on Exam form 1 = 0.561 logits



Exam Form
0.561 logit
equated score
out of
1
2
3
4
5
6
58.7
57.7
58.2
59.6
62.4
59.9
98
99
100
100
100
99
59.90% 58.30% 58.20%

62.4% - 58.2% = 4.2%
59.60% 62.40% 60.50%
Conclusion

Item Banking is a way of assessing the psychometric properties of EMQs
that have been administered over different test forms

Can identify and adapt poor questions

Can perform a comparative analysis of relative test form difficulty

Should the pass mark be amended every term?
References
1.
2.
3.
4.
5.
6.
7.
8.
9.
Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine
1990; 65: S63-7
McHarg J, Bradley P, Chamberlain S, Ricketts C, Searle J, McLachlan JC. Assessment of
progress tests. Medical Education 2005; 39: 221-227.
Muijtjens AMM, Hoogenboom RJI, Verwijnen GM, Van der Vleuten CPM. Relative or absolute
standards in assessing medical knowledge using progress tests. Advances in Health Sciences
Education 1998; 3: 81-87.
McManus IC, Mollon J, Duke OL, Vale JA. Changes in standard of candidates taking the
MRCP(UK) Part 1 examination, 1985 to 2002: Analysis of marker questions. BMC Medicine
2005; 3 (13).
Downing S M. Item response theory: applications of modern test theory in medical education.
Medical Education 2003; 37: 739-745
Wass V, Van der Vleuten C, Shatzer J, Jones R: Assessment of clinical competence. Lancet
2001, 357(9620): 945-94
Angoff W H. Scales, norming, and equivalent scores. In: Thorndike R L, editor. Educational
Measurement. 2nd ed. Washington (DC): American Council on Education; 1971. p508-600
Wright B D, Bell S R. Item Banks: what, why, how. Journal of Educational Measurement 1984;
21(4): 331-345
Smith R M. Applications of Rasch Measurement. Chicago: Mesa Press; 1992
New Book
Smith EV Jr. & Stone GE (Eds.). Criterion Referenced Testing:
Practice Analysis to Score Reporting Using Rasch Measurement
Models. Maple Grove, Minnesota. JAM Press; 2009
Contact Details
Mike Horton:
[email protected]
Matt Homer:
[email protected]
Alan Tennant: [email protected]
Website:
http://www.leeds.ac.uk/medicine/rehabmed/psychometric/
Course
Introductory
Intermediate
Advanced
Date
March 10-12
May 12-14
Sept 15-17
Dec 1-3
March 23-25
May 18-20
Sept 14-16
Nov 30-Dec 2
May 17-19
Sept 20-22
Dec 6-8
May 23-25
Sept 19-21
Dec 5-7
Sept 23-24
Sept 22-23
2010
2010
2010
2010
2011
2011
2011
2011
2010
2010
2010
2011
2011
2011
2010
2011