Transcript ASSESSMENT METHODS FOR MEDICAL STUDENTS

ASSESSMENT OF
KNOWLEDGE
Multiple Choice
Question
Dr. Eqbal Anwer
Assistant Professor,
Deptt. of Physiology, ELMCH
LEARNING OBJECTIVES





What is MCQs?
Parts of MCQs?
Importance, Advantage & Limitations of
MCQs?
Where to use these MCQs?
How to construct MCQs?
What is Assessment?
The process of finding out how much each student
knows or can do.
2 TYPES
1. Formative assessment (Diagnostic)
 Done during the course.
 Provide feedback to the student on his/her progress.
 Provide feedback to the teacher with data for
modification of his/her teaching.
2. Summative assessment (Certifying)
 Done at the end of period.
 For placing the student in order of merit.
 During giving any certificate.
Why Assessment ???
 Ranking of student.
 Measurement of improvement in a student.
 To diagnose student difficulties.
 Evaluate method of teaching
 Motivate student to study.
CATEGORISE
GOOD ASSESSMENT
1.
Validity
–
2.
Reliability
–
3.
Reproducibility
Acceptability
–
4.
Measuring what it tends to measure
Acceptability by students/assessor
Consequences of Assessment
–
Educational Impact
What to Assess?
&
How Often?
Domain Of Learning
1.
Cognitive Domain

2.
Psychomotor Domain

3.
Knowledge
Practical Skills
Affective Domain

Attitudes
Bloom’s Taxonomy
LEARNING COMPONENTS
1. KNOWLEDGE
Dose the student or doctor know
what he or she should do?
Communication skill
2. SKILLS
Psychomotor skill.
Is he or she able to do it?
3. ATTITUDE/ RESPONSE
In practice, does the student or doctor
choose to do it when confronted with a
situation?
Clinical competence is demonstrated when a
task is performed using learned skills and
knowledge.
CLINICAL COMPETENCE
Does
Shows How
Knows How
Knows
MILLER’S PYRAMID
METHODS USED FOR THE ASSESSMENT
OF MEDICAL STUDENT
1.
Long Essay Questions LEQ.
2.
Oral Examination.
3.
Short Essay Questions SEQ.
4.
Multiple Choice Question MCQ.
5.
Modified Essay Question MEQ.
6.
Patient Management Problem PMP.
7.
Observational Assessment, OSCE, OSPE.
About Multiple-Choice Tests
MULTIPLE CHOICE QUESTIONS
(MCQ)


In these Question, you have to choose “
one-best answer” from multiple choices.
Questions = Items
Anatomy of an ITEM
•
•
•
Given a problem – Stem
Followed by single question - Lead – in
Option / Choices –
–
–
Correct answer – Key
Other incorrect Answers- Distracters
Example
Q. A 1-year-old infant is known to
have heart disease and is noted to
be cyanosed.
Which of the following is the most
likely diagnosis?
a. Atrial septal defect
b. Patent Ductus Arteriosus
c. Ventricular septal defect
d. Tricuspid atresia
Example
Q. A 1-year-old infant is known
to have heart disease and is
noted to be cyanosed.
Which of the following is the
most likely diagnosis?
a.
b.
c.
d.
Atrial septal defect
Patent Ductus
Arteriosus
Ventricular septal
defect
Tricuspid atresia
STEM
LEAD - IN
DISTRACTOR
KEY
Importance of MCQ


An ability to sample a wide range of
knowledge.
Easy to administer




to large numbers of students
in a short period of time and
minimal human intervention
Highly Acceptable – preferred tool for
written assessment across most of the
academic disciplines.
Advantages
MULTIPLE-CHOICE ITEMS can provide:

Versatility in measuring all levels of cognitive ability,

Highly Reliable test scores,

Scoring Efficiency and Accuracy,

Objective measurement of achievement or ability,

Wide sampling of content or objectives,

Reduced guessing factor compared with true-false items, and

Different response alternatives which can provide diagnostic
feedback.
(Ory & Ryan, 1993)
Limitations

Are difficult and time-consuming to construct,

Place a high degree of dependence on the
student’s reading ability and instructor’s writing
ability, and

Are particularly subject to clueing. (Students
can often deduce the correct response by
elimination.) – need plausible distractor

Attention to security of Question paper
(Ory & Ryan, 1993)
When to Use

To test a variety of levels of learning

When you have a large number of individuals
taking the test

When you have time to construct the test items

When time is limited for scoring

When it is not important to determine, how well
individuals can formulate their own answer

When you want to prepare individuals for future
assessments, that use a similar format
(Clegg & Cashin, 1986)
Planning to construct MCQ
Multiple-choice Test Construction
“… the greater your experience in their
construction, the longer it takes per
[multiple-choice] item to construct a
reasonably fair, accurate, and inclusive
question.”
- Wilbert J. McKeachie
Initial focus

Standard / Scope of the examination
conducted

Number of question – time available for
examination

Source of Question
General guidelines

Pre – plan well


Peer review/ peer validation


Best time when you teach
More critical review – more you gain
Trial run

You‘ll determine facility index & discriminative
index
General Rules

Each item should be based on an important
learning objective

Test for significant/important information

Focus on single idea in single test item

Use items to assess higher order of thinking

Keep language simple
Constructing Test Items
Components
Stem: presents the problem
Correct or keyed options: correct option
Distractor options: incorrect options
(Clegg & Cashin, 1986)
Developing an Item
1.
Choose an important concept
2.
Write the stem
3.
Write the correct answer (key)
4.
Develop distractors





common misconceptions
errors that could be made
plausible, yet less important information
similar in style, length to the key
every distractor should be reasonable
(Clegg & Cashin, 1986)
Issues Related to Test-wiseness
• Grammatical Cues
• Logical Cues
• Absolute Terms
• Long Correct answer
• Word Repeats
(Clegg & Cashin, 1986)
Issues Related to Irrelevant Difficulty
• Options long
• Numeric data not stated consistently
• Vague terms
• Language not parallel
• Options in no logical order
• “none of the above” is used
• Stems tricky or unnecessarily complicated
• Answer to an item is “hinged” to the answer of a related
item
(Clegg & Cashin, 1986)
STEM
Writing Items
•
Write items on significant concepts, not
trivial facts.
•
Write items that have a definite answer.
•
Communicate clearly.
•
•
•
Don’t give away the answer by including
irrelevant cues in the item.
Don’t write items that require skills or
knowledge irrelevant to what you are
trying to measure.
Have items reviewed by knowledgeable
persons other than the composer of the
question if possible.
(Clegg & Cashin, 1986)
Writing Stems
• Ensure that the directions in the stem are
very clear.(examinee should know exactly
what is being asked)
 Include the central idea in the stem instead
of the choices.
 Avoid window dressing (excessive verbiage).
 Word the stem positively, avoid negatives
such as NOT or EXCEPT. If negative words
are used, use the word cautiously and always
ensure that the word appears capitalized and
boldface.
(Haladyna, Downing & Rodriguez, 2002)
Writing Stem



Avoid very short stems
Write stem in either question form or
clinical scenario but only relevant text
Clinical scenario is better than
statement as this stimulate higher level
of thinking
Writing Stems
Avoid stems that ask for a series of multiple true-false responses.
Better:
Q. Which of the following is true
about the middle adult years?
A.
B.
C.
D.
It encompasses ages 19 to 30.
It is the most conflict-free period
of life.
It is characterized by dramatic
changes in our sense of values.
It is marked by a conflict between
intimacy and isolation.
Q. According to Erickson, the
middle adult years are
characterized by the conflict
between ____ and ___ .
A.
B.
C.
D.
intimacy; isolation
generativity; stagnation
integrity; despair
industry; despondency
(Ory & Ryan, 1993)
Writing Stems
Eliminate excessive wording and irrelevant information.
Q. Sheldon developed a highly
controversial theory of personality
based on body type and temperament
of the individual. Which of the following
is a criticism of Sheldon’s theory?
A. He was influenced too much by
Freudian psychoanalysis.
B. His ratings of physique and
temperament were not independent.
C. He failed to use an empirical
approach.
D. His research sample was improperly
selected.
Better:
Q. Which of the following
is a criticism of Sheldon's
theory of personality?
(Ory & Ryan, 1993)
Writing Stems
Include in the stem any word(s) that might otherwise be
repeated in each alternative.
Q. The receptors for the
vestibular sense are located
A.
in the fovea.
B.
in the brain.
C.
in the middle ear.
D.
in the inner ear.
Better:
Q. The receptors for the
vestibular sense are located in
the
A.
fovea.
B.
brain.
C.
middle ear.
D.
the inner ear.
(Ory & Ryan, 1993)
Writing Stems
Use negatively stated stems sparingly.
When used, underline and/or capitalize the negative word.
Q. Which is not a major
technique for studying brain
function?
A.
B.
C.
D.
accident and injury
cutting and removing
electrical stimulation
direct phrenology
Better:
Q. Which is NOT a major
technique for studying
brain function?
A.
B.
C.
D.
accident and injury
cutting and removing
electrical stimulation
direct phrenology
(Ory & Ryan, 1993)
Writing Stems
When using incomplete statements avoid beginning with the
blank space.
Better:
Q. ___ is the least severe
form of behavior disorder.
A.
B.
C.
D.
Psychosis
Panic disorder
Neurasthenia
Neurosis
Q. The least severe form of
behavior disorder is ___ .
A.
B.
C.
D.
Psychosis
Panic disorder
Neurasthenia
Neurosis
(Ory & Ryan, 1993)
Writing Stems
Use familiar language.
Better:
Q. According to Freud the
raison d’être for hysteria
was
A.
B.
C.
D.
sexual conflicts.
unresolved feelings of
guilt.
latent tendencies.
repressed fear.
Q. According to Freud
hysteria was caused by
A.
B.
C.
D.
sexual conflicts.
unresolved feelings of
guilt.
latent tendencies.
repressed fear.
(Ory & Ryan, 1993)
Writing Stems
Provide sufficient information in the stem to allow students
to respond to the question.
Q. How many interrelated
stages to creative problem
solving are there?
A.
Three
B.
Four
C.
Seven
D.
Ten
Better:
Q. The textbook indicates that
there are ___ interrelated
stages to creative problem
solving.
A.
Three
B.
Four
C.
Seven
D.
Ten
(Ory & Ryan, 1993)
WRITING OPTIONS
Writing Options





Develop as many effective choices as you
can, but research suggests three is
adequate.
Make sure that only one of these choices is
the right answer.
Vary the location of the right answer
according to the number of choices
Place choices in logical or numerical order.
Keep choices independent; choices should
not be overlapping.
(Haladyna, Downing & Rodriguez, 2002)
Writing Options

Keep choices homogeneous in content and grammatical
structure.

Keep the length of the choices about equal.

None-of-the-above should be used carefully.

Avoid All-of-the-above.

Make all distractors plausible.

Use typical errors of students to write your distractors.

Use humor if it is compatible with the teacher and the learning
environment.
(Haladyna, Downing & Rodriguez, 2002)
Writing Options


Phrase choices positively; avoid negatives
such as NOT.
Avoid giving clues to the right answer, such
as:
o
o
o
o
o
specific determiners including always, never, completely,
and absolutely.
clang associations, choices identical to or resembling
words in the stem.
Very noticeable correct choice.
pairs or triplets of options that clue the test-taker to the
correct choice.
blatantly absurd, ridiculous options.
(Haladyna, Downing & Rodriguez, 2002)
Writing Options
Make sure there is one correct or best response.
Q. Which of the following does
not belong with the others?
A.
B.
C.
D.
Wundt
Structuralism
James
Titchener
(Ory & Ryan, 1993)
Writing Options
Make all alternatives plausible and equally attractive to both
less-knowledgeable and skillful students.
Q. The number of photoreceptors in
the retina of each human eye is about
A.
1000,000.
B.
2 million.
C.
115 million.
D.
2.37 billion.
Better:
A.
B.
C.
D.
5 million.
35 million.
65 million.
115 million.
(Ory & Ryan, 1993)
Writing Options
Minimize the use of the all-of –the-above and none-of-the-above
alternatives.
Q. Problem representation
involves
A.
B.
C.
D.
determining which factor
matters and which do not.
the initial state of problem
solving.
both a and b.
neither a nor b.
Better:
A. determining which
factors matter and which
do not.
B. the initial state of
problem solving.
C. reducing the problem to
manageable segments.
D. all of the above.
(Ory & Ryan, 1993)
Writing Options
All alternatives should be approximately equal in length.
Better:
Latane and Darley smoke-filled
room experiment suggested that
people are less likely to help in
groups than alone, because
people
A.
B.
C.
D.
in groups talk to one another.
who are alone are more attentive.
in groups do not display pluralistic
ignorance.
in groups allow others to define
the situation as a non-emergency.
Latane and Darley smoke-filled room
experiment suggested that people are
less likely to help in groups than
alone, because people
A.
B.
C.
D.
talk to one another.
are less attentive than people
who are alone .
do not display pluralistic
ignorance.
allow others to define the
situation as a non-emergency
(Ory & Ryan, 1993)
Writing Item Options
Make alternatives parallel in construction and consistent
with the stem.
Which of the following is
NOT a defense
mechanism?
A.
B.
C.
D.
Conflict.
Repression.
Reaction formation.
Rationalization.
Better:
A.
B.
C.
D.
Rationalization.
Repression.
Reaction formation.
Regression.
(Ory & Ryan, 1993)
Writing Item Options
When possible, present alternatives in some logical order
(e.g. most to least and chronological)
Q. In the course of a dark
adaptation , the eye’s best
sensitivity to wavelength
shifts to
A.
B.
C.
D.
580
477
505
600
millimicrons.
millimicrons.
millimicrons.
millimicrons.
Better:
A. 600 millimicrons.
B. 580 millimicrons.
C. 505 millimicrons.
D. 477millimicrons.
(Ory & Ryan, 1993)
Writing Item Options
Make the alternatives mutually exclusive.
Q. Rods are found in the
A.
B.
C.
D.
blind spot.
fovea.
periphery of the retina.
back of the eye.
Better:
A. blind spot.
B. periphery of the fovea.
C. periphery of the retina.
D. cornea.
(Ory & Ryan, 1993)
Writing Item Options
Avoid overly wordy alternatives that become confusing
and difficult to read.
Flooding differs from systematic
desensitization in that
A.
B.
C.
D.
the former is based on classical
conditioning and the latter on operant
conditioning.
systematic desensitization requires
insight and the flooding does not.
flooding has you start at the top of your
fear hierarchy and systematic
desensitization has you start at the
bottom and work up gradually.
flooding emphasizes the use of
cognitions to a much greater extent
than does systematic desensitization.
Better:
Flooding differs from systematic
desensitization in that flooding
A.
B.
C.
D.
is based on classical
conditioning rather than
operant conditioning.
doesn’t require insight.
starts at the top of the fear
hierarchy.
places greater emphasis on the
use of cognitions.
(Ory & Ryan, 1993)
Writing Item Options
Avoid irrelevant cues such as grammatical structure,
well-known work associations, or connections
between the stem and the correct answer.
Q. School psychologists who
examine and place children in
special education settings often
apply the research done by
A.
B.
C.
D.
Better:
School psychologists often
apply the research done by
biopsychologists.
educational psychologists.
clinical psychologists.
counseling psychologists.
(Ory & Ryan, 1993)
Writing Options
Avoid language that may offend or exclude a particular group of
individuals.
Q. Which of the following is a
characteristic of persons with
Down’s syndrome?
A.
B.
C.
D.
Larger than normal head
Obesity
Oriental-like skin folds over
the eyes
Above average height.
Better:
A. Larger than normal head
B. Obesity
C. Downward sloping skin
fold over the eyes
D. Above average height.
(Ory & Ryan, 1993)
SUMMARIZE
For developing options






Place option in logical/ numerical
sequence
Use letters in front of options
Keep options independent – mutually
exclusive
Keep content of all options
homogeneous
Keep length of option nearly equal
Phrase the options positively
Rules for Key


One and only one
Balanced – in a set of items, position of
correct option should same no. of times
for each position
Rules for Distracter


Plausible – more similar to correct answer
No too closer to correct answer – lead to
test for precision
AVOID







Cueing one items with another
Tricky items that mislead examinees
Items based on assumptions
Negative phrasing
Providing clues – grammatical / making
correct answer longer
Use of never/ always
None of the above/ all of the above
Tips for Writing Tests
•
•
•
•
•
•
Compose test items over time.
Test what you really want individuals to
learn.
Check borrowed items carefully.
Create a test bank.
Start easy to build confidence.
Get feedback on items.
(Nilson, 2010)
Planning a Test



Use a test matrix or blueprint.
Identify major ideas and skills
rather than specific details.
Use Bloom’s cognitive taxonomy
or something appropriate for
your context.
(Nilson, 2010)
Bloom’s Cognitive Domain
Evaluation
Synthesis
Analysis
Application
Comprehension
Knowledge
Test Matrix
Objectives at Different Levels
Level: Knowledge
Objective: State the average effective radiation dose from
chest CT.
Q. What is the average effective radiation
dose from chest CT?
A.
B.
C.
D.
1 mSv
8 mSv
16 mSv
24 mSv
Objectives at Different Levels
Level: Comprehension
Objective: Knows the disease or not
Q. A patient is 54 yr male who looks weak &
confused. We suspect it to be diabetes mellitus
Which of the sign & symptom serve best to
confirm diagnosis
A.
B.
C.
D.
Poor skin turgor with tenting
Recent decrease in appetite
Increase thirst and urination
Unexplained bruising of the abdomen
Objectives at Different Levels
Level: Application
Objective: Apply knowledge of clinical feature for diagnosis
Q. A 72-yr male who had several attacks of
coronary disease is admitted to hospital for
gangrene of one leg and found on admission to
have sign & symptom of intestinal obstruction.
Which is the most probable cause of
obstruction?
A.
B.
C.
D.
Cancer of colon
Intussusceptions
Mesenteric Thrombosis
Volvulus
Objectives at Different Levels
Level: Analysis and Synthesis
Objective: Analyze data and synthesize the diagnosis
Q. A 68-yr female stuporous with respiratory
distress. O/E:
Pulse – 136/min, weak
Resp – 32/min with crackles
BP – 60 mm Hg Palpatory
Skin appears cool and diaphoretic. Capillary Refill
Time is significantly delayed
Which is the most probable cause?
A.
B.
C.
D.
Cardiogenic Shock
Neurogenic Shock
Hypovolumic Shock
Septic shock
Item Analysis
ITEM ANALYSIS
Important step in the development of any assessment strategy
It is a statistical technique which is
used for selecting and rejecting the items
of the test on the basis of their difficulty
value and discriminated power


Item statistics are used to assess the
performance of individual test items
Statistical method used –
1.
2.
3.
Facility value – (item difficulty)
Discriminative index – (item discrimination)
Distractor efficiency
STEPS
1.
2.
3.
Evaluate answer card - MARKING
Arrange marks list in descending order
Divide whole list into 2 half


High achiever
Low achiever
If list is bigger – divide upper 30%or25%
and lower 30%or25%
EXAMPLE
Q. A 1-year-old infant is known to have
heart disease and is noted to be
cyanosed.
Which of the following is the most likely
diagnosis?
a. Atrial septal defect
b. Patent Ductus Arteriosus
c. Ventricular septal defect
d. Tricuspid atresia
Total 100
a
b
c
d
HAG(n=50)
0
02
08
40
LAG(n=50)
12
26
02
10
1. Facility Value (F.V)
Difficulty index
“The facility value of an item is defined as the
proportion or percentage of the examinees who
have answered the item correctly”
J.P. Guilford
Calculation
Proportion of respondents selecting the right answer to that item
FV =
No. of right answer (H + L)
X 100
N (Total no. of students)
a
b
c
d
HAG(n=50)
0
02
08
40
LAG(n=50)
12
26
02
10
40+10/100 = 50 %
RANGE



FV ranges from 0 – 100%
Low FV – too difficult Question
High FV – too easy Question
Easy – ≥ 85%
Medium – 51%-84%
Hard - ≤ 50
Significance




Determine that students have learned the
concept
Discriminate between students who knows
and who don’t
Help in designing better paper
Start test with easy (high FV) items
towards more difficult items (low FV)
2. Discrimination Index (D.I)
“It is the ability of an item to discriminate
between skilled and unskilled examinees”
- Blood and Budd(1972)
Calculation
Helps to differentiate who are knowledgeable
DI =
2 x (H - L)
N
a
b
c
d
HAG(n=50)
0
02
08
40
LAG(n=50)
12
26
02
10
2x(40-10)/100 = 0.6
RANGE



DI ranges from -> -1 to +1
DI = 1 indicates item perfectly
discriminates between those who
knows from those who don’t
Negative – either ambiguous question
or wrong key
RANGE

Item with high DI are preferred
≥ 0.35 – GOOD
<0.2 – 0.35> – INTERMEDIATE
≤ 0.2 - UNACCEPTABLE
Significance




Flaws in item can be identified
Provide feedback for improvement
Learning can be improved
Misconception in learning can be identified
3. Distractor Efficiency


Distractor should be PLAUSIBLE
Any distractor that is not picked by at
least 5% of student – is not a good
distractor
a
b
c
d
HAG(n=50)
0
02
08
40
LAG(n=50)
12
26
02
10
GOOD
Change the distractor
QUESTION BANKING


Large pool of question with set of
information regarding question
Goal – ability to deposit, discover and
retrieve question
Information on Card
Side A
Content Area: Nerve Muscle Physiology
Chapter: 21 (Muscle Blood Flow During Exercise)
Item:
Stem : main cause of increased blood flow to exercising
muscles is
Options:
a.
b.
c.
d.
Raised BP
Vasodilatation due to local metabolite
Increased sympathetic discharge
Increased Heart Rate
Key: Vasodilatation due to local metabolite
Reference: Guyton physiology 11/e p 247
Side B
Options
Year
Class
Q. No.
a
b
c
d
FV
DI
2009
PGET
122
21
18
61
25
49
0.35
2010
PGET
135
11
16
72
13
64
0.29
2011
PGET
148
28
21
69
14
52
0.31
2012
PGET
133
25
15
78
08
62
0.25