Transcript Document

Question Writing Guide
ABNM SAM Questions
J. Anthony Parker, MD PhD
Associate Executive Director, ABNM
Beth Israel Deaconess Medical Center
Boston, Massachusetts
[email protected]
Caveat Lector
Psychometrics
Origin for Questions Writing Rules
Psychometrics is the field of study concerned with
the theory and technique of psychological
measurement, which includes the measurement of
knowledge, abilities, attitudes, personality traits,
and educational measurement. The field is primarily
concerned with the construction and validation of
measurement instruments such as questionnaires,
tests, and personality assessments.
– Wikipedia
Test Goal is to Measure Ability
Probability
Logistic regression where the probability of a correct
answer is a function of
Fn, the ability of person, n
bi, the difficulty of question, i
Difficulty of question, i, bi
Ability 0.3
Wikipedia
Ability of person , n, Fn
Can Also Measure Question Difficulty
Probability
Logistic regression where the probability of a correct
answer is a function of
Fn, the ability of person, n
bi, the difficulty of question, i
Ability of person , n, Fn
Difficulty 0.3
Wikipedia
Difficulty of question, i, bi
Question Discrimination
Probability
The slope determines how well a question
discriminates between candidate abilities
ai, ability discrimination
log-odds correct answer = ai • (Fn - bi)
Ability of person , n, Fn = 0.3
Slope, ai
Discrimination
Difficulty 0.3
Wikipedia
Difficulty of question, i, bi
ABNM Procedure
Assume questions are good and correctly keyed
Measure candidate tentative ability
Using tentative abilities measure question difficulty
Discard questions that are too hard or too easy
Also measure question-ability discrimination
Correct wrong keys
Discard questions that not correlated with ability
Using improved test, measure final candidate ability
Adjust to standard independent of candidates and test
Writing Good Questions:
Questions that Correlate with Ability
Psychometricians have developed rules
for writing good questions
Rules are based on question performance
Bad questions test abilities unrelated to
the test goal
Some Rules for Good Questions
No negatives (not, except, none)
No absolutes (always, never, all, every, only, must)
No overlapping ranges
No “all of the above” or “none of the above”
No mutually exclusive answers (one is correct)
Logical order (numerical, alphabetical, category)
Single concept per question
No multiple true / false questions
Which of the following is true
in Nuclear Medicine?
MIBG is used to image somatostatin receptors
A photon is completely absorbed during a
Compton interaction
18F-FDG
is transported into cells by GLUT
Lymphoma is never FDG-avid
Multiple true / false “which is true?”
Multiple unrelated concepts
“Never” is an absolute
With regard to the thyroid which is correct?
Pentetreotide is taken up in the parafollicular cells
Follicular cells are non-iodine-avid
Stromal cells take up iodide
Pentetreotide is not taken by the thyroid
Multiple true / false “which is correct?”
First and last answers are mutually exclusive
One of them must be correct
“Non” and “not” are negatives
In the thyroid, pentetreotide is
taken up MOST avidly in:
parafollicular cells (C cells)
follicular cells
colloid
stroma
In the thyroid, pentetreotide is
taken up MOST avidly in:
parafollicular cells (C cells)
follicular cells
colloid
stroma
Which of the following is not correct?
11C-acetate
is metabolized in the the TCA cycle
11C-choline
can be used for prostate cancer
18F-fluorodexoyglucose
(FDG) is not
phosphorylated in the cell
18F-fluorothymidine
(FLT) is correlated with cellular
proliferation
Not in stem and in 3rd answer
Multiple true / false “which is correct?”
Multiple unrelated concepts
MOST closely correlated
with proliferation?
11C-acetate
11C-choline
18F-fluorodexoyglucose
18F-fluorothymidine
(FDG)
(FLT)
MOST closely correlated
with proliferation?
11C-acetate
11C-choline
18F-fluorodexoyglucose
18F-fluorothymidine
(FDG)
(FLT)
Good distractors are often partially true,
but not the best answer
What is the BEST administered dose to
use for a 18F-FDG PET pediatric scan?
≤ 1 MBq/kg (≤ 0.03 mCi/kg)
1 - 4 MBq/kg (0.03 - 0.11 mCi/kg)
4 - 16 MBq/kg (0.11 - 0.43 mCi/kg)
10 - 64 MBq/kg (0.27 - 1.73 mCi/kg)
Overlapping ranges
4-16 and 10-64 both include 10-16
1-4 and 4-16 both include 4
What is the BEST administered dose to
use for a 18F-FDG PET pediatric scan?
1 MBq/kg (0.03 mCi/kg)
4 MBq/kg (0.11 mCi/kg)
16 MBq/kg (0.43 mCi/kg)
64 MBq/kg (1.73 mCi/kg)
What is the BEST administered dose to
use for a 18F-FDG PET pediatric scan?
1 MBq/kg (0.03 mCi/kg)
4 MBq/kg (0.11 mCi/kg)
16 MBq/kg (0.43 mCi/kg)
64 MBq/kg (1.73 mCi/kg)
Gelfand: Pediatric radiopharmaceutical administered
Doses: 2010 north american consensus guidelines.
J Nucl Med 2011;52:318
Parallel Answers / Avoid Teaching
Answers should be parallel in construction
and length
Good test takers know that “teachers like to teach”
The correct answer is often the longest
Tendency to make correct answer exactly correct
For testing, avoid teaching
Urinary excretion of FDG is related to:
FDG concentration overwhelms GLUT
renal GLUT has low affinity for FDG
both of the proximal and distal convoluted tubule
sodium glucose transporters (SGLT1 & SGLT2
respectively) have low affinity for FDG
there is little hexokinase in the kidney
Asymmetric answers
Teaching
Urinary excretion of FDG is related to:
FDG concentration overwhelms GLUT
renal GLUT has low affinity for FDG
SGLT1/2 have low affinity for FDG
there is little hexokinase in the kidney
Urinary excretion of FDG is related to:
FDG concentration overwhelms GLUT
renal GLUT has low affinity for FDG
SGLT1/2 have low affinity for FDG
there is little hexokinase in the kidney
Glucose Reuptake in the Kidney
Stem (Question)
1. Ask a question
Given …, what is the MOST likely diagnosis?
2. Give and incomplete statement
Given …, the BEST next step is:
3. Describe a situation followed #1 or #2
A one-year-old girl had jaundice and a
5 x 6 cm mass in the porta. Total bilirubin
is 7 mg/dL. …
Stem (Question)
Put most of the content in stem
Answers short
Move common words in answers to stem
Remove unnecessary words or sentences
In a 73-year-old man with metastatic
melanoma, what is the MOST likely cause
of the FDG uptake shown in the figure?
prostate cancer
melanoma
normal bladder
normal urethra
In a 73-year-old man with metastatic
melanoma, what is the MOST likely cause
of the FDG uptake shown in the figure?
prostate cancer
melanoma
normal bladder
normal urethra
Distractors (Incorrect Answers)
Should be plausible
Logical misconceptions
Common misunderstandings
Reasonable
not the BEST answer
not the MOST likely answer
Testing vs. Teaching
Wright Map
Testing
candidates
items
more able
hard
less able
easy
distinguish candidates
difficult items
judgment required
low passing %
Teaching
help learning
easier items
emphasize key points
Question Taxonomy
Recall (fact)
The energy of 99mTc is:
Interpretive (diagnosis)
A patient with … had an FDG-PET/CT shown
in the figure. The MOST likely diagnosis is:
Problem solving (management, judgment)
A patient with … had an FDG-PET/CT shown
in the figure. The BEST next step is:
Higher Taxonomy Preferred
Often involve more advanced reasoning
More complex medical task
Still single clinical concept
Distractors may be easier to write
Some topics require lower taxonomy questions
Stable iodine (SSKI or Lugol’s solution)
should be administered prior to:
pentetreotide (Octreoscan®)
iobenguane (MIBG)
a contrast FDG-PET/CT
ibritumomab tiuxetan (Zevalin®)
Stable iodine (SSKI or Lugol’s solution)
should be administered prior to:
pentetreotide (Octreoscan®)
iobenguane (MIBG)
a contrast FDG-PET/CT
ibritumomab tiuxetan (Zevalin®)
Recall question
pentetreotide confused with MIBG
ibritumomab tiuxetan confused with tositumomab
Based on the electrocardiogram, what
is the BEST diagnosis?
Difficult to come up with 4 plausible diagnoses
A 45 y/o diabetic man with weakness
and shortness of breath for 3 days
presents for exercise MPI. The preexercise ECG is shown.
What should be done next?
Obtain right precordial leads.
Use dipyridamole stress instead.
Accompany the patient to the EW.
Proceed as planned.
What should be done next?
Obtain right precordial leads.
Use dipyridamole stress instead.
Accompany the patient to the EW.
Proceed as planned.
Writing Distractors
Difficulty writing reasonable distractors
Change taxonomy
Management distractors often easier to write
The BEST diagnosis is:
The BEST diagnosis is:
normal thymus
mediastinal metastases
?
?
Diagnostic question
The uptake in the chest is related to:
The uptake in the chest is related to:
the Na / I symporter
somatostatin receptor, type 2
brown fat
metastases
Diagnosis & biodistribution
A syringe of 90Y-ibritumomab tiuxetan
(Zevalin ®) measuring 1025 MBq (27.7 mCi)
needs to be discarded due to QC failure.
After transfer to a vial, measurements are
vial: 815 MBq (22.0 mCi), syringe: 15 MBq
(0.4 mCi). What should be done next?
Declare a medical event
Survey the area
Call the radiation safety officer (RSO)
Store for decay
A syringe of 90Y-ibritumomab tiuxetan
(Zevalin ®) measuring 1025 MBq (27.7 mCi)
needs to be discarded due to QC failure.
After transfer to a vial, measurements are
vial: 815 MBq (22.0 mCi), syringe: 15 MBq
(0.4 mCi). What should be done next?
Declare a medical event
Survey the area
Call the radiation safety officer (RSO)
Store for decay
Which Isotope is MOST Difficult to
Accurately Measure in a Dose Calibrator?
18F
90Y
99mTc
111In
Which Isotope is MOST Difficult to
Accurately Measure in a Dose Calibrator?
18F
90Y
99mTc
111In
What radiopharmaceutical was administered?
99mTc-pertechnetate
67Ga-citrate
111In-pentetreotide
123I-MIBG
What radiopharmaceutical was administered?
99mTc-pertechnetate
67Ga-citrate
111In-pentetreotide
123I-MIBG
What’s the Radiopharmaceutical?
Stupid Questions, but I Use Them
Should know
normal and common variant biodistribution
at organ, cell, and receptor level
To be able to
identify abnormal biodistribution
identify radiopharmaceutical QC
understand unusual findings
After injection of 111In-DTPA in a patient
with normal renal function?
the half-time of excretion is 16 minutes
the half-time of excretion 8 minutes
the half-time of excretion 1.7 hours
the half-time of excretion 36 minutes
Repeated words in answers
How long after injection of 111In-DTPA is
1/2 excreted with normal renal function?
16 minutes
8 minutes
1.7 hours
36 minutes
Answers not in logical order
How long after injection of 111In-DTPA is
1/2 excreted with normal renal function?
8 minutes
16 minutes
36 minutes
1.7 hours
How long after injection of 111In-DTPA is
1/2 excreted with normal renal function?
Renogram
8 minutes
16 minutes
36 minutes
1.7 hours
http://www.doseinfo-radar.com/
Some Rules for Good Questions (1/2)
No negatives (not, except, none)
No absolutes (always, never, all, every, only, must)
No overlapping ranges
No “all of the above” or “none of the above”
No mutually exclusive answers (one is correct)
Logical order (numerical, alphabetical, category)
Symmetric answers
Some Rules for Good Questions (2/2)
Put concepts and common words in stem
Distractors should be reasonable
Try to use higher taxonomy
Often easier to write distractors
Some fact questions are appropriate
Single concept per question
No multiple true / false questions
Medically, not question, complexity
Reference Guidelines
3-page guide
ABNM_Well_Formatted_SAM_Question.docx
1-page list of does and don’ts
ABNM_QuestionWritingDoesAndDonts.docx
26-page guide (ABNM psychometric consultants)
http://www.measurementresearch.com/
24-page guide
ABR_Item_Writers_Guide.pdf
Applause