Test Item Writing:

Higher Order Thinking

Agenda

• Assessment vs. Evaluation • Forms of assessment • Item mechanics • Best practices • Leveled questioning • Practice

Assessment Writing:

I also want to know …

Assessment vs. Evaluation

• Assessment • Focuses on learning, teaching, and outcomes •

Evaluation

– Focuses on grades or scores.

• Provides information for improving learning and teaching. • An interactive process between students and faculty that informs faculty how well their students are learning what they are teaching. – May reflect classroom components other than course content and mastery level.

Forms of Assessment

• Case studies • Direct observation • Examinations • Knowledge surveys • Self assessments • • • • Simulations Portfolios Presentations Performance projects

Spotlight: Examinations

• Multiple Choice Questions (MCQ) • Short Answer (SA) • Essay (vignettes, scenarios, case studies) • Performance (demonstration, presentation, case studies)

Terminology

Item

• Question Mechanics • A 30-yr old man presented with a 4-month history of dyspnea, low-grade fever, cough, and fatigue. Given the following chest radiograph, what is the most likely diagnosis?

a) adult respiratory distress syndrome

Keyed response

b) c) pulmonary alveolar proteinosis pulmonary edema

d) streptococcal pneumonia Options Stem Distractors

• Item • Relate to instructional objectives • Test at the same level of learning as objectives • Construct items to measure different levels of learning (i.e. Bloom’s Taxonomy)

Test Writing Best Practices

• Stem • Use complete statements • Determine level of complexity (Bloom’s taxonomy) • Include only relevant information and graphics • Keep succinct • Question for the correct answer, not the ‘wrong’ answer • Avoid absolutes: “always”, “never”, “all”, or “none” • Avoid imprecise terms: “seldom”, “rarely”, “occasionally”, “sometimes”, “few”, or “many” • Avoid cues: “may”, “could”, or “can” • Define all eponyms, acronyms, or abbreviations

Test Writing Best Practices: MCQ

• Options • Keep grammatically consistent with stem • Incorrect options should be clearly incorrect, but plausible • Ensure options are categorized together (all diagnoses, tests, etc.) • All options similar to the correct answer regarding lengthy, grammar, and complexity • Avoid “none of the above” or “all of the above” • Place options in logical order (numerical, chronological, etc.) • Ensure options are independent not overlapping • Provide 3 to 5 options • Vary keyed responses (B, C, D, A instead of B, B, A, B,)

Item Patterning: Medical Education

A basic pattern can help formulate a good stem A (describe the patient with age, gender, and ethnicity, if pertinent) presents with (give the signs , symptoms or other pertinent findings) to the (list site of presentation) in (describe the condition of the patient). Which of the following is the most likely caused this

presentation?

You may want to start here

List 4 or 5 Appropriate Options of roughly equal length

(homogeneous, i.e., list of five drugs or conditions or disorders responsible for the presentation cited). Follow guidelines for answer options carefully Compliments of NBOME, 2010.

Item Patterning Example

A (

patient description

) presents complaining of (

somatic/visceral symptoms

). A structural examination reveals (

structural and palpatory findings

). The most likely diagnosis is or the most appropriate treatment for this presentation is?

Option list (

list five treatments as options

).

Compliments of NBOME, 2010.

Leveled Question Formatting for Higher Order Thinking

•

Remembering factual knowledge:

• Write the question as a factual statement.

• • Offer similar, but discernable options Ask: Provide the answer using a series of words or phrases (select) •

Understanding conceptual knowledge:

• Write the question as a factual statement.

• Ask: Provide the answer using fill-in-the-blank. (identify) • • • •

Applying procedural knowledge:

• Provide a diagram or data set and ask for application • Ask: Given the following chart, which of the following is most likely to occur next? (predict)

Analyzing conceptual knowledge

• Provide a diagram or data set and ask for analysis.

• Ask: Given the following data, which illustrates a significant savings? (calculate)

Evaluating procedural knowledge

• Provide a case study, scenario, or current event situation • Ask: Which of the following would have been a better plan of action? (compare/contrast)

Creating a solution

• Provide a case study, scenario, or current event situation • Ask: Develop a plan to resolve this problem. (create)

Leveled Questions: Application

Remember/Understand

Leveled Questions: Analysis

Apply

Leveled Questions: Evaluation

Analyze

Leveled Questions: Creation

Evaluate

MCQ Critique

1. Identify the problem areas of each question 2. Modify the question so that it is correctly written

Test Writing Best Practices: Short Answer and Essay Exams

• Stem • Consistent with MCQ best practices • Provide only pertinent information • Present in a clear and consistent manner • Option Response: (rubrics, checklists) • Defined criteria • Scaled using levels of mastery to determine quality of the area being rated • Set target goals (consistency)

SA and Essay Question Format Example

1.

Identify 1 concept you want your students to be able to do/know in your program.

Apply appropriate policy management techniques 2.

3.

List what these key words are or would entail: policy management techniques: • planning • organizing • decision making • staffing • motivating • budgeting Identify when this concept would be used Developing and implementing new federal policies that implement campaign promises acquired from the newly elected President.

4.

Identify someone who may do this in your profession and a situation where this may occur Elvis, is the Assistant Director of the ASPA.

Item Example

• Elvis, is the Assistant Director of the ASPA. He has been asked to help develop and implement new federal policies that implement campaign promises acquired from the newly elected President. • Discuss how Elvis would accomplish this task by applying his knowledge of appropriate policy management techniques.

Rubric Response

Rubric Component (A-B)

Concept #1

(Apply policy and management skills)

Descriptors (C)

Key Concepts

• planning • organizing • decision making • staffing • motivating • budgeting

Beginning 1 Developing 2 Exemplary 3

The student was only able to discuss the following (1 2) key concepts in applying policy management skills relating to the given scenario.

The student was only able to discuss the following (3 5) key concepts in applying policy management skills relating to the given scenario.

• planning • organizing • decision making • staffing • motivating • budgeting • planning • organizing • decision making • staffing • motivating • budgeting The student was able to appropriately discuss the following (6) key concepts in applying policy management skills related to the given scenario.

• planning • organizing • decision making • staffing • motivating • budgeting

Performance Critique - handout

1. Identify the problem areas of each question 2. Modify the question or answer so that it is correctly written

Revise in Small Doses

References

Collins J. Education techniques for lifelong learning: writing multiple-choice questions for continuing medical education activities and self-assessment modules. Radiographics. 2006;26:543–51. JozefowiczRF, Koeppen BM, Case S, Galbraith R, Swanson D, Glew H. The quality of in-house medical school examinations. Academy of Medicine 2002;77: 156–161.

Kemp JE, Morrison GR, Ross SM. Developing evaluation instruments. In: Designing Effective Instruction. New York, NY: MacMillan College Publishing Company, 1994:180-213.

Palmer E, Devitt PG. Assessment of higher order cognitive skills in undergraduate education: modified essay or multiple choice questions? Research paper. BMC Med Educ. 2007 Nov 28;7:49.

Schuwirth LWT, van der Vleuten CPM. Different written assessment methods: what can be said about their strengths and weaknesses? Medical Education 2004; 38:974-979.

The National Board of Medical Examiners. Constructing written test questions for the basic and clinical sciences. Available at: http://www.nbme.org/publications/item-writing-manual.html

Accessed October 20, 2010.

More Information…

Test Item Writing

http://digitalcommons.hsc.unt.edu/test_items/

Christine Savi, PhD

CLD & QEP: Assessment 817.735.2942