The Learning Side of Testing

Milwaukee Area Technical College

Dental Hygiene Program

February 25, 2005

What does this mean? What does it tell us about how testing fits into performance-based learning?

Test to Learn

Learn to Test

National Board Results Testing Practices Other

Situational Analysis

Plus Minus Interesting

Situational Analysis

What do we know about National Board Results?

250 200 150 100 50 0 2001 2002 2004

MATC performance in comparison to all others.

MATC All

Situational Analysis

What do we know about National Board Results?

50.00% 45.00% 40.00% 35.00% 30.00% 25.00% 20.00% 15.00% 10.00% 5.00% 0.00% April December 2000 2001 2002 2003 2004

MATC failure rate.

Situational Analysis

What do we know about National Board Results?

1 0 -1 -2 -3 -4 -5 Anat. Sci Physo-Bio-Nutr Micro-Immun Patho Pharm Patient Assess Radiology Care Mgt.

Perio Supp TRT Comm Hlth Case-Based

Situational Analysis

What do we know about testing practices?

Situational Analysis

What else might impact the issues?

Testing as Learning

Testing promotes learning.

Testing requires learning.

Practice and Learning Sensing Memory



selector



large capacity



short duration

Practice and Learning Working Memory



processor



powerful



fragile

short duration small capacity

Practice and Learning Long-Term Memory



information storage

 

large capacity requires retrieval

Five mental processes control information flow:

attention ( ( sensing ) ) encoding ( ( working / long-term ) rehearsal/practice ( ( working ) retrieval ( long-term / / working ) ) metacognition ( working )

Practice and Learning

Learning is enduring and meaningful only when the:

sensing memory attends to the stimulus working memory processes it and sends it to long-term memory long-term memory stores it

Practice and Learning

Learners apply what they’ve learned only when they can retrieve it from long-term memory .

Practice and Learning

Research has found that providing learners with retrieval practice is much more powerful than giving them the same amount of time to restudy the learning material. Typical improvements range from 30% to 100%.

Will Thalheimer

Simulation-Like Questions: How and Why to Write Them

www.work-learning.com

Practice and Learning

Metacognition

Learners need feedback in order to take responsibility for monitoring and adjusting their own learning.

. . . hmm, the answer to this question isn’t what I thought it was. What do I need to do about it?

Practice and Learning

Feedback is essential when learners answer questions because it helps them correct their misconceptions. Feedback typically improves learning outcomes by 15 – to 50%.

Will Thalheimer

Simulation-Like Questions: How and Why to Write Them

www.work-learning.com

Practice and Learning

Too much information too fast yields

Fried Brain Syndrome

Practice and Learning

Fried brain syndrome leads to:

Overload Overload

Cognitive overload slows down or even stops learning!

Practice and Learning

To avoid cognitive overload:

•

break learning into manageable chunks

•

clear the working memory with meaningful practice

Performance vs. Practice

• Intended to accomplish a task • Real thing/big bites • On your own • High stakes • Feedback after the fact • Limited opportunities to correct • School of hard knocks • Intended to promote learning • Protected/chunked • Coached • Low stakes • Concurrent or immediate feedback • Repetition • Built-in opportunities to correct

Testing as Learning

Tool Description

Educational Assessment Exam Test Quiz CAT

Process of obtaining evidence about the learners’ knowledge, skills, and attitudes.

Measure/document knowledge or skills for the purpose of credentialing (graded). Measure knowledge or skills for the purpose of informing the learner and teacher about the learner’s current level of knowledge or skill. (likely to be graded) Measure a learner’s knowledge or skills for the purpose of providing feedback and identifying gaps.

Provide immediate feedback to teachers and learners to guide adjusts in teaching and learning.

Measure vs.

Learn Stakes Measure Measure Promote Learning Promote Learning Promote Learning High Medium Low Very Low Slide from derived http://questionmark.com

Testing as Learning

What does this mean?

Testing provides practice.

Testing requires practice.

Testing as Learning

Strategies for Using Testing as Learning

 Require learners to “test drive” their knowledge by completing low stakes quizzes at all stages of learning. [Could be self-check.]  Formulate classroom quizzes and tests using the same test formats used in the National Boards.

 Present learner with questions that require higher level thinking to make decisions, determine solutions, or draw conclusions.

Testing as Learning

Strategies for Using Testing as Learning

 Design questions so that learners apply knowledge in the context of Dental Hygiene practice.

 Provide corrective feedback for incorrect answers.

 Build in branching to present additional questions after missed questions.

 Teach learners early and often how to succeed on tests and quizzes.

Question Writing Process

What do you know about writing multiple choice test items?

1. What should you do?

2. What makes a good question?

3. What should you avoid?

Question Writing Process

• Start with your competencies and learning objectives • Write question stems • Write correct answers • Write distractors • Edit

Start with target competencies and learning objectives to:

• ensure validity • ascertain significance • check coverage • determine scope of test

Write the Stem

• Derive the question from a competency or learning objective.

• Identify target decision, conclusion, or solution.

• Present a single, definitive problem with a definitive answer. • Make stem grammatically consistent and parallel in form to options.

Write the Stem

What to avoid:

• Negatively phrased questions (used in National Boards) • Absolutes such as “always” and “never” • Unnecessary information • Obvious clues about the answer • Bias (gender, cultural, ethnic, regional) • Humor and trick questions

Avoid Grammatical Clues

Option A is the only option that makes grammatical sense. The best way to increase the reliability of a test is to: A. Increase the test length. [Correct] B. Removing poor quality items. C. Test should be readable for all test takers.

Avoid Association Clues

Option B includes the word “seven.” Who are the “Magnificent Seven?” A. A pro basketball team [Correct] B. A group of seven fictional, heroic, Western figures [Correct] C. A protest group made famous during the 1960s D. A vegetarian rock group

Write the Options

• Write three to five options. • Make all options grammatically consistent and parallel in form to stem.

• Write all options so they are comparable in content, length.

• Ensure that options are not repetitious.

• Order them logically or numerically. • Avoid “All of the above” and “None of the above.”

Order Options Numerically

What is the cost of an item that normally sells for 9.99 USD that is discounted 25%?

Wrong A. 5.00

B. 7.50

C. 2.50

D. 6.66 Right A.

2.50 B.

5.00

C.

6.66

D.

7.50

Order Options Logically

What are the three important concerns in achievement testing? V=validity R=reliability E=efficiency O=objectivity Wrong A. V, R, E B. O, V, R C. R, O, E D. E, O, V Right A.

V, R, O B.

V, R, E C.

V, E, O D.

R, E, O

Write the Options - Distractors

• Write two to four distractors (giving 3-5 options).

• Make all distractors plausible.

• Make distractors similar in form or otherwise harmonious with the answer.

Write the Options - Correct Answer

• Be sure that correct answer is the only correct one or clearly the best one.

• Vary the correct answer position in the list of options.

Raising the Level of Thinking

• Evaluation • Synthesis • Analysis • Application • Comprehension • Knowledge Critical Thinking

Raising the Level of Thinking

Evaluation Synthesis Analysis Application Comprehension 

Knowledge define label list recall recognize

Raising the Level of Thinking

Evaluation Synthesis Analysis Application 

Comprehension

Knowledge

explain paraphrase describe summarize

Raising the Level of Thinking

Evaluation Synthesis Analysis 

Application

Comprehension Knowledge

apply decide determine use

Raising the Level of Thinking

Evaluation

classify

 Synthesis

Analysis compare describe structure determine cause/effect

Application

detect patterns

Comprehension

diagnose

Knowledge

Raising the Level of Thinking

Evaluation 

Synthesis

Analysis Application Comprehension Knowledge

construct create design develop formulate write

Raising the Level of Thinking



Evaluation

Synthesis Analysis Application Comprehension Knowledge

appraise assess conclude evaluate prescribe

Raising the Level of Thinking

How do you help learners practice higher level thinking skills?

Continually present questions that require them to:

 make a decision  draw a conclusion  solve a problem

Tools

• Higher Level Question Worksheet • Case-Based Question Planner

Perception

A Web-based Tool for Q&A Assessment

Report on the results. Reports include coaching reports, item analysis, gap analysis, and others.

Create questions and group into quizzes, tests, exams and surveys using a web browser or Windows PC Deliver assessment using a standard, or secure, browser, or using a Windows PC Specify security parameters, schedule assessments and potentially link to other systems

The Dental Hygiene National Boards High Stakes Exam Consequences to candidate Legal liabilities Proctoring standards Validity & Reliability checking Planning (testing and item significance) Psychometrician involvement Low Medium High

Slide from http://questionmark.com

The Dental Hygiene National Boards

   

Context and Purpose

   Intended to assess ability to apply knowledge in a problem solving context Key to qualifying for licensure Developed by dental hygiene and science educators Subject to on-going validity and reliability checking Administered in highly secure environment with significant consequences for cheating and breach of security Administered according a consistent set of strict rules Administered to a large, diverse population across a broad geographic area.

The Dental Hygiene National Boards

  

Scope



Discipline-Based (200 questions)

 Scientific Basis for Dental Hygiene Practice  Provision of Clinical Hygiene Services  Community Health Services

Behavioral Sciences (infused into 5%) Ethics and Risk Management (infused into 5%) Cased-based items (150

 Assessing patient characteristics  Obtaining and interpreting radiographs  Planning and managing dental hygiene care  Performing periodontal procedures  Using preventative agents  Providing supportive treatment service

The Dental Hygiene National Boards

  

Scope



Discipline-Based (200 questions)

 Scientific Basis for Dental Hygiene Practice  Provision of Clinical Hygiene Services  Community Health Services

Behavioral Sciences (infused into 5%) Ethics and Risk Management (infused into 5%) Cased-based items (150

 Assessing patient characteristics  Obtaining and interpreting radiographs  Planning and managing dental hygiene care  Performing periodontal procedures  Using preventative agents  Providing supportive treatment service

The Dental Hygiene National Boards Format – Multiple Choice: Seven Question Types



Completion



Question



Negative



Paired True-False



Cause-and-Effect



Testlet



Case-Based

The Dental Hygiene National Boards Format – Completion Typical Level – Knowledge Example

Radiographic intensifying screens are used for A. magnifying lamps.

B. increasing detail.

C. reducing the exposure time.

D. speeding processing time.

E. decreasing the processing time.

The Dental Hygiene National Boards Format – Question Typical Level – Knowledge, Understanding, Application Example (all)

Root hypersensitivity diminishes as the tooth forms which of the following?

A. Mantle dentin B. Secondary dentin C. Cellular cementum D. Acellular cementum

The Dental Hygiene National Boards Format – Negative Typical Level – Knowledge, Understanding, (all) Example

Each of the following characterizes an ideal chemical disinfectant EXCEPT one. Which is the EXCEPTION?

A. Heat-stable B. Long shelf-life C. Water solubility D. Activity against vegetative microbes E. Activity against microbial spores

The Dental Hygiene National Boards Format – Paired True-False Typical Level – Understanding, Analysis, Evaluation Example

Protection from excessive exposure to radiation is aided by the use of aluminum filters and a lead diaphragm.

The filters reduce the amount of soft radiation.

A. Both statements are TRUE.

B. Both statements are FALSE.

C. The first statement is TRUE; the second is FALSE.

D. The first statement is FALSE; the second is TRUE.

The Dental Hygiene National Boards Format – Cause-and-Effect Typical Level – Analysis Example

Adolescent growth spurts more in the maxilla than in the mandible BECAUSE, at puberty, the lymphoid tissue present in the nasopharynx decreases.

A. Both the statement and the reason are correct.

B. The statement is correct, but the reason is NOT.

C. The statement is NOT correct, but the reason is correct.

D. NEITHER the statement NOR the reason is correct.

The Dental Hygiene National Boards Format – Testlet (Community Health) Typical Level – Application, Analysis, Evaluation Description

Presents a community health scenario followed by five or more multiple-choice items that address its content. Questions are likely to be in the question format, but could be in any format.

For an example see 2005 National Board Dental Hygiene Examination Candidate Guide, p. 40.

The Dental Hygiene National Boards Format – Case-Based (Patient Care) Typical Level – Application, Analysis, Evaluation, Synthesis Description (12-15 on exam)

Present patient histories, dental charts, radiographs (sometimes intra- and extra-oral photographs). Test items, typically in the question format, address functions that a dental hygienist is expected to be able to perform.

Exams will include at least one case regarding the following types of patients: geriatric, adult periodontal, pediatric, special needs, medically compromised.

For an example see 2005 National Board Dental Hygiene Examination Candidate Guide, pp. 41-45.

Enhancing Your Tests

Critique existing tests.

   

Validity Test items Feedback Use of results

Enhancing Your Tests

Critique existing tests.

Strengths Weaknesses Interesting

Enhancing Your Tests

Outline test improvement strategies.

Enhancing Your Tests

 Create an assessment plan that includes both performance assessment tasks and Q&A tests.

 Use competencies and learning objectives to plan Q&A tests: ensure validity, ascertain significance, check coverage and determine scope.

Enhancing Your Tests

 Review and revise existing tests and exams using the

Checklist for Q & A Quizzes and Tests

.

 Link all test questions to a stated learning outcome (ex. competency, learning objective).

 Increase the number of higher thinking level questions.

 Incorporate feedback that corrects misconceptions and refers learners who answer incorrectly to remedial or further learning for each question.

 Incorporate branching to direct learners to remedial, similar, or higher level questions, as needed. (Online delivery)  Increase the variety of questioning formats used in all quizzes and tests.

Enhancing Your Tests

 Consider using a tool like Questionmark to:  support online delivery  increased security  authoring of varied question formats  immediate feedback  branching for additional testing  referral to learning materials  ease of scoring  analysis of results, and competency/learning objective driven test development

Enhancing Their Learning

Provide effective practice throughout at all phases and levels of the learning process.

Maintenance Practice/Rehearsal Elaborative Practice/Rehearsal

•

keeps information in working memory

•

clutters working memory

•

inefficient

•

connects with what learners already know

•

applies to solution of problems

• •

creates something new efficiently encodes to long-term memory Effective practice is elaborative.

Enhancing Their Learning

Provide effective practice throughout at all phases and levels of the learning process.

Maintenance Practice/Rehearsal Elaborative Practice/Rehearsal

• • •

memorization repetition recitation

•

drill & practice

• •

classification

•

analysis skill practice

• •

critique problem-solving

•

product creation Effective practice is elaborative.

Enhancing Their Learning

 Make them work for it by using learner centered, active learning activities at the comprehension phase.

 Guide learners to process information using thinking maps and application exercises  Engage learners in collaborative information gathering and processes activities

Enhancing Their Learning

 Present Thinking Maps ® or other visual learning tools; have learners create visual learning tools.

 Engage learners in the process of developing higher level thinking questions and case studies.