CLEAR 2008 Annual Conference

“Strategies and Outcomes of Pre-testing for Performance-Based Assessments”

Carol O’Byrne, PEBC Anchorage, Alaska

Objective Structured Clinical Examination (OSCE) • Why an OSCE?

• Examination Structure • OSCE Stations – Patient interactions – Health professional interactions – Non-interactive tasks CLEAR 2008 Annual Conference Anchorage, Alaska

Stations Types – some examples • SP Interactions – Provide information or advice – Counsel on new medication – Manage adverse drug reactions • HP Interactions – Assist with ‘medication management’ – Identify and resolve therapeutic problems (prescriptions, patient outcomes, etc.) • Non-client Stations – Review new orders – Reconcile medication orders CLEAR 2008 Annual Conference Anchorage, Alaska

Scoring Candidates’ Performance • Checklists – Critical items – Non-critical items – Incorrect items – Unique responses – Misinformation – Risk • Rating scales – Communication – Problem solving / outcome – Overall performance CLEAR 2008 Annual Conference Anchorage, Alaska

Station Development

• Author workshop • Editing • Review workshop #1 • Pre-test or bank for next/future exam?

• Review workshop #2 • Exam • Quality assurance, standard setting • Revise, create variations CLEAR 2008 Annual Conference Anchorage, Alaska

Why Pretest?

Check functionality of the station from the perspectives of – Candidates – Assessors – Standardized Clients • Check station difficulty and length • Confirm validity CLEAR 2008 Annual Conference Anchorage, Alaska

Why NOT Pretest?

• Exposure – Stations are memorable – Active grapevine • Timeliness or criticality of content – Issue is time-sensitive, e.g. Health Canada or WHO advisory CLEAR 2008 Annual Conference Anchorage, Alaska

Selection Criteria for Pre-testing

• New station format, candidate response format • New technology • Ill-defined problem, e.g. ethical dilemma • Complex station – More than one major objective – More than one problem • Emerging, future-oriented practice or role CLEAR 2008 Annual Conference Anchorage, Alaska

Pre-test Strategies

• Include in the examination (entry to practice candidates) – One station per track (in addition to the 15 scored stations) – Each station pre-tested in two ore more sites (n=80+) – Data from actual candidates, under exam conditions – SP and Assessor feedback questionnaires • Pre-test workshops (practising pharmacists) CLEAR 2008 Annual Conference Anchorage, Alaska

Feedback Questionnaires

• SPs – Questions, findings – Volunteer comments • Assessors – Questions, findings – Volunteer comments • Candidates – Questions, findings – Volunteer comments CLEAR 2008 Annual Conference Anchorage, Alaska

Case Studies

• Standardized Patient Interaction #1 – Pretest version, data and feedback – Revisions – Exam version, data and feedback – Recommendations • Standardized Patient Interaction #2 – No pretest – Exam version, data and feedback – Recommendations CLEAR 2008 Annual Conference Anchorage, Alaska

Recommendations / Options

• Assessor and SP feedback – Revise questionnaires - capture more information – Ensure that assessors and SPs complete the questionnaires • Sequester assessors and SPs to debrief and complete questionnaires • Explain how feedback is used; provide example(s) • Invite and record verbal comments CLEAR 2008 Annual Conference Anchorage, Alaska

Recommendations / Options

• SP Trainers and Chief Examiners – Develop feedback/report template/s • pilot report template(s) • include non-client and interactive stations – Gather questions/issues addressed during training and on exam day – Develop option to submit online in a secure manner CLEAR 2008 Annual Conference Anchorage, Alaska

Contact Information

Carol O’Byrne Manager, PEBC Qualifying Examination – Part II (OSCE) Pharmacy Examining Board of Canada 717 Church Street Toronto, ON M4W 2M4 Tel: 416-979-2431, ext 226 Email: [email protected]

Web Site: www.pebc.ca

CLEAR 2008 Annual Conference Anchorage, Alaska