Transcript Document

Kim Cavanagh, DHSc, PA-C, Chair, Assessment Institute
Michel Statler, MLA, PA-C, Senior Director, Academic Affairs

Overview of the new platform for all PAEA
testing products
 Exam ordering
 Exam administration
▪ Entering student and class information
▪ Setting up proctors
 Score reporting
Assessment is a required part of our programs
Assessment takes many forms and occurs at
various times throughout the curriculum
 Creating high quality assessments takes time
and requires continued attention
 A survey of program directors in the Fall of 2010
revealed that 78% of programs were interested
or very interested in an end of rotation
examination product


The Test Item Bank Advisory (TIBA) Committee was
formed to explore the potential offering of additional
exam products by the PAEA
 Their recommendations included:

 Development of the following new testing products:
▪ End of rotation examinations for core rotations
▪ Potential comprehensive examination
 PACKRAT to remain in its current form
 Creation of a new Assessment Institute to include:
▪ Advisory Council
▪ Rotation Exam Review Committees
▪ Item Writing Subcommittees
Assessment
Institute
PACKRAT
Item Writing
Subcommittee
Rotation Exam
Review
Committees (3)
Item Writing
Subcommittees
Comprehensive
Exam Review
Committee
Item Writing
Subcommittee

Three Rotation Exam Review Committees
(RERCs) were assembled early 2012
 Internal Medicine/Family Medicine
 Women’s Health/Pediatrics/Psychiatry
 Emergency Medicine/Surgery

Members of these committees are
responsible for new item generation based
upon specific item writing assignments, as
well as review of items.



Each Rotation Exam Review Committee has
an Item Writing Subcommittee (IWSC) to
assist with new item development
These members write items based upon a
specific writing assignment
No responsibility for item review or in person
meeting
Fixed exam forms
Core content areas
Follow specific blueprints
120 questions per exam (100 scored, 20 non-scored
for statistics gathering only)
 Multiple choice
 Two versions at time of
release with additional
versions to follow




CONTENT AREAS
TASK AREAS
Internal medicine
 Family medicine
 Emergency medicine
 General surgery
 Women’s health
 Pediatrics
 Psychiatry


Diagnosis
 History and physical
examination
 Diagnostic studies
 Clinical therapeutics
 Clinical intervention
 Health maintenance
 Scientific concepts

Questions for initial versions will come from:
 Existing item bank
 New items generated by review committees and
subcommittees of item writers

Items submitted by PA programs in response
to one-time request will be integrated into
future versions following review by RERCs
and medical editors

Each item is coded by topic (organ
system and content code) and task
 Items are also assigned keywords
All items are peer reviewed by RERC members,
edited by trained medical editors and then
undergo a second peer review by the RERC prior
to item selection and exam assembly
 Each examination form is assembled following
the specific exam blueprint

© Copyrighted by the Physician Assistant Education Association, 2012


Items will be reviewed annually for
performance and the process for refreshing
the item pool will be ongoing
Programs will receive feedback on the items
submitted once integrated
into an exam
Student reports and program reports will be
generated at the end of each exam
administration
 Reports will be available through the Program
Exam Management System
 Student reports will include:
 Individual raw score
 Breakdown by organ system
and task area
 Keywords of items missed

Student report comparative data will be
updated weekly (more stability as more
students test)
 Delay in scoring until 300
students see test form
 Reports will be available for
2 years or until test forms are
retired

PACKRAT Performance Report
Suzie Q. Smith
Exam Date: 10/17/2012
Raw Score
10
Feedback by Content Area of the PACKRAT
Content Area
Your %
Content
of
Your
Blueprint Items
Percentile
Correct
SPECIALTIES
Cardiology
16%
0%
0
Dermatology
5%
0%
0
Endocrinology
6%
7%
0
ENT
9%
23%
0
10%
4%
0
Hematology
3%
0%
0
Infectious Diseases
3%
0%
0
Neurology
6%
7%
0
Obstetrics/Gynecology
8%
0%
0
Ophthalmology
6%
14%
0
10%
0%
0
6%
7%
0
12%
7%
0
6%
0%
0
Clinical Intervention
14%
6%
0
Clinical Therapeutics
18%
2%
0
Diagnosis
18%
0%
0
Diagnostic Studies
14%
0%
0
Health Maintenance
10%
13%
0
History and Physical
16%
5%
0
Scientific Concepts
10%
9%
0
Gastrointestinal/Nutritional
Orthopedics/Rheumatology
Psychiatry/Behavioral Medicine
Pulmonology
Urology/Renal
TASKS
Feedback by Keyword
Content Category Coding, Task Area, Diagnosis
The following list includes keyword feedback for the questions you answered incorrectly on the exam. If
you missed multiple questions in one Content Category, the topic is only listed once. Therefore, you
should not assume that the number of entries below directly corresponds to the number of questions you
answered incorrectly.




















Cardiology: Vascular Disease, Health Maintenance, Abdominal Aortic Aneurysm screening
Cardiology: Vascular Disease, Health Maintenance, Deep vein thrombosis prevention
Cardiology: Vascular Disease, Health Maintenance, Varicose veins
Dermatology: Tumors of Skin, Scientific Concepts, Squamous Cell carcinoma
Dermatology: Viral Infections, Diagnosis, Varicella Zoster
Endocrinology: Other Endocrine Diseases, History and Physical, ACTH anterior pituitary tumor
Endocrinology: Other Endocrine Diseases, Scientific Concepts, Anterior pituitary tumor
ENT: Nose/Sinuses/Throat, Scientific Concepts, Acute Bacterial Sinusitis
Gastrointestinal/Nutritional: Stomach, Clinical Therapeutics, Helicobactor pylori
Gastrointestinal/Nutritional: Stomach, Scientific Concepts, Zollinger-Ellison syndrome
Hematology: Other Hematologic Diseases, Diagnosis, Fever of unknown origin
Hematology: Polycythemia, Diagnostic Studies, Polycythemia vera
Infectious Diseases, Scientific Concepts, human immunodeficiency virus
Neurology: Other Diseases of the CNS, Diagnosis, Tourettes association with OCD
Neurology: Vascular Diseases, Diagnostic Studies, Aneurysm rupture as a cause of CVA
Neurology: Viral Infections, Diagnostic Studies, herpetic encephalitis
Obstetrics/Gynecology: Other Diseases/Disorders of the Male/Female Reproductive System, Clinical
Therapeutics, Polycystic Ovarian Syndrome
Obstetrics/Gynecology: Other Diseases/Disorders of the Male/Female Reproductive System, Scientific
Concepts, Urinary incontinence
Obstetrics/Gynecology: Ovary, History and Physical, Ovarian cancer
Obstetrics/Gynecology: Uncomplicated Pregnancy, History and Physical, Gestational physical exam

Program reports will include:
 Individual student scores by exam
 Overall class scores
 Breakdown of task/organ system by class
 Comparative data will be available following 300
administrations of the specific exam form



Program reports updated every time
REPORT BY CLASS is opened
Latest updated data will be “pulled” so
comparison data as well as your own CLASS
data will be changing based on new data
Individual student scores by exam will be
available after completion of test
Hypothetical University Performance Report by Class
2012 PACKRAT
This report presents information on students and scores for the 2012 PACKRAT examination. Specific
information about your program is confidential and is not released to any other program.
Number
Mean Standard Number of
ContentFeedbackSummaryList of
Score Deviation Candidates
Exams
All PROGRAMS:
All Examinations (Year One)
2
39.5
2
First Time Takers (Year One)
2
39.5
2
All Examinations (Year Two)
0
0
0
First Time Takers (Year Two)
0
0
0
All Examinations (Year One)
2
39.5
2
First Time Takers (Year One)
2
39.5
2
All Examinations (Year Two)
0
0
0
First Time Takers (Year Two)
0
0
0
CLASS Fall 2012:
PACKRAT Content Area Feedback
CLASS Fall 2012
Year
One
Year
Two
All
All PROGRAMS
Year
One
Year
Two
All
%
%
%
%
%
%
Correct Correct Correct Correct Correct Correct
SPECIALTIES
Cardiology
13.89
0
13.89
13.89
0
13.89
Dermatology
13.64
0
13.64
13.64
0
13.64
7.69
0
7.69
7.69
0
7.69
ENT
26.92
0
26.92
26.92
0
26.92
Gastrointestinal/Nutritional
18.18
0
18.18
18.18
0
18.18
Hematology
21.43
0
21.43
21.43
0
21.43
Infectious Diseases
14.29
0
14.29
14.29
0
14.29
Neurology
21.43
0
21.43
21.43
0
21.43
Obstetrics/Gynecology
19.44
0
19.44
19.44
0
19.44
Ophthalmology
21.43
0
21.43
21.43
0
21.43
Orthopedics/Rheumatology
18.18
0
18.18
18.18
0
18.18
Psychiatry/Behavioral
Medicine
17.86
0
17.86
17.86
0
17.86
Pulmonology
16.67
0
16.67
16.67
0
16.67
Urology/Renal
21.43
0
21.43
21.43
0
21.43
Endocrinology
CLASS Fall 2012
Year
One
Year
Two
All
All PROGRAMS
Year
One
Year
Two
All
%
%
%
%
%
%
Correct Correct Correct Correct Correct Correct
TASKS
Clinical Intervention
13.33
0
13.33 13.33
0
13.33
Clinical Therapeutics
17.07
0
17.07 17.07
0
17.07
Diagnosis
21.95
0
21.95 21.95
0
21.95
Diagnostic Studies
19.35
0
19.35 19.35
0
19.35
Health Maintenance
20.45
0
20.45 20.45
0
20.45
History and Physical
10.26
0
10.26 10.26
0
10.26
Scientific Concepts
23.81
0
23.81 23.81
0
23.81

Pricing based upon the total number of types
of end of rotation exams purchased; prices
listed indicate the price per exam per student.
 $50 – 1-3 exams
 $40 – 4-6 exams
 $30 – all seven exams

PACKRAT pricing is unchanged
 $40 for members and $45 for non-members




The End of Rotation exams will be 120
minutes in duration
Two 60 question sections with an optional
break to be set by the program
Once a section is completed the student can
not go back to any items in that section
Time and a half and double time are available
and will be set by the program
For ordering information:
[email protected]
301-617-7820
Lessons Learned: Allow 24-hour turnaround time
between ordering and exam administration

Allows for:
 Entry of program users
▪ Faculty/Administrator/Proctor/Purchasing agent
 Class entry
 Student entry
 Examination scheduling
 Examination administration
 Examination performance report access

Users assigned either the program administrator
or faculty roles can do the following:





Add, edit, or disable accounts of other users
Add, edit, or delete classes
Add, edit, delete, or disable student accounts
Add, edit, or cancel examinations
Assign and un-assign the proctor role for admin and
faculty users
 Order exams from PAEA
 View the program's current exam counts

Only those users assigned the faculty role
can:
 View all reports
 Invalidate, validate, or release a student's exam
score

Users assigned the purchasing role can:
 Order exams from PAEA
 View the program's current exam counts
Role
Permissions
Add,
Edit, &
Disable
Other
Users
Add,
Edit, &
Delete
Classes
Add,
Edit, &
Disable
Students
Add,
Edit, &
Cancel
Exams
Assign &
Unassign
Proctors
Program
Faculty
X
X
X
X
X
Program
Admin
X
X
X
X
X
Release or Order View all Proctor
Invalidate
Exams Reports an Exam
Student’s
Exam Score
X
X
Proctor
Purchasing
X
X
Program Faculty
Program
Faculty
Program
Admin
Not Allowed
Proctor
Allowed
Program Admin
Proctor
Not Allowed
Allowed
Allowed
Allowed


Upon account initiation one program member
will be assigned a username and password
(program director)
This initial user can then create additional
accounts within the Program Exam
Management System
 Each new user must be assigned a role
 New users will receive an email with user name and
temporary password
 New users will be prompted to create a new password
at time of first login
• New class will need to be created in the Program Exam
Management System
• Once the class has been created students can then be
entered

Need to enter the following:
 Class name
 Year
 Duration (months)
 Graduation Date*
*The graduation date can be overridden if needed
for a particular student (ie – leave of absence)
• The Students tab in the Program Exam Management
system allows programs to:
• Add students
• Edit student information
• Delete or deactivate students
• Schedule examinations for a student
• Reset student passwords
Students are manually entered to avoid error from
group upload
 Student information required includes:
 Name (first and last)
 ID (optional)
 Class (graduation date will default to the one
entered for that class, however the graduation
date can be overridden)
 Email address

• Proctors are required for all EOR exams; optional for
PACKRAT exams
• While Proctors can be program admin and/or faculty they
must be assigned the Proctor role in order to proctor the
exam
• The Proctor Schedules screen displays a list of the
program’s proctors and allows the Program user to see
all Exams scheduled for a particular proctor
• The Examination Scheduling Tab is used to:
• Add exams
• Edit or cancel exam schedules
• Release results (faculty users only)
• Add/assign proctors
• Add students
• Resend confirmation emails regarding exams
• System compatibility test
Feedback from Liaison International, Inc.
indicated that the most common technical
challenge with administration of PACKRAT was
due to an IT mismatch between systems
 Email reminders will be sent to programs and
students to check system compatibility prior to
the exam.
 Compatibility test must be performed on the
computer where the exam will be administered
(ie – computer lab or student laptop)



Pop-up Blockers are usually turned on by
default via Internet browsers.
When students and/or programs receive
email confirmation that an examination has
been scheduled, pop-up blockers need to be
disabled prior to exam administration.

Pausing the exam (e.g. – bathroom break)
 Pause button needs to be hit again (much like
remote for TV/DVR).

Password protection
 If a student doesn’t enter their password (which
they set up) correctly 3 times, the system will
temporarily disable their account for 10minutes.
 Students can be reauthorized to take the exam,
but you can’t shorten the window.






Lab reference sheet integrated into exam
administration portal
Flexible testing time of up to 6 hours
Score reports featuring keyword feedback to allow
for increased exam security and to encourage
students to take ownership of the learning process
Improved program access to scoring reports
Administered only as an online examination
to further increase exam security
Direct communication with students to
provide access to information
Launch date – Nov 15

Utilization of images, electrocardiograms,
and radiographs to augment questions

Kim Cavanagh, DHsc, PA-C, Chair, AI
 [email protected]

Michel Statler, MLA, PA-C, Sr Dir, Academic
Affairs
 [email protected]

For ordering information
 [email protected]
 Phone: 301-617-7820