Transcript Primary purpose of Pilot Study

An Introduction to Understanding the
OTA Fieldwork Performance Evaluation
(FWPE)
Karen Atler, MS, OTR
Colorado State University, Fort Collins, CO
Roberta Wimmer, OTR/L
Pacific University, Forest Grove, OR
 2003 The American Occupational Therapy Association, Inc.
www.AOTA.org
Task Force Members

Carole Dennis, PhD, OTR


Carole Hays, MA, OTR


Pueblo Community College, Colorado
Karen Atler, MS, OTR, Co-Chairperson


Springfield Hospital Center, Maryland
Becky Robler, MEd, OTR


Ithaca College, New York
Colorado State University
Roberta Wimmer, OTR, Co-Chairperson

Pacific University, Oregon
2
Objectives for Today:

Describe the…




Entry-level practice competencies for OT
and OTA students.
Purpose, format, content, and scoring of
the companion evaluation forms.
New concepts and terminology used in the
FWPE from the OT Practice Framework.
Begin to score items on the FWPE.
3
Task Force’s Charge


Revise/develop evaluation tools to measure
assistant and professional Level II fieldwork
student performance.
Expectations:




Conduct review of literature across disciplines.
Synthesize feedback on current AOTA FWE/OT
forms.
Incorporate 1997 NBCOT Practice Analysis results.
Address identified desired characteristics.
4
Desired Characteristics

Companion documents for assistant and
professional level that…






Measure entry-level competence.
Focus on occupation-based practice.
Reflect current and future practice.
Can be used in a variety of settings.
Provide feedback to students.
Can be easily used in a timely manner.
5
NBCOT Practice Analysis 1997
What OTs & OTAs DO







Determining needs/priorities for interventions.
Identifying/designing interventions.
Implementing interventions.
Reporting/evaluating intervention effectiveness.
Providing OT services for populations.
Managing delivery of OT services.
Advancing effectiveness of the OT profession.
6
NBCOT Practice Analysis 1997
What OTs & OTAs NEED TO KNOW







Human development and performance.
Principles/strategies in the identification/evaluation of
strengths and needs.
Principles/strategies in intervention/treatment
planning.
Principles/strategies in intervention.
Nature of occupation and occupational performance.
Service management.
Responsibilities as a professional.
7
Standards of Practice
for OT


Identifies minimum standards.
Identifies key performance areas for the OT
and OTA:








Professional standing and responsibility
Referral
Screening
Evaluation
Intervention plan
Intervention
Transition services
Discontinuation
8
ACOTE:
Minimum Standards and Outcomes for OTA
Be a generalist.
 Achieve entry-level competence.
 Work under the supervision of and in
cooperation with the OT.
 Articulate, apply, and justify interventions
related to occupation.
 Keep current with best practice.
 Uphold the ethics, values, and
attitudes of the profession.

Goal of Level II Fieldwork
Education for the OTA Student



Develop competent, entry-level
generalists.
Include an in-depth experience in
delivering OT services.
Be designed to promote reasoning,
enable ethical practice, and develop
professionalism.
10
The Process





Began with OTA evaluation.
Reviewed by experienced panel.
Submitted to COE.
Made revisions.
Completed pilot studies (2 OTA, 1 OT).
11
Design and Analysis of Pilot Studies:
The Rasch Measurement Model
less able
Student Ability
more able
Results of Pilot Studies




Good representation in pilot samples.
Students and educators preferred new form.
Good scale and response validity.
Inaccurate use of scale.
Rating Scale Usage in Pilot Studies II and III
Rating Scale Descriptors
% Usage OT
% Usage OTA
1 = Unsatisfactory
0
0
2 = Needs Improvement
10
13
3 = Meets Standards
56
53
4 = Exceeds Standards
34
29
13
FWPEs for OT and OTA Students







Companion documents
Terminology
Content layout
Purpose
Design
Rating scale
Scoring system
14
FWPEs OT and OTA
Companion Documents

Focus




The OT process
The clinical reasoning process
Roles and responsibilities of the OT and OTA
Structure



Collaborative process—student and FW
educator
Same layout
Same rating/scoring system
15
Terminology of the FWPEs

Reflects…



Standards of Practice for OT and ACOTE
Education Standards
OT Practice Framework
The glossary
16
Content Layout of FWPEs





Summary Sheet
Overview/instructions
Organization of items
Space for comments—midterm and final
Performance Rating Summary Sheet
17
Content of OT and OTA Evaluations
OTA
OT
Fundamentals of practice (3)
Fundamentals of practice (3)
Basic tenets (3)
Basic tenets (4)
Evaluation/screening (5)
Evaluation/screening (10)
Intervention (6)
Intervention (9)
Service Management (5)
Communication (2)
Communication (4)
Professional behavior (6)
Professional behavior (7)
18
Primary Purposes of the FWPEs

Measures entry-level competence:


Designed to differentiate the competent
student from the incompetent student.
NOT designed to differentiate levels above
entry-level competence.
19
Purpose (continued)

Provides student with accurate
assessment of his/her competence for
entry-level practice over time:




Growth occurs over time.
Midterm and final scores reflect this change.
Midterm scores: Satisfactory/unsatisfactory.
Final scores: Pass/no pass.
20
Purpose (continued)


Provides feedback to student.
Provides opportunity for student selfassessment.
21
Design of the FWPEs



The “doing” of the OT process is
evaluated, not the individual tasks in
isolation.
NOT all items are equal in level of
difficulty (i.e., simple to complex).
Evaluation is supplemented with
development of site-specific objectives.
22
Easier
210-1-2Harder
Modifies Intrven Plan;17
Interprets Assessment;9
Selects Intervention;13
Activity Analysis; 15
Evidence BasedPractic 6
Plans Intervention; 12
Reports; 10
OT Philosophy; 4
Administer Assessmnts;8
Establishes Goals; 11
OT/OTA Roles 5DataGather
Verbal Communication; 18
Self-responsibility; 20
I Implements intervntn; 14
Written Communication;19
Therapeutic Use Self;16
Work Behaviors; 22
Interpersonal skills;24
Responds to Feedback;21
Safety; 2 and 3
Cultural competence; 25
Ethics; 1
Rasch Ordering of Items OTA
23
Rating Scale of FWPEs
4 = Exceeds
Standards
Performance is highly skilled and self-initiated. This
rating is rarely given and would represent the top 5% of
all the students you have supervised.
3 = Meets
Standards
Performance is consistent with entry-level practice.
This rating is infrequently given at midterm and is a
strong rating at final.
2 = Needs
Improvement
Performance is progressing but still needs improvement
for entry-level practice. This is a realistic rating of
performance at midterm and some ratings of 2 may be
reasonable at the final.
1 = Unsatisfactory
Performance is below standards and requires
development for entry-level practice. This rating is given
when there is concern about performance.
24
Scoring System of FWPEs





Each item must be scored.
Ethics and safety items must be passed.
Each item rating recorded on
Performance Rating Summary Sheet.
All items summed up at midterm and
final.
Score compared to scales provided.
25
Midterm and Final Scores
OTA
Overall Midterm Score
 Satisfactory: 54 & above
 Unsatisfactory: 53 & below
Overall Final Score
 Pass: 70 & above
 No Pass: 69 & below
26
Rating Performance Using the FWPEs
Case Scenarios
 Sandra—OTA Evaluation/Screening
 David—OTA Intervention
27
Individualizing the FWPEs

Designed for additional objectives to
be written to add clarification:



Site-specific objectives
NOT supervisor-specific
If an item is very clear and meets the
RUMBA test, then there is no need to
write another objective.
28
Objectives: An Example
From FWPE for the OT Student

16. Establishes accurate and
appropriate plan


School—Develops behavioral-based,
measurable OT goals during IEP process.
Acute Care—Overall intervention plan is
achievable within client’s length of stay.
29
Summary




Evaluations designed to measure entrylevel competence, NOT level of
performance above competency.
OT practice examined as a generalist.
Evaluations reflect the OT process.
Performance develops over time.
30