Accurate-Assessments
Download
Report
Transcript Accurate-Assessments
Overview of ODRC’s ORAS Quality Assurance Program
What is Quality Assurance?
ORAS File Review
Development of the QA Program
Results of the QA Program
Recommendations
A planned and systematic pattern of actions
necessary to provide adequate confidence
that the product optimally fulfills customer
expectations.
Proactive
Improve
Confidence
Credibility
Create a culture of quality
Good fidelity = replication of results
Mission “reduce recidivism”
FAQ
Flow charts
Internal audit standards
Quality Assurance Reports
Case file reviews
Quality Assurance Program
Keep these 12 things clean for good
assessments
Duplicates
Sources used
Verifying collateral information
Notes
Pilot site evaluations
Benchmarking
Literature Review
University of Cincinnati
Goal = accurate assessments
What contributes to the goal?
◦ Fidelity to the assessment process
◦ Good interview skills
How will the QA program be delivered?
◦ Create measures/tools
◦ Timelines
Implement the QA program
Analyze and share results
Evaluate program and make adjustments.
QA Coordinator
Random selection of staff
Scheduling
Communication
Work flow
Selection of Evaluators
ORAS Direct Observation Tool
Performance Measurement Guide
Pre-brief meeting
Direct Observation of the interview
Individual Scoring of the tool
Debrief and Coaching
Results to the QA Coordinator
Goals
Program process
Results
End users
All levels of management
Subject matter experts/Certified ORAS
Trainers
IPC skills
Recommendation from their immediate
supervisor.
Completion of evaluator training.
Direct Observation Tool
Performance Measurement Guide
Double coding
General process
Common rating errors
Providing good feedback that is balanced
Giving and receiving criticism
Survey Evaluation
◦ Evaluators (19)
◦ Assessors (116)
◦ Do survey results demonstrate gaps? Perceived
value verses real value.
Share results
Value of the pre-brief meeting
◦ 65% of the assessors and 59% of the evaluators rated it
successful or very successful.
Concentrated areas of coaching
◦ Assessors and evaluators rated using the interview guide
and using collateral information highest.
Did assessors learn anything new?
◦ 61% of the assessors and 90% of the evaluators
responded yes.
Ability of the evaluators to add value
◦ 82% of the assessors rated the evaluators with the ability
or strong ability to add value.
How important are accurate ORAS assessments?
◦ 65% of the assessors rated it as important or very
important.
What are the barriers to completing accurate
ORAS assessments?
◦ The length of time it tales to complete the interview
and input the information into the portal. Also the
program runs slow.
◦ Clients not being fully truthful.
◦ Not having an investigation in the file.
◦ Time consuming when you have a lot of other
things to do.
What was most valuable about the ORAS QA
process?
◦ Getting reassurance that I was doing some things
right.
◦ The tool has potential if done accurately.
◦ Helped understand my struggles were being felt by
others also being evaluated.
◦ One on one discussion and feedback.
◦ Don’t think we need this, to concerned about QA
and not the time consuming ORAS tool.
How would you improve the ORAS QA
process?
◦ It was fairly upfront and open. The evaluator did a
good job with me. Maybe assess everyone at some
point.
◦ I think the ORAS QA process can be very effective, if
administered by someone that has actually
completed an ORAS assessment of some kind.
◦ Have the assessor sit in on more than one
assessment.
◦ Figure out a way across the state to bring all
officers to the same page period. Good Luck!
Interview
Skills
Reliability
Direct Observation
Tool
Double Coding
Survey
Survey
Trends
Domains 6 and 7
Interview Skills
Resources
Subject Matter Experts
Have a QA Coordinator
Explain QA process to the offender
Scoring Guide
Have follow up plan ready to implement
Staff support – supervisors need to be more
knowledgeable.
Sustainability
Interview training
Booster trainings focused on weak areas
and/or changes
RIB rules card
Supervisor staffing requirements
Individual Improvement Plan
Consider assessor models
Continue QA efforts statewide with all user
groups
Goals
Concerns/barriers for staff
Resources
Process flow
Direct Observation
Training
Communication
Implementation
Evaluation
Follow up – use results to impact positive change
Proactive
Improve
Confidence
Credibility
Deborah Herubin
[email protected]
614-995-0181
John Geras
[email protected]