Transcript Benchmarks of Quality Training

Benchmarks of
Quality Training
Kentucky Center for
Instructional Discipline
www.kycid.org
Training Objectives
• Introduction to the Benchmarks of
Quality (BoQ)
• Understand how to complete, submit,
and use the BoQ
Why is Evaluation of PBIS
Implementation Important?
• To examine the fidelity of
implementation
− “Are we really doing what we think we are
doing?”
• To document effectiveness of
implementation
− “Is what we’re doing working?”
• To identify and examine strengths
and weaknesses of implementation
− Celebrate successes
− Identify areas to improve
What We’ve Been Doing
• Team Implementation Checklist
(quarterly)
• Effective Behavior Support survey
(annually)
• School-wide Evaluation Tool (annually)
What We’re Now Doing
• Team Implementation Checklist (three times
a year)
• Effective Behavior Support survey (annually)
• School-wide Evaluation Tool (annually)
• Benchmarks of Quality (annually)
Why the BoQ and not the SET?
• The BoQ provides deeper analysis
• Self-assessment heightens awareness
• Takes less time
• SET scores may sometimes be inflated
• SET may not reflect current activities
Benchmarks of Quality
• Developed by Florida PBIS at the University of
South Florida
• Assesses development and implementation of
school-wide PBIS
• Lists 53 benchmarks of quality in school-wide
PBIS programs within 10 critical elements
• Completed by school teams annually to
identify areas of strength and weakness
• Used by state and districts to guide TA and
training and evaluate outcomes related to level
of implementation
Benchmarks of Quality
• 53 items aligned with PBIS training
process addressing areas of:
− Faculty commitment
− Effective procedures for dealing with
discipline
− Data entry and analysis plan
− Expectations and lesson plans developed
− School-wide recognition system established
− Implementation plan
− Crisis plan
− Evaluation
BoQ Validation Process
• Expert review
• Pilot studies in Florida and Maryland
• Reliability: Test-retest, inter-rater both >.01
• Concurrent Validity – SET/ODR’s
• For more details see JPBI, Fall 2007
Completing the Benchmarks
Timeline
• January – training for coaches on the
BoQ
• February – complete the BoQ
assessment
• By March 30 – submit results to your
Area Coordinator
Benchmarks of Quality
Rating and Scoring Components
• Team Member Rating Form
– Completed by team members
independently
– Returned to PBS Coach
• Scoring Form, Tally Sheet, and Max
Scores per Element
– Completed by Coach using Scoring Guide
– Used for reporting back to team
• Scoring Guide
– Describes administration process
– Rubric for scoring each item
Method of Completion
• Coach
− Uses Scoring Guide to ascertain the
appropriate score for each item
− Collects Team Member Rating forms
− Resolves any discrepancies by
reaching consensus with the team
− Reports final findings to the team
Completion of BoQ
Step 1: Coach’s Scoring
• The Coach uses his or her best
judgment based on personal experience
with the school and the descriptions
and exemplars in the Benchmarks of
Quality Scoring Guide to score each
of the 53 items on the Benchmarks of
Quality Scoring Form.
• Do not leave any items blank.
Step 2: Team Member Rating
• The Coach will give the Benchmarks of
Quality Team Member Rating Form to
each PBIS Leadership Team member to be
completed independently and returned to
the Coach upon completion.
• Team members should be instructed to rate
each of the 53 items according to whether
the component is In Place, Needs
Improvement, or Not in Place. Team
members place a check in the box to
indicate their rating.
Step 2: Team Member Rating
Note: Some of the items relate to product
and process development, others to
action items; in order to be rated as In
Place, the item must be developed and
implemented (where applicable).
Step 3: Scoring the BoQ
• Coaches collect the Team Member Rating Forms
and tally the team’s most frequent response
using the Tally Sheet.
• Record the most frequent response for each
item on the Benchmarks of Quality Scoring
Form in the Step 2 Column, using ++ for In
Place, + for Needs Improvement, and – for Not
In Place.
• In the Step 3 Column, place a check in the box
where any discrepancies in scoring occur
between the team and coach ratings
Step 3: Scoring Guidelines
• For items with a range of 0 to 1:
– 1 = In Place
– 0 = Needs Improvement or Not in Place
• For items with a range of 0 to 2:
– 2 = In Place
– 1 = Needs Improvement
– 0 = Not in Place
• For items with a range of 0 to 3:
– 3 = In Place
– 2 or 1 = Needs Improvement
– 0 = Not in Place
Step 4: Team Report
• The Coach completes the Team
Summary on p. 3 of the Benchmarks of
Quality Scoring Form, recording areas
of discrepancy, strength, and weakness.
Step 4: Team Report
• Discrepancies - If there were any items for
which the team’s most frequent rating varied
from the Coach’s rating based upon the
Scoring Guide, the descriptions and
exemplars from the guide should be shared
with the team.
• If upon sharing areas of discrepancy, the
Coach realizes that there is new information
that according to the Scoring Guide would
result in a different score, the item and the
adjusted final score should be recorded on
the Scoring Form.
Step 5: Reporting Back to Team
• The coach reports back to the team
using the Team Summary page of the
Benchmarks of Quality: Scoring Form.
• If needed, address items of discrepancy
and adjust the score.
Step 5: Reporting Back to Team
• The coach then leads the team through
a discussion of the identified areas of
strength (high ratings) and weakness
(low ratings).
• The team identifies celebrations to
share with staff.
• The team identifies area of focus and
documents on the PBIS Tier 1/Universal
Action Plan.
Step 6: Submitting Your BoQ
• The Coach submits the Benchmarks of
Quality: Scoring Form (both pages) and
the Team Summary to his or her KYCID
Area Coordinator.
• BoQ results should be submitted by
April 30.
Final Notes
• District level PBIS team members or
KYCID staff may validate BoQ findings
using the PBIS Walkthrough or the SET.
• In addition, your team can use the PBIS
Walkthrough anytime throughout the
year for a quick self-assessment
Questions or Assistance?
• For additional information and support on
completing and/or using the data from the
BoQ, contact your Area Coordinator:
-Judy Boggs, Central
-Kelly Davis, West
-Steve Hutton, North
-Lea Brown, West
-Janeah Gullett, East
For more information about the
BoQ: