http://miblsi.cenmi.org

Evaluation in Michigan

’

s Model

Steve Goodman [email protected] National PBIS Leadership Forum October, 2011

Mission Statement

To develop support systems and sustained implementation of a data-driven, problem solving model in schools to help students become better readers with social skills necessary for success.

MiBLSi Project Evaluation Team Anna Harms Evaluation Coordinator Jennifer Rollenhagen Measurement and Evaluation Specialist Ed Huth Data Analyst Nicole Matthews Data Entry Evaluation contributes to Project by: •Developing and providing resources to enhance local capacity related to measurement and evaluation, consistent with the implementation research. Evaluation supports the competencies and capacity necessary for implementation specialists and local districts to engage in effective data-based decision making as part of an integrated behavior and reading RtI model.

•Reporting on program activities and project outcomes to evaluate and improve effectiveness and efficiencies of the project to ensure value added to consumers and stakeholders. This is accomplished by implementing Plan-Do-Study-Act cycles.

Several Purposes of MiBLSi Assessments

• • •

Audit

– for “ taking stock ” of current strengths/weaknesses and action planning

Formative evaluation

– for improving program while it is in the process of being implemented

Summative evaluation

– for improvement of future reiterations

Internal Evaluation (within the project)

MiBLSi Value-Added Work System Capital Resources People Materials Information Work Systems: Providing the RtI practices and the supports for the these practices to take place successfully within schools and districts Technical Assistance Professional Learning Evaluation Financial Feedback Stakeholders Investments: • Funding • Visibility /Funders • Political support Returns: • Addressing critical issues (Discipline/Ethnicity) • Addressing program directives (State Performance Plan) Valued RtI Products/ Services Consumers (schools, districts, ISDs)

Organizational Level Process Level Finance Evaluation Tech. Assistance Prof. Learning Worker Worker Worker Worker Stake Holders Consumers Evaluation At Organizational, Process and Performer (worker) level: At each , measurement takes place at determined interval. This information is compared to established standards and provided as feedback for the system.

Level

Organizational Process Performer (worker)

Levels of Internal Evaluation

Description

Evaluating impact of the RtI project • Stakeholders • Addressing critical issues (e.g., Discipline/Ethnicity) • Addressing program directives (e.g., State Performance Plan) • Consumers • Valued results (e.g., Improved quality program, fidelity of implementation, successful student outcomes)

Tools

• Annual reports • Measures of program quality • Measures of fidelity of implementation • Measures of student outcomes • Survey of consumer satisfaction Evaluating operational productivity and efficiency of each unit within organization Evaluating worker productivity and efficiency Project management tools • Fasttrack, MS Project • Basecamp • Job models

Job Model: Measurement and Evaluation Specialist

File Maker Pro Data Base

External Evaluation (outside the project)

Collecting information to evaluate implementation effects and using this information for continuous improvement MiBLSi Project ISD Leadership Team •Fidelity of implementation (state) •Systems integrity (project) •Student success (project-wide) •Fidelity of implementation (across districts) •Systems integrity (district-ISD) •Student success LEA District Leadership Building Team Leadership Team •Fidelity of implementation (across schools) •Systems integrity (district-LEA) •Student success (district-wide) •Fidelity of implementation (across grades) •Systems integrity (school) •Student success (school-wide) Building Staff •Student success/Intervention effectiveness

Assessments • • • • • • • • • • •

Elementary Schools

Major Discipline Referrals PBIS Self-Assessment Survey PBIS Team Implementation Checklist Benchmarks of Quality (BOQ) Schoolwide Evaluation Tool (SET) Benchmarks for Advanced Tiers (BAT) Dynamic Indicators of Basic Early Literacy Skills (DIBELS) Planning and Evaluation Tool (PET) for Effective Schoolwide Reading Programs Effective Reading Support Team Implementation Checklist Special Education Data Collection Form Schoolwide Reading Analysis Support Page • • • • • • • • •

Middle/Junior High Schools

Major Discipline Referrals PBIS Self-Assessment Survey PBIS Team Implementation Checklist Benchmarks of Quality (BOQ) Schoolwide Evaluation Tool (SET) ORF/MAZE through AIMSWeb School-Wide Evaluation and Planning Tool for Middle School Literacy (SWEPT) Middle School Reading Team Implementation Checklist Special Education Data Collection Form

Building Level

Assist Teams in Using Data for Decision-making

• First Year – Winter systems review – Spring Data Review • Second Year – Fall data review – Winter data review – Spring data review • Third Year – Fall data review – Winter data review – Spring data review

Assessment Booklet • Description of assessments • Data collection schedule • Data summary • Data forms and assessment forms

Team Evaluation of Outcome, Process and Systems Data

Assessment Schedule

(for Cohort 7 from MiBLSi website)

Video examples for completing and submitting PBIS assessments

Improving the accuracy and Consistency of Recording Office Discipline Referrals

Developing Fluency with Discipline Referral Categories A.

B.

C.

D.

Example Situation Issac picked up his lunch and notice that another student was sitting in his usual spot at the table. Issac squeezes into his spot forcing the student to abruptly slide off the seat and onto the floor. Ian comes to school wearing a T -shirt with a sexually suggestive picture.

During the test, Timothy is looking at Jennifer’s paper and copying her answers. Taneshia walks to school each morning. She plays in the yard across from the school until the bell rings. After she sees al l the student run into the building, she walks towards the school, late again.

Example Exercise 2: Match the example situation below to the correct problem E.

H.

behavior on the discipline categories answer sheet. Write Lying/Cheating Disruption I.

J.

the letter in the column for Exercise 2.

Excessive Tardiness Truancy K.

Property Damage During lunch, Franz collects the empty milk cartons from his classmate. He sets them on the floor in a line and then stamps on the cartons making a loud noise with each stomp . F.

Problem Behavior

Inappropriate language G.

Fighting physical injury Alexandra has been following Ben around the playground. She sometimes makes various noises (that seems to bother him) when she is near. Sometime she tells him how she and her brother will stop him on the way home from school and beat him up.

Susan and Jill are best friends. Jill lives several houses down from the school. During lunch Jill talks Susan into coming over to see her new puppy. The school has a rule about staying on school property at lunch.

says that math is stupid while not taking out his book. His teacher then tells him he needs to get his book from his desk. Billy says he hates math, he hates school and he is not going to do it.

walk into the boy restroom to find Matt and Chris wadding up paper towels and stuffing them into the toilet.

teacher, Ms. Swanson, sent a note home with Rebecca to be signed by her parents and returned the next day. However, when the note was returned the signature was Rebecca’s mother’s name but written in Rebecca’s handwriting. The first grade class writes in a journal. The topic focuses on a particular letter each day. During th e letter “F” day, Sarah writes a word she has all of the students around her.

Other

District Level

Focus on Implementing with Fidelity

using Benchmarks of Quality (BoQ)/ODR ’06-’07 and ’07-’08 Average Change in Major Discipline Referrals: One District Example (13 elementary schools) 10% 5% 0% -5% -10% -15% -20% Increase 8% Decrease 14.6% Not Met Criteria (< 70) n=5, range: 41-65 Met Criteria (> 70) n=8, range: 72-94

District Implementation Tracking Form

Leadership-Implementation Support Team Self Assessment

Lesson Learned

• Teams need to be taught how to analyze and use data • Emphasis on directing resources to need and removing competing activities • As we grow, it is even more important to systematic gather data that is accurate and then act on the data for continuous improvement • More work is needed in developing feedback cycles

“Even if you’re on the right track, you ’ll get run over if you just sit there ” - Will Rogers