Transcript Slide 1
SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
Charles Smith, Ph.D.
Executive Director, David P. Weikart Center for Youth Program Quality Vice President of Research, Forum for Youth Investment
May 9, 2013; 9:00-9:30 am SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
Agenda
• Quality Improvement Systems 1. Building QIS 2. Site Level Process 3. System Accountabilities • Local Models of Quality Improvement System Accountability • Why Build Systems for Developmental Settings?
• APPENDIX SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
Building QIS
SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
2012-2013 Dissemination
l 85 Networks/ Systems >3250 Sites >21,125 Staff Estimate based on mean of 6.5
staff per site in YPQI Study Sample >276,250 Child & Youth Estimate based on mean daily attendance of 85 youth per day in YPQI Study Sample SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
Building a QIS: Stages and tasks
SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
Site Level Continuous Improvement Process
SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
Instructional Practices
“Quality” at the Point of Service Level Setting SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
Four Continuous Improvement Practices
Organization Level Setting (Plus 10 hours of TA/coaching for site managers to implement the four CI practices) SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
Targeted Staff Trainings for Instructional Skills
CI Practice #4 SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
System Accountabilities
SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
System Supports for CI Practices
Policy:
Eligibility, Targeting, Low/high stakes
Training, TA & Coaching Evaluation
External Raters, Program Evaluation
SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
System Accountabilities: Higher Stakes
Higher Stakes Accountabilities
Objective Data Publicity Action Improved Outcomes
SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
System Accountabilities: Lower Stakes
Higher Stakes Accountabilities Objective Data Meaningful Information Action/ Expertise Improved Outcomes Interpretive Community •Team Self Assessment •Review external scores Team Planning and Implementing •Improvement planning •Performance coaching Lower Stakes Accountabilities SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
Higher Stakes: System Needs and Challenges
• System Needs monitoring • Challenges 2 1 – Standards beyond licensing regulations – Accountability policies based on assessment and – Program and practitioner outreach and support – Financing incentives specifically linked to compliance with quality standards – Differences in structure and design (e.g. measures) – Lack of coordination across agencies and data systems – Policies lack clarity about goals, timing and expectations for improvement 1. National Child Care Information and Technical Assistance Center. (2009). Quality Rating Systems: Definition and Statewide Systems. Fairfax, VA: National Child Care Information and Technical Assistance Center.
http://www.dataqualitycampaign.org/files/4_NCCIC_QRIS.pdf
2. Tout, K., Zaslow, M., Halle, T., and Forry, N. (2009). Issues for the next decade of quality rating and improvement systems. Washington, DC: Office of Planning, Research and Education.
SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
Local Models of Quality Improvement System Accountability
SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
Requirements
Policy Stakes Accountabilities
Participation Mandatory Financial Incentives
Lower Indianap Middle Seattle / WA MI / OK 21 CCLC Kansas City UW
x x x x Published Scores
CI Practices
Self Assessment External Assessment Planning with Data Perf Feedback & Coaching Training: Instruction Methods
Indianap
x x x x
Seattle / WA MI / OK 21 CCLC Kansas City UW
x x x x x x x x x x x x x x x
VT
x
VT
x x x x
Higher Oakland AR QRIS
x x x
Oakland
x x
AR QRIS
optional x x x x x x x x x SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
Incentives/Punishments
Policy Stakes Incentive Structures Voluntary
,
emphasis on supports Lower Indiana p Seattle / WA MI / OK 21 CCLC Middle Kansa s City UW VT Higher Oaklan d AR QRIS x x x x Required, emphasis on supports x x x x Required, participation monitored Management clarity
(“know what to do”)
Information useful Financial incentives Public ratings x x x x x x x x x x x x x x x x x x x x x
SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
x x x x x
Sources of Data for Public Ratings
• Oakland – Participation records – Program Quality Assessment (PQA) – Stakeholder surveys – Academic records – http://publicprofit.net/Services/Evaluation/ • Arkansas QRIS – Program or Business Administration Scale (PAS or BAS) – Traveling Arkansas Professional Development Registry (TAPP) – Program Quality Assessment (PQA) – School-Age Care Environmental Rating Scale (SACERS) – Various other criteria – http://www.arbetterbeginnings.com/wp content/uploads/2011/12/sagridswithdef.pdf
SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
YPQI Study “Best” Practices
Requirements
Accountabilities
Incentives/Punishments
Participation Mandatory
x
Voluntary
,
emphasis on supports Financial Incentives Required, emphasis on supports
x
Published Scores CI Practices Self Assessment External Assessment Planning with Data Perf Feedback & Coaching Training: Instruction Methods
x x x x x
Required, participation monitored Management clarity
(“know what to do”)
Information useful Financial incentives Public ratings
x x x SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
Prime Time Palm Beach County
SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
Evolution
• Piloted QIS elements with four providers in 2010 • Confirmed what we suspected: expertise in content; opportunity to strengthen youth development • Included all enrichment programs in the modified QIS • Changed name of enrichment activities to “expanded learning opportunities” in 2012 to reflect new expectations • Moving to greater alignment with the school day through the Common Core framework • Moving to more concrete learning measures and youth outcomes SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
Background • Founded in 2000 • Primary Areas of Service:
• Quality Improvement • Professional Development • Community Engagement and Supports
• Supported enrichment activities for nine years • Belief that a variety of experiences is essential for positive youth development
SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
QIS annual cycle
SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
QIS Level System
Entry • 12 months • Introduction to system • QA supported self assessment Intermediate • 12-24 months • QA preparation of director for maintenance role and requirements Maintenance • Ongoing • Based on benchmark scores and director accomplishments • Recognizes high quality programs and directors • Provides flexible time expectations based on needs SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
Why Build Systems for Developmental Settings?
A Frame for Developmental Systems
SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
Frames
• Positive Youth Development – Substitution – Skill building • Systems as… – Protective factors (Fragmentation = risk) – School reform (“Expanded Learning”) • Reinventing Government / Social Sector – Regulating core processes (instead of inputs) – Building performance cultures SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
Charles Smith, Ph.D.
Executive Director, David P. Weikart Center for Youth Program Quality Vice President of Research, Forum for Youth Investment
http://cypq.org/ypqi SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
APPENDIX
SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
High Stakes Examples
SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
Higher Stakes Models
Policy Stakes Accountabilities
Participation Mandatory Financial Incentives Published Scores
CI Practices
Self Assessment External Assessment Planning with Data Perf Feedback & Coaching Training: Instruction Methods
Higher Oakland AR QRIS
x x x x x
Oakland AR QRIS
optional x x x x x x x x x • Oakland requires participation, scores are tied to funding, and reports go to the city government • AR’s system is voluntary, but once in it, scores feed into 3 tier system that are used for incentive grants and published ratings for families SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
Oakland Unified School District (OUSD) and Oakland Fund for Child and Youth (OCFY)
• http://publicprofit.net/Services/Evaluation/ • OUSD and OCFY are funders • Public Profit is the intermediary and evaluator • System is voluntary for self assessment, required for external assessment and planning • Process: – All sites get external assessments of 2 program offerings – All sites receive individualized “planning with data” type meetings with the evaluators, who go over their scores and work with them to create improvement plans – Reports of external assessments are also sent to city government – All sites have access to Methods trainings – Programs with low scores receive additional coaching • Incentives: – Programs with extremely low scores (scale scores under 2) that don’t improve over the course of 2-3 years could lose their funding – No sites have lost funding almost no programs score that low SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
Arkansas Quality Rating and Improvement
• • • • •
System (QRIS)
http://www.arbetterbeginnings.com/ Arkansas State University serves as intermediary for system for school-age care programs System is voluntary Process: – A three-tiered rating system that the PQA scores feed into, but it is only one of multiple measures. – Components included in the ratings are: • Administration • Administrator/Staff Qualifications & Professional Development • Learning Environment • Environmental Assessment • Child Health & Development Incentives: – Ratings are used to offer both incentive grants and published ratings for families to use to decide where to send their children – Incentive Grants are available upon meeting certification standards at each of the 3 levels • At level 1 and level 2, it is renewable for a maximum of 9 years (not to exceed 6 years at either level 1 or level 2). • At level 3, the Incentive Grant is available annually, as long as the facility continues to meet the standards.
• Incentive grant amounts are based on a combination of licensed capacities, current Level and the number of years spent at that level.
SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
Middle Stakes Examples
SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
Middle Stakes Models
Policy Stakes Accountabilities
Participation Mandatory Financial Incentives Published Scores
CI Practices
Self Assessment External Assessment Planning with Data Perf Feedback & Coaching Training: Instruction Methods
Middle MI / OK 21 CCLC
x
Kansas City UW
x x
MI / OK 21 CCLC
x
Kansas City UW
x x x x x x x x x
VT
x
VT
x x x x • MI/OK require process to maintain funding, but focus on supports and coaching • Kansas City has 3 tier incentive system based on completion of YPQI elements to get funding at the different levels • VT has 5 tier recognition program based on various practices where programs get funding, public awareness, discounts, and funding opportunities SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
Kansas City United Way
• http://www.unitedwaygkc.org/nonprofits/qualitymatters.html
• UW is the intermediary, partners with Francis Institute for coaching and University of Missouri, Kansas City for evaluation/external assessments • System is voluntary • Half the sites that participate are United Way funded programs, but United Way funds the YPQI process for all sites • Process: – Have high fidelity to YPQI assess-plan-improve, with all sites doing assessments, planning and receiving coaching • Incentives: – Have a 3 tier incentive system that is based on completion of the elements of the YPQI, and they receive $300, $500 and $750 accordingly – Example: Participation Level 3: $750 – Completion of: • Conduct a fall team based PQA self-assessment which includes observation and team consensus and enter data into the
Online Scores Reporter
by [DATE].
• Program Improvement Plan created and entered into
Online Scores Reporter
by [DATE].
• 75% completion rate of goals set in Program Improvement Plan**by [DATE] • Attend 7 different workforce training (1-4 site staff may attend each training and must stay for entire session) by [DATE] SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
Michigan 21
st
CCLC and Oklahoma 21
st
CCLC
• http://www.michigan.gov/mde/0,4615,7-140-6530_6809 39974--,00.html
• http://www.ok.gov/sde/21cclc • Similarly structured systems, they are both funder and intermediary • Participation is required since it is tied to funding • Process: – Data from both the PQA and evaluation are incorporated into the QIS process – Coaches are key component, offering comprehensive services to select programs • Incentives: – Focus is on implementation and improvement supports – Require that all sites complete full process in order to maintain good standing on grant – Scores are not used punitively, but can be used to target coaching services SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
Vermont Center for Afterschool Excellence
• • • • • • http://www.vermontafterschool.org/ http://dcf.vermont.gov/cdd/stars Are intermediary, they serve both 21 st CCLC and AHS/QRIS programs Both systems encourage use of the YPQI, but is voluntary Process: – Have a 5 tier star system for recognition of programs, based on practices in these areas: • Compliance with state regulations • Staff qualifications and training; • Interaction with and overall support of children, families, and communities; • How thoroughly providers assess what they do and plan for improvements; and • The strength of the program’s operating policies and business practices.
Incentives: – The benefits for the star system, tiered based on number of stars the program has earned: • The Child Care Financial Assistance Program (CCFAP) pays a higher rate on behalf of families. • Bonus payments for EACH level achieved: • Public awareness of STARS participation if requested. Options include: listing on the STARS website, supply of STARS brochures, and a customized press release.
• The opportunity to apply for grants open only to programs that are in STARS or are nationally accredited • Discount on purchases from a list of corporate sponsors. SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
Lower Stakes Example
Indianapolis Marion County Commission on Youth (MCCOY)
SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
Lower Stakes Models
Policy Stakes Accountabilities
Participation Mandatory Financial Incentives Published Scores
CI Practices
Self Assessment External Assessment Planning with Data Perf Feedback & Coaching Training: Instruction Methods
Lower Indianap Seattle / WA Indianap
x x x x x
Seattle / WA
x x x x x • Main incentive is access all supports, learning community, flexibility and choice • MCCOY is very low stakes, all parts are suggested but optional • Raikes also offers funding to programs to subsidize participation SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
Indianapolis Marion County Commission on Youth (MCCOY)
• http://www.mccoyouth.org/ • MCCOY is intermediary • Very, very low stakes--Participation is entirely voluntary • Process – Sites do the full Assess-Plan-Improve sequence – Sites can do self and external assessment, usually only once a year, but could do more – No coaching or training in instructional coaching – Methods workshops are offered for all sites – Recruitment can be hard, but they focus on partnerships and Methods to get sites engaged – Programs choose to join cohorts (2-3 cohorts per year) – Programs can choose to participate multiple times but there is no emphasis on tracking year to year improvement.
• Incentives – Programs have access to supports and learning community, and a lot of flexibility in choosing how to improve SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
Seattle & Washington State- Raikes Foundation and School’s Out Washington
• http://raikesfoundation.org/Secondary.aspx?file%3daboutmission • http://www.schoolsoutwashington.org/index.htm
• Raikes Foundation is the funder • Schools Out Washington is the intermediary, with other local supports • Has funded 1-3 cohorts across the state since 2008 • System is voluntary, programs apply • Programs span funding and accountability streams • Process – There is an intensive application process that comes with funding to programs to subsidize their participation and all of the supports are free – High level of fidelity to YPQI – Sites get less and less supports over a 3 year period…beyond that they can apply for funding to get a al carte services • Incentives: – Participation comes with funding to programs to subsidize their participation and all of the supports are free SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
YPQI Evidence
SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
CQI Systems: Cross-Level Roles
Settings Actors Behaviors
Network Leaders
…enact standards and supports …engages standards and supports
Managers
…enacts continuous improvement practices …engages in continuous improvement practices
Staff Youth
…enacts instructional practices Youth engage in instruction and build skills SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
Management CI Skills
YPQI Study Baseline
100 90 80 70 60 50 40 30 20 10 0 Assess instruction quality Create improv plan Coach staff on instruction Meet with a TA/coach
SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
Staff CI Skills
N=366, YPQI Study Baseline
1 0.8
0.6
0.4
0.2
• • •
State of the Field
10% of staff were engaged in all CI practices 22% were not engaged in CI practices 68% were engaged in some practices
0 Full Imp
Attend training
Mid Imp
Conduct program planning Note: Profile of 3 exemplary clusters from an 6 cluster solution
Non-imp
Conduct observation with tool SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
Staff Instructional Skills
3 N= 600 different youth workers and teachers Occurred For All
Q u a l i t y
5 4 Occurred For Some 3 Did Not Occur 2 1 PYD I Staff Cent I Positive Youth Development N=166, 28% Staff Centered N=231, 39% Low Qual II Low Quality N=193, 33% welcome inclusion active learning grouping choices planning reflect SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
CQI Systems: Cross-Level Roles
Settings Actors Behaviors
Network Leaders
…enact standards and supports …engages standards and supports
Managers
…enacts continuous improvement practices …engages in continuous improvement practices ES =1.87
ES =.98
ES =.52
Staff
ES =.55
Youth
…enacts instructional practices Youth engage in instruction and build skills SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
Participant Satisfaction
100 90 80 70 60 50 40 30 20 10 0 I gained relevant knowledge and/or developed valuable skills.
The quality of instruction improved at my site.
Youth were more engaged Program Managers Youth developed skills.
Overall, YPQI was a good use of my time and effort Direct Service Staff N=128 site managers, 178 staff from Atlanta, Baltimore, Chattanooga, Maryland, Nashville, Richmond, Vermont, Washington Overall, YPQI was supported by my supervisor SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?