Transcript MA DYS 2001 Legislative Report

State Agency Reform: The Experience of the
Massachusetts Department of Youth Services
Introductory Remarks
Tarek Anandan
Director, NDTAC
Evaluating N or D Programs
 The What
 Formative evaluation
 Summative evaluation
 The Why
 Legal requirement
 Program quality improvement
 Program expansion/contraction
 The How
 Execution - internal or external
 Funding – Federal, State, local, foundations
3
Evaluating N or D Programs
“Nationwide, there is a scarcity of literature specific
to effective or ‘best practice’ juvenile justice
education, or, for that matter, systemic reform of
similar systems”
Evaluation of the Department of Youth Services Education Initiative
UMass Donahue Institute, 2008
4
Evaluation and Title I, Part D
SEC. 1431. Program Evaluations
(a) SCOPE OF EVALUATION—Each State agency or local educational
agency that conducts a program under subpart 1 or 2 shall
evaluate the program, disaggregating data on participation by
gender, race, ethnicity, and age, not less than once every 3 years,
to determine the program's impact…
(d) EVALUATION RESULTS—Each State agency and local educational
agency shall—
(1) submit evaluation results to the State educational agency and
the Secretary; and
(2) use the results of evaluations under this section to plan and
improve subsequent programs for participating children and
youth
5
Evaluation and Title I, Part D
Funding
 Evaluation is an allowable use of Part D funds
 State agencies and local education agencies can use a portion
of their granted funds to support evaluation activities
 Funds used for evaluation purposes should be reasonable in
respect to the amount of funding that goes to the provision of
educational services
 State education agencies are allowed to reserve the greater of
1% or $400,000 for administrative purposes, including
evaluation activities of SEA administrators
6
Evaluation Resources
 U.S. Department of Education
Understanding Evaluation: The Way to Better Prevention Programs
http://www.ed.gov/PDFDocs/handbook.pdf
 Bureau of Justice Assistance
Center for Program Evaluation
http://www.ojp.usdoj.gov/BJA/evaluation/
 Justice Research and Statistics Association
Program Evaluation Briefing Series
http://www.jrsa.org/pubs/juv-justice/index.html
 “Approaches to Assessing Juvenile Justice Program Performance”
 “Hiring and Working With an Evaluator”
 “Strategies for Evaluating Small Juvenile Justice Programs”
7
Presenters
 Christine Kenney
Director of Educational Services
Massachusetts Department of Youth Services
 Lonnie Kaufman
Research Manager
UMass Donahue Institute
 Steven Ellis
Research Manager
UMass Donahue Institute
8
State Agency Reform:
The Experience of the Massachusetts
Department of Youth Services
Jane Tewksbury, Commissioner
Edward Dolan, Deputy Commissioner
Today’s Presenters
Christine Kenney, Director of Educational Services, DYS
Lonnie Kaufman, Research Manager, UMass Donahue Institute
Steve Ellis, Research Manager, UMass Donahue Institute
DYS Vision
The Department of Youth Services is a leader in the field of juvenile
justice that collaborates with youth, families, communities,
government, provider agencies, and others to prevent juvenile crime.
We provide the most innovative and effective treatment and skill
development services available to the youth committed to our care.
The Department exemplifies diversity in the management of all our
work and relationships. We effectively manage the resources
necessary to achieve the Department’s mission.
DYS Functions
DYS is the Commonwealth’s Juvenile Correctional Agency
Functions
– Detention
– Residential Treatment
– Parole/Community
Each Year:
– 18,000 to 20,000 arraignments
– 5,700 bail admissions with average length of stay of 17 days
– 1,200 commitments with average length of stay of 2.5 years
On any given day there are 2,000 committed youth in
custody.
– 49% in Residential Placements
– 51% in Community Supervision
Building Educational Capacity
Educational improvement efforts were spurred by the findings of
the MA DYS 2001 Legislative Report
Key Findings Related to Educational Services included concerns
regarding:
–
–
–
–
–
–
–
Teacher Recruitment and Retention
Transitions
Special Education
Learning Methods, Curriculum and Other Resources
Vocational Education and Job Training
Computers and Technology
Infrastructure
DYS Reform Strategies 2001-Present
DYS in close collaboration with the Commonwealth Corporation and
the Hampshire Education Collaborative, initiated a multiple phase
education reform effort for DYS educational programs
– Develop capacity to support education reform across the agency
– Align all education programming to MA Curriculum Frameworks
– Develop an effective community placement and transition process
This initiative reflected a collective commitment to meet state and
federal standards for educational programming by developing a better
coordinated, appropriately resourced educational system
In spring 2006, DYS initiated a system-wide evaluation to help
measure progress and to guide further education improvement
Evaluation of DYS Education Initiative
Why and How?
DYS wanted to revisit the original legislative report and assess change
in areas that had been targeted for improvement
Minimum education requirements had provided a preliminary, but
limited, view of the Education Initiative’s progress
A comprehensive evaluation would encompass a broader and more
systemic view of the Education Initiative
Talked to local universities about how to approach
DYS made a commitment to allocate funding for evaluation for FY07/08
and issued an RFP
RFP was based on a logic model which guided core education
strategies and activities
University of Massachusetts Donahue Institute was awarded the
evaluation contract
Evaluation: Phase One
Phase One: June 1 through September 30, 2006
Literature Review
Preliminary Fieldwork and Evaluation
–
–
–
–
Interviews with key leaders of DYS, CommCorp and HEC
Site visit interviews with teachers and coordinators at eight programs
Attendance and participation at various system events
Review and cataloging of key system documents
Development of a Comprehensive Evaluation Plan
Presentation and Discussion of Preliminary Evaluation
Findings
Evaluation Plan
Evaluation Guided by Three Core Research Questions
1.
2.
3.
To what extent has DYS successfully implemented strategies
designed to improve the educational and life outcomes for youth
they serve? What factors have influenced the extent or success of
implementation?
Have the Education Initiative's core strategies led to, or are they
leading to anticipated short or mid term impacts? What other
impacts or outcomes have resulted from the Education Initiative?
Based on key findings of the evaluation, what should DYS focus on
over the coming years in order to realize the long term
goals/outcomes of the Education Initiative?
Evaluation Plan
The Evaluation Also Identified:
– Emerging promising practices
– Causes and potential solutions for mixed finding and impacts
– Causes and potential solutions for gaps in evidence, including
additional data which need to be collected to properly measure certain
outcomes outside the realm of the evaluation
– Apparent gaps in the current Education Initiative
– Key lessons learned through the implementation of the Initiative
Evaluation: Phase Two
Phase Two: October 1, 2006 through August 31, 2007
Several major data collection and analysis activities:
–
A DYS teacher survey that gathered the perspectives of over 90% of
respondents
–
In-depth interviews of approximately 50 educators working at 12 DYS
Education programs
–
Collection and analysis of all available institutional data
–
Case studies of six “featured practices” resulting from the Initiative
–
A second, targeted review of literature
–
On-site observations of selected programs and events, including
statewide professional development
Top-Level Evaluation Findings
Findings show remarkable progress in the development and implementation of
key interventions, and positive initial impacts. Evidence shows important
systemic changes to:
–
–
–
Educational service delivery
Support of student transitions
Overall system management
Important new programs, policies, and infrastructure are now in place, which
have facilitated improvements in the stability and quality of DYS’s:
–
–
–
Teacher workforce
Instructional resources and methods used in DYS classrooms
Operative culture in which teachers and students function each day
Continued progress is likely, assuming DYS can sustain its vigorous
investment of time, resources, and attention
Findings underscore not only the need to continue and expand improvement
efforts in the coming years, but also the challenges associated with delivering
educational services to adjudicated youth
Sample of Approach to Presenting Findings
Topic: Educational Service Delivery
Key Finding: DYS has dramatically improved the competitiveness of education
staff salaries in order to attract and retain well-qualified teachers
1. Prior to the Education Initiative
A 2001 DYS Legislative Report identified “concern about the extraordinarily high
turnover rate amongst DYS teachers and its effect on the quality of educational
services. It concluded that high turnover and problems with recruiting qualified
staff were primarily attributable to salary inequities between DYS and public
school districts
2. In response
DYS implemented a more competitive salary scale that increased the minimum
salary for bachelor’s-level teachers, bringing starting salaries in line with state
averages. The new salary structure also offers compelling incentives for teachers
to pursue full professional licensure
3. Looking ahead
As DYS continues to consider important reforms to its salary structure, it may
need to offer higher salaries in key sub-regions where public school salaries are
highest. Further, DYS may need to equalize teacher benefits and working
conditions across program vendors
Implementation Findings
1.
Educational Service Delivery
DYS has dramatically improved the competitiveness of education staff salaries in
order to attract and retain well-qualified teachers
DYS has made substantial progress in the development and implementation of a
system-wide curriculum approach supported with instructional guides, templates, and
material resources.
DYS and its partners have implemented an extensive professional development
system and instructional coaching model.
2.
Transition Services
DYS has identified a need for more robust employment pathways for appropriate
DYS youth and is actively piloting and assessing new programs, and taking other
steps, to support this need.
DYS has improved the infrastructure to support student transitions within DYS and
into academic settings in the community. New models are being pursued.
Implementation Findings
3.
Education System Management
DYS has expanded its management capacity and educational expertise through use
of external vendors – most notably CommCorp and HEC.
DYS has improved its education system coordination and has taken important steps
toward establishing more effective system-level accountability.
Short-Term Impacts of the Education Initiative
Outcomes remain preliminary; however, it appears to have resulted in
several positive short-term impacts within DYS Education
Perceptions of Education Quality
–
DYS teachers’ perceptions of the quality of education provided to DYS youth have
markedly improved, and future expectations are for continued improvements.
Trends in DYS Teacher Perceptions of Educational Quality
Proportion Indicating Good or Excellent
100%
Trends in DYS Teacher Perceptions of Educational Quality
Proportion Indicating Excellent
40%
37%
86%
85%
30%
80%
70%
20%
19%
10%
55%
8%
52%
0%
40%
January 2005
January 2007
January 2008
January 2005
January 2007
January 2008
Short-Term Impacts of the Education Initiative
Workforce Stability
Data show positive impacts on the stability of the DYS teacher workforce over the past three
years, including a decrease in teacher turnover and in August vacancy rates
Data also showed a substantial improvement in teacher qualifications
DYS Teacher Vacancy Data
Aug
Sept
12 Month Average
FY 04
20.2%
11.2%
6.6%
FY 06
3.8%
6.0%
4.2%
Additional Short-Term Impacts
Changes in Instructional Practices
According to teachers, new curriculum tools and templates have made a positive
impact on instructional practice, and instructional coaches’ reports indicate that
teachers are now implementing recommended practices with increasing frequency
and quality.
Teachers report that statewide PD has had a moderate to high impact on their
instructional practice and student learning in the past three years. Data also suggest
that instructional coaching helps many teachers reflect on their teaching style and
encourages them to adopt new teaching strategies.
Impact on Educational Culture
Data suggest a positive cultural shift has resulted from the adoption of education
reform within DYS, and that a stronger professional culture has emerged within DYS
Education. These trends are strongest among treatment programs, but evident
system-wide.
Other Features of the Final Evaluation Report
Strategic Considerations
A set of strategic considerations intended to provoke discussion and inform debate among
DYS and its partners as they move into the next phase of education reform
Featured Practice Case Studies
A section featuring six case studies that focus on new programs, tools, and processes
developed through the Education Initiative
– A statewide professional development system
– A system-wide instructional coaching model
– A process for effective piloting of youth employability programs
– A work-based training program in partnership with Putnam Vocational Technical High School
– A universal student transcript to support student credit recovery
– A “Community Transition School” to support student re-entry into public schools
Other Reports
Additional evaluation reports available include two literature review reports, and
results from the DYS teacher survey administered in January 2007
Lessons Learned
Money, time, experience
–
More money would have allowed development of additional data and more in-depth
research
Barriers
–
–
Data still inconsistent across the system; not always available in a useful format
Data often incomplete or not up to date
What DYS would do in the future
–
–
Look at what baseline data would be needed for future evaluations to ensure more
quantitative results
Look at specific initiatives that would be evaluated and tied to evidence based
research practices
For Further Information
Christine Kenney, Director of Educational Services
Massachusetts Department of Youth Services
[email protected]
617-960-3324
Lonnie Kaufman, Project Manager
University of Massachusetts Donahue Institute
[email protected]
413-587-2419
http://www.donahue.umassp.edu