Transcript On-Line Placement Tool Kit

Emerging Practices in VR:
Moving from Pilot Project
to System Integration
6th Annual Summit on VR Program
Evaluation & Quality Assurance
September 16, 2013
Chip Kenney, Deborah Greene
2
Session Objectives
• Share methods used by Southeast TACE
to evaluate integration of emerging
practices in VR systems
• Review/Update findings from 2012
evaluation of Southeast TACE pilot
projects designed to introduce practices.
• Present protocols and tools used to
evaluate impact and measure/track
systems integration.
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2013
3
Overview
• Since late 2009, the Southeast TACE has
been partnering with a number of state
vocational rehabilitation (VR) agencies to
integrate innovative practices within the
VR service delivery mainstream.
• Various states are piloting or integrating
approaches such as Customized
Employment and Asset Development.
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2013
4
Overview, cont.
• Five of these “Innovations Projects” have
•
progressed to a point where their impact can be
assessed to inform future action at local, state
and regional levels.
In 2012 TACE worked with project teams to
evaluate individual project experiences as well
as the Innovations initiative as a whole, in terms
of their impact on the adoption of best
practices within VR systems in the Southeast.
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2013
5
Overview, cont.
• All 5 projects are now taking steps to
integrate piloted practices into VR delivery
systems
• Three additional Innovations Projects are
under way
• Evaluation of project activities continues
through networking, tracking tools and
Integration Scale measures
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2013
6
Overview, cont.
Current status
•5 Southeast states, 5 state agencies
•8 projects in various stages of design or
implementation, focused on…
 Aspects of Customized Employment (7)
 Asset Development (1)
•5 have completed a pilot & are implementing
further integration strategies
•2 projects from original count on hold due to
agency internal changes
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2013
7
Overview, cont.
Summary of evaluation findings
•VRCs in 4 states have viable option to serve
jobseekers with most significant disabilities, where
previously they had no effective option
•Results directly attributable to pilot projects
•Value of investment evidenced in…
 States’ further strategies to fully adopt the practice
 Refinements/Evolution of the practice itself
 Continuous improvement of the Innovations initiative
•New design/evaluation tools (Roadmap, Integration
Scale, Community of Practice)
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2013
8
Innovations strategy: Goal
Increase the capacity of state
agencies and their community
partners to improve employment
and self-sufficiency outcomes for
individuals with significant and
most disabilities
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2013
9
Innovations strategy: Premise
Adoption of innovative practices requires more
than training, hence a phased process from pilot to
sustainability to integration
•Incubator to understand/address feasibility and systems
implications before taking it to scale
•Partners can try the practice out, get their feet wet, make it
their own, get buy-in
•Admin. and service delivery partners experience and
influence the process – credibility, support for the practice
and its feasibility
•Phases designed according to local needs, resources,
readiness, logistics
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2013
10
Approach to Innovation Projects
• Roadmap
• TA/CE
Pilot
Evaluate
•
•
•
•
Impact
Feasibility
ROI
Outcomes
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2012
• Expansion
• Community of
Practice
• Ongoing TA
Sustain
Integrate
• Integration
Scale
11
Innovations strategy: Process
• Meet with agency leadership/staff re. goals, success
•
•
•
•
•
•
•
•
•
measures, strategy
Agree on multi-phase design
Assign project manager/team
Develop project roadmap (workplan, resources)
Identify pilot sites, partners, resources
Conduct performance-based training
Facilitate partnerships and infrastructure supports
Evaluate pilot phase
Refine plans for expansion/sustainability/integration
Launch subsequent phases
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2013
12
Description of Pilots
Original 5 projects
• Florida (2)
 Customized Employment - Transition
 Discovery Certification - Transition and Adult
• Mississippi
 Customized Employment - Adult
• Georgia
 Customized Employment - Transition and Adult
• Kentucky
 Customized Employment - Adult
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2013
13
Description of Pilots, cont.
3 new projects since 2012
• Kentucky
 Asset Development
• Georgia
 Discovery in Residential Setting - Adult
• North Carolina
 Customized Employment - Transition
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2013
14
Evaluation, cont.
Design principles
• Demonstrate value of Innovations approach
•
•
•
to support adoption of emerging practices
Assess impact to date of individual projects
and initiative as a whole
Directly connect results with the intervention
Establish process/protocols for ongoing
evaluation and continuous improvement
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2013
15
Evaluation, cont.
Targets
• Outputs
 Trainings conducted, numbers trained
 Meetings facilitated
 Products generated (reports, tools, samples)
• Impact
 Change in attitude/behavior, i.e., awareness/
understanding/utilization of the practice
 Systems changes, e.g., partnerships and
infrastructure measures to operationalize the practice
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2013
16
Evaluation, cont.
Targets, cont.
• Return on investment: Start-up and operational
•
•
costs, value-added
Customer satisfaction: Agency/Partner
assessment of project experience and quality of
TACE services
Jobseeker outcomes
 Completed profiles
 Job placements
 Financial security measures
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2013
17
Evaluation, cont.
• Subjects: Projects that…
 Completed the pilot or training phase (at least), and
 Took or are considering measures to ramp up
• Types of data
 Quantitative (trainings conducted, # participants,
active practitioners, etc.)
 Qualitative (partnerships & infrastructure, attitudes &
behavior)
• Overall Innovations initiative as a TACE
technical assistance activity
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2013
18
Evaluation, cont.
Data sources
• Tracking tools, reports and other documentation
•
•
•
•
(communications, products, etc.)
Southeast TACE team (staff & consultants
involved in development of initiative, design and
execution of projects, etc.)
Agency leadership
Project leads & participants (incl. jobseekers and
their families)
Communities of practice
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2013
19
Evaluation, cont.
Process
• Agree on measures, design evaluation strategy
•
•
•
•
with input from TACE team, agencies, projects
Develop/Update evaluation Work Plan
Design data collection and interview protocol
(source documents, key informant interviews,
focus group)
Collect/Analyze prelim. Data
Draft individual project reports and aggregate
discussion paper
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2013
20
Evaluation, cont.
Process, cont.
•Conduct focus group among project leads
•Synthesize findings – focus group report
•Disseminate white paper summarizing findings
and recommendations (TACE website)
•Publish article in Journal of Rehabilitation
Administration
•Construct new evaluation tools
 Project Roadmap
 Integration Scale
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2013
21
Evaluation, cont.
Process, cont.
•Convene communities of practice
 Customized Employment
 Asset Development
•Update evaluation measures and strategy
 Outputs (projects, trainings, TA activities, certified or
authorized practitioners)
 Systems integration/impact indicators (policies,
MOUs, curricula, expansion activities)
 Operations indicators – changes in attitudes, practices
 Client impact indicators – quantitative, qualitative
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2013
22
Evaluation, cont.
Findings: Infrastructure
•CE/Discovery defined/endorsed as discrete
services in policy/regulation (MS, GA, FL, KY)
•CE/Discovery incl. in VR strategic plans (FL, GA)
•Fee code changes (MS, GA, FL, KY)
•Long term supports




FL: DD agencies
GA: MOUs between GVRA and DD/MH
KY: HCBS waiver – require referral to VR
MS: HCBS waiver – in discussion
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2013
23
Evaluation, cont.
Findings: Capacity-building
• Internal Capacity-building
 Creation/Modification of positions to support CE
 Creating mentor/support teams
• Provider/Partner Capacity-building
 Outreach/Funding strategies to train/develop a pool of
VR-approved CRPs (GA)
 Communities of practice (FL, MS, GA)
 Mentoring strategies (MS, FL, KY)
 University-based training or credentialing program in
progress (GA, FL, KY)
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2013
24
Evaluation, cont.
Findings: Partnerships
•School Districts (early collaboration for transition)
•Employment First
•DD and MH agencies
•Universities: capacity-building, evaluation
•Joint outreach & tracking strategies
•Shared recognition of practices as cross-system
continuum of services requiring collaboration
•Reduced ‘finger pointing’
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2013
25
Evaluation, cont.
Findings: Systems Change
• Awareness of CE as an option and
increasingly requesting CE or Discovery
for a given jobseeker
• Understanding of the innovation as a
change in VR practice
• Use of CE by VR systems practitioners
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2013
26
Roadmap
• Step-by-step guidance in developing Innovation
•
projects
Ensures successful implementation, fidelity







Objectives, success measures, evaluation
Phases
Stakeholders, partners
Expectations
Advance work
Workplan, resources
Deliverables
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2013
27
Roadmap, cont.
• Tools to plan project, conduct meetings,
negotiate agreements, e.g.





SWOT analysis
Resource assessment
Readiness/Feasibility assessment
Risk assessment
Meeting agenda template
• Use for design, development, tracking and
evaluation
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2013
28
Integration Scale
Key indicators
• Agency Buy-in
• Policy and Procedures
• Capacity/Resources
• Data Collection
• Accountability/ Results Reporting
• Evaluation/Impact/
Continuous Improvement
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2012
29
Integration Scale: Agency Buy-In
• Agency Leadership
• Agency Champion
• Administrative Level
• Field Service Level
• Partner Level
• Success Measures incorporated in
strategic plan, agency identity,
accountability
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2012
30
Integration Scale: Policy/Procedures
• Policy documents, communications
• Fee structure that adequately
•
•
compensates providers
Field Service Manual
Built into staff job descriptions,
performance expectations
(management, supervision, field services
and support)
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2012
31
Integration Scale: Capacity/Resources
• Competencies Identified
• Qualified Providers
• Skilled Staff (management, supervision, field services
•
•
•
•
•
•
•
and support)
Adequate financial resources
Ongoing training
Certification Process for Providers
Mentoring
Tools
Technical Assistance
Partners/ Formal Agreements
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2012
32
Integration Scale: Data Collection
• Tracking
 Employment outcomes
 Client satisfaction
 Longitudinal
• Planned Service
• Purchased Service
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2012
Integration Scale:
Accountability/ Results Reporting
• Performance expectations
• Reporting requirements
• Periodic reviews
• Community of Practice with
designated agency liaison
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2012
33
Integration Scale:
Eval./ Continuous Improvement
• Evaluation strategy, workplan
• Periodic review of results
• Results factored into strategic
plan, milestones, infrastructure,
capacity-building
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2012
34
35
Community of Practice
• Representatives of Innovation Projects
meet at least quarterly
 Customized Employment
 Asset Development
•
•
•
•
Ongoing technical assistance
Peer networking, exchange
Ongoing needs assessment
Ongoing accountability and evaluation
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2012
36
Impact Measures
Extent to which advances in integration can be
attributed to TACE strategy
•Outputs (projects, trainings, TA activities, certified
or authorized practitioners)
•Systems integration indicators (policies, MOUs,
curricula, expansion activities)
•Operations indicators – changes in attitudes,
knowledge; use of practices; referral rates
•Client impact indicators – quantitative, qualitative
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2013
37
Observations: Highlights
• Leadership willingness to take risks, devote
•
•
•
•
•
resources to a practice untested in their system
Systems changes that substantially increased
capacity to serve jobseekers with the most
significant disabilities
Outcomes clearly attributable to the intervention
Costs deemed worth the return on investment
Pilot results are basis for decisions to expand,
replicate, sustain the practice
New products, new directions (e.g., residential)
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2013
38
Observations: Challenges
• Buy-in: Reinforcing that this is about serving a
•
•
population for whom VR traditionally has not had
a successful strategy; that building capacity will
take time/effort and will be a departure from
business as usual
Consistently engaging leadership &
management commitment sufficient to support
and sustain a project (even w/ personnel changes)
Project planning to create environment for
successful pilot
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2013
39
Observations: Challenges
• Articulating success measures directly connected
•
•
•
•
to project activities
Identifying terms/triggers for moving from one
phase of implementation to another
Prioritizing resources for project management and
face-to-face training and technical assistance.
Communicating project parameters, expectations
Logistics: Scheduling training, assigning cases
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2013
40
Recommendations
• Phased approach for practices that are a
significant departure from business as usual
• Success measures clearly connected to
project activities and decisions to expand/
sustain the practice
• Project workplan covering all phases (pilot to
evaluation/replication to full implementation)
• Project management and facilitation
resources
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2013
41
Recommendations
• Clear communication of expectations/
•
•
•
•
implications in advance of committing to pilot
Supplemental resources to finance project
activities and ongoing operations
Distance learning platforms rather than
conference calls
Leadership program for administrators
providing info. on promising practices
Supports to sustain/integrate practices
(Roadmap, Integration Scale, Community of Practice)
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2013
42
Questions
&
Discussion
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2013
43
Southeast TACE Region IV
Toll-free: (866) 518-7750 [voice/tty]
Fax: (404) 541-9002
Web: TACEsoutheast.org
MyTACE Portal: TACEsoutheast.org/myportal
Email: [email protected]
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2013
44
Disclaimer
This presentation was developed by the
TACE Center: Region IV ©2013
with funds from the U.S. Department of
Education, Rehabilitation Services
Administration (RSA) under the priority of
Technical Assistance and Continuing Education
Projects (TACE) – Grant #H264A080021.
However, the contents of this presentation do
not necessarily represent the policy of the RSA
and you should not assume endorsement by the
Federal Government [34 CFR 75.620 (b)].
TACE Center: Region IV, a project of the Burton Blatt Institute.
Funded by RSA Grant # H264A080021. © 2013