Transcript General Elements in Evaluation Research

Overview of the Individual
Self-Sufficiency Planning
(ISSP) Project Evaluation
1
Type of Evaluation
2
Outcome (or impact)
Evaluation
• Outcomes of interest to leaders of
California’s State Partnership Initiative (SPI)
project have been (1) increased employment
(number working; hours); (2) increased
earnings; (3) job stability and attachment to
the workforce; (4) leaving the SSA benefit
roles; (5) reducing reliance on such benefits;
(6) reducing use of selected, publicly-funded
mental health services; and (7) contributing
findings to public policy.
3
Process Evaluation
• Q1. Have there been any unintended or
unanticipated outcomes experienced,
positive or negative?
• Q2. What barriers have been encountered
in delivery of the intervention strategies
or achieving intended outcomes and, if so,
how were those barriers addressed?
• Q3.How have State and local structures
and procedures changed as a result of the
demonstrations?
4
Process Evaluation
• Q4. What State and Federal policy changes
would have made systems change efforts more
effective and/or permanent?
• Q5. What was the role of the project in
expanding access to health care coverage for
individuals with disabilities?
• Q6. How did the activities of the project assist
in the design and implementation of benefits
planning and assistance activities?
5
Process Evaluation
• Q7.How was the project involved in efforts
to expand access to generic employment
services on the part of individuals with
disabilities? Was the project involved in
attempts to improve the ability of One-Stop
Centers to meet the needs of individuals
with disabilities?
6
Process Evaluation
• Q8.Did the project design and implement
new and innovative service strategies that
led to increased support for individuals
attempting to reduce dependence on
Federal benefits and obtain or expand
employment opportunities?
• Q9.How did the project collaborate with
other reform initiatives underway in the
State at the same time as the SPI project,
such as the WIG, MIG, Medicaid Buy-In,
BPAO, Ticket, etc.?
7
Research Design and
Data Collection
8
Quasi-experimental
Design
• Some SPI projects (e.g., New York) tested
the impact of work-related information, and
used experimental designs with random
assignment to treatment (e.g., basic plus
more advanced information and assistance)
and control groups (e.g., basic information
only).
9
California: A Matched
Comparison Group Design
10
Target Group
• Those signing up for enhanced services (i.e.,
enrolling in the project) had to have
– (1) a severe psychiatric disability;
– (2) be receiving SSI, SSDI, or both; and
– (3) be receiving vocational rehabilitation
and/or employment services.
11
Service Settings
• 17 of about 29 DMH/DOR Cooperative
Projects expressed interest and submitted
proposals.
• Two sites (Vocational Rehabilitation Services,
in San Mateo County; and Kern County
Vocational Services in Bakersfield) were
selected.
• One criterion was engagement with One-Stop
Career Service Centers, because of interest
at both State and federal levels.
12
Enhanced Services
• Over and above existing DMH/DOR
Cooperative program services (assessment,
planning, pre-job search, job search, job
coaching, etc.), “enhanced services” involve
1 Benefits Coordinator and 1 Service
Coordinator for 50 individuals (50:2 ratio).
• San Mateo has had 100 slots, with 2 BCs and
2 SCs. Kern’s program is much smaller, and
has had 50 slots with 1 BC and 1 SC.
13
Matched Comparison Groups
14
Three “Matched
Comparison Groups”
• All selected from the DOR database.
• 1- DOR service recipients in Sacramento and
Fresno counties.
• 2- DOR service recipients across all of the
other DMH/DOR Cooperative Project sites.
• 3- DOR service recipients in selected
counties without a DMH/DOR Cooperative
Project, where the recipient received
services through a “Generalist VR
Counselor.”
15
Stratified Systematic
Sampling
• Random start used across two strata: (1)
Major Disability (Psychosis; Mood Disorder;
Other Psychiatric); and (2) SSA Benefit Status
(SSI; SSDI; Both).
• If information had been available, a “match”
on work history would have been useful
• SSA Benefit Status was chosen as a proxy for
work history
16
Measuring Outcomes
• VCU and Mathematica Policy Research (MPR),
oversee the 17 SPI projects for the SSA.
• Designed a data collection system and are
carrying out a multi-state evaluation.
• ISSP project staff use about 30% of their time
collecting initial demographic and quarterly
update information and transmitting it to
VCU.
17
Measuring Outcomes
• Much of the outcome information through
this system is “self-reported.”
• DOR obtains quarterly UI covered earnings
data through EDD.
• MPR working with SSA to obtain benefit and
related information from SSA files.
• DMH collaborated with DOR to retrieve
selected Medi-Cal information on mental
health services, without jeopardizing privacy
and confidentiality.
18
Measuring Services or
Interventions
• Some use has been made of DOR status
codes, including types of case closures (e.g.,
26s, 28s).
• Most project-related service intervention
data comes from site reports to VCU and
MPR, as part of their data collection system.
19
Measuring Services or
Interventions
• Individuals participate in the project for
varying amounts of time. Some individuals
will leave the project; sites may “graduate”
others; and some participants will become
unreachable.
• Initially, it was thought that the ISSP project
would touch about 500 individuals over the
five years. This target cumulative
enrollment was subsequently reduced to
250, which has been achieved.
20
Site Visits and Project
Meetings
• Decided to make five annual visits to the two
demonstration sites
• Every other year visit the two chief
comparison sites.
• Subsequent visits asked about changes over
the preceding 12 months, in the kinds of
people entering the project, services and
supports, immediate outcomes, significant
community developments and the like.
21
ISSP Statewide
Coordinating Committee
• Initial meetings were monthly, then decided
on day-and-one-half quarterly meetings.
• First half-day devoted to site reports and
data elements (Data Elements Workgroup).
• Second half-day devoted to the work of a
Waiver Workgroup.
• Morning of second day devoted to summary
of preceding day’s work and “big picture”
matters. This group is known as the ISSP
Statewide Coordinating Committee.
22
ISSP Statewide
Coordinating Committee
Expressed a desire for additional information:
• relationships between project personnel and local
SSA office staff;
• the role and importance of natural supports;
• ramifications of having someone specializing in
benefit planning/assistance for the rest of the team
providing voc rehab and employment services;
• job descriptions;
• the paradox of service systems helping some
individuals get benefits, while helping others reduce
reliance on public benefits; and so forth.
23
Lessons Learned, Things to
Think About
24
National Data Collection
• Can California influence the national data
collection scheme, if there is one, to make it
fit better with Bridges and generate the kind
of information Bridges project leaders
envision?
25
National Data Collection
• In ISSP project, data collection forms were
announced, but changes not possible.
• Sites ended up developing their own basic
information system to guide their work and
to report quarterly.
• Some aspects of the national evaluation data
collection scheme seem “excessive.”
• National evaluation uses a much different
comparison group than the California
evaluation.
26
National Evaluation
Should the California Evaluator serve as a
facilitator for the National Evaluation, or
should there be parallel evaluations (one
California; one across the set of States
participating in the Youth Initiative)?
27
Gathering Information
What kind of support will be needed at DOR,
CDE, and SSA to get, store, and analyze
information of value?
28
Resource and
Information Needs
• It will make sense to follow an interactive
process in reaching agreement on what
exactly the California Evaluation team
should do.
• Every aspect of the evaluation has resource,
cost, and other considerations (e.g.,
whether data is shared on paper, via pen or
pencil, or electronically).
29
Outreach and Intake
Very important for several reasons:
(1) choice of comparison group;
(2) advertising potential waivers may
encourage more “at-risk” youth and their
families and more SSA benefit recipients to
be referred and to apply to the project;
and
(3) it may prove difficult to “stay on message.”
30