Transcript Slide 1
Situational Analysis as a Process Tool in Human Services Programming
Presented By Gary Bess, PhD Jim Myers, MSW
Gary Bess Associates School of Social Work, California State University, Chico
1
Presentation Overview
1. Process evaluation defined, and its contribution to developing programs 2. Evaluator’s role in process evaluation in developing programs 3. SWOT assessment defined 4. Using the SWOT in a pilot program 5. Longitudinal tracing of themes within the developing program
2
Process Evaluation
Process evaluation assesses reasons for successful or unsuccessful performance, and provides information for potential replication
[italics added].
Process Evaluation
focuses on how a program was implemented and operates. It identifies the procedures undertaken and the decisions made in developing the program. It describes how the program operates, the services it delivers, and the functions it carries out. *
3 * http://www.bja.evaluationwebsite.org/html/glossary/p.html
Process Evaluation
As a newer form of evaluation research, process evaluation is generally associated with qualitative methods, in that “process” is by definition a qualitative exercise. Concurrent assessments, however, of quantitative results (e.g., number of clients serviced or changes in functioning) -- above or below projections -- are used as measures of the program having met its designers’ expectations.
4
Process Evaluation
Process evaluation is a method of recording and documenting salient ideas, concerns, activities, administrative and management structures, staffing patterns, products, and resources that emerge during the program’s developmental cycle.
5
Process Evaluation
Similar to outcome evaluation that measures the results of a project’s implementation against programmatic projections.
Process evaluation assesses
responses.
a priori
assumptions, such as staff qualifications and training, usefulness of resources provided by other agencies, family or stakeholder receptivity to engagement, client acceptance criteria, and other planned activities and anticipated
6
Evaluator's Role in Process Evaluation
The evaluator’s relationship with program staff is key.
Given the intimacy of interaction, a trust level must be developed between the evaluation team and the program team.
It is essential that evaluators convey their support for the project’s success, tempered by a constructive objectivity, which translates into formative reflection and input.
7
SWOT Assessment
Having addressed the evaluator’s role in process evaluation, we wish to introduce an experimental technique for documenting developmental processes
within programs. It is an approach that creates a
new functional role for the evaluator.
8
SWOT Assessment
The technique is a facilitated conducted by the evaluator. SWOT is an acronym for Strengths, Weaknesses, Opportunities, and Threats (Barry, 1986).
SWOT
analysis, In a conventional SWOT analysis, variations of following questions are sequentially asked . . .
9
SWOT Assessment
1. What are the program’s internal and external strengths?
2. What are the program’s internal and external weaknesses?
3. What opportunities are present, within the program and in the environment that surrounds the program?
4. What threats exist – either internally or externally – that could adversely affect the program’s efforts?
10
Overview of the Pilot Program
We used the SWOT as part of a 30-month pilot initiative in which the authors were engaged by a northern California county consortium of public agencies to evaluate a California Board of Correction’s (BOC) funded Mentally-Ill Offender Crime Reduction Project (MIOCR) – known on the county level as FOREST: Forensic Resource Team.
Process evaluation data was obtained between May 2002 and June 2004, during which time the authors documented the developmental model.
11
Overview of the Pilot Program
The partners were: County’s Superior Court
Department of Behavioral Health
Sheriff ’s Department
Probation Department
District Attorney’s Office
Public Defender’s Office
12
SWOT Assessment: Pilot Program
The SWOT assessment’s dual focus on internal and external dynamics was important, in that the developmental process could have been influenced by environmental changes (external) as well as program-related issues (internal).
13
SWOT Assessment: Pilot Program
The SWOT was conducted approximately every three months in a office or meeting room. Key participants – project staff and managers – were invited, often as part of, or in place of, a regularly scheduled staff meeting. Supplies included one or two large newsprint pads an easels, markers, and tape. As sheets were completed, they were taped on walls so participants were able to review previously cited perceptions.
14
SWOT Assessment: Pilot Program
Ground rules pertained to:
Allowing everyone to speak
Not challenging others’ ideas, but offering your own perceptions, even if they may differ
No labeling of responses as either correct or incorrect
Focus on issues and not solutions
15
SWOT Assessment: Pilot Program
Sometimes clarification was requested as statements were ambiguous or incomplete, such as when “resources” were noted as a
weakness
which could refer to monetary or programmatic materials or professional expertise. ,
16
Themes Manifesting
An assessment of content specific themes identified during early SWOTs, and their progression throughout the program’s lifecycle are presented in Figures 1 through 7.
17
Figure 1 : Commitment of Staff to FOREST Program Month of SWOT
May 2002
Strengths
Commitment of staff
Weaknesses
October 2002 February 2003
Continued commitment of staff [Staff] turnover low
May 2003
Implications for staff turnover – staff transitions
September 2003 December 2003 March 2004 June 2004
BCBH to continue staff positions after FOREST ends that are dedicated to current clients No premature staff departure
Opportunities Threats
Staff attrition as the end of the program looms 18
Figure 2 : Cross-Training Among Collaborative Partners Month of SWOT Strengths Weaknesses
May 2002 October 2002
Lack of cross-training among disciplines
February 2003 May 2003 September 2003
Opportunities
Need more cross-training on other disciplines
December 2003 March 2004 June 2004
Cross-training/ sensitivity among staff Develop BOC web resources on training correctional officers [to work with mentally-ill offenders]
Threats
19
Figure 3: Seeking New Grants/Continuation Funding to Sustain FOREST Month of SWOT
May 2002 October 2002
Strengths Weaknesses
February 2003 May 2003 September 2003 December 2003 March 2004
Opportunities
New grants
Threats
Future grants/ private funding Grant submissions & Jurisdictional mental health grant being submitted Locate future funding for FOREST to continue Looking for SAMHSA grant Future funding has not yet been located or approved
June 2004
No money to directly help clients beyond June 30, 2004 Helping new mental health court programs could produce possible funding for FOREST 20 Funding ending – no new treatment funding
Figure 4 : Client Entry into FOREST – Client Eligibility and Screening Issues Month of SWOT
May 2002 October 2002 February 2003 May 2003 September 2003 December 2003 March 2004 June 2004
Strengths Weaknesses
Speedy intakes; Quick intake screening process Clear definition of eligibility. Lack of credibility in the screening process Continuous education about eligibility requirements
Opportunities
Getting clients on court calendar. Ineligible referrals. Protocols aren’t clear Process of getting screened clients on court calendar. Delay in making client eligibility decisions because of client substance abuse issues Processing potential FOREST participants Getting clients calendared – slow process. Enrollment into FOREST driven by push for dollars and study requirements Intake process compromised due to emphasis in making numbers. Limited access to clients in jail; limited background on which to base acceptance. Study required no change to admission criteria, although experience could have helped in selection
Threats
Eligibility hearings adds to calendar and slows process down 21
Figure 5 : Transitioning Clients Out of FOREST Month of SWOT Strengths Weaknesses
May 2002 October 2002 February 2003 May 2003 September 2003 December 2003 March 2004 June 2004
Smooth transitions Successful linkages for clients leaving programs Client transitions are too slow Transition of clients to regular services Treatment services post graduation is insufficient as clients transition from intensive levels of care to lower levels of care
Opportunities Threats
Lack of transition planning Clumsy transitions for clients not ready to exit when program ends 22 [Forced] transition for clients not yet ready to leave
Figure 6 : Multi-Agency Collaborative Approach – Internal Workings Month of SWOT Strengths Weaknesses
May 2002 October 2002 Feb. 2003 May 2003 September 2003 December 2003 March 2004
Cooperation across staff. Collaborative history among agencies. Credibility due to multi agency participation. Sheriff-to-be is on team Clear understanding of collaborative. Positive relationship among agencies Problem resolution across departments easier and quicker to achieve. Team achieves what individual agencies can’t. Flexibility. “We” identity Agree to disagree. Good team support. Staff increasing knowledge of other agencies Lack of District Attorney (DA)/Public Defender (PD) participation in meetings. DA and PD interface is adversarial. Lack of continuity for intervention planning – different agencies have differing approaches Diverging ideologies – sometimes agencies have different goals Learning to disagree without disagreeing Communication among team Program Coordinator has strengthened and tightened-up team cooperation Collaboration across all agencies. Court room feels like a safe place…not adversarial among agencies Some lack of professional behavior at FOREST meeting and pre-court meetings
Opportunities
To be even more collaborative Relationships between agencies allow for new collaborative ventures
Threats
Conflict between DA & PD on client screenings into FOREST Issues in other agencies “spill over” into collaborative Lack of FOREST staff projecting professional image at FOREST meetings and pre-court meetings 23
Figure 7: Public Relations Month of SWOT Strengths
May 2002
Public awareness of the program. Awareness of programs among allied agencies
October 2002
Weaknesses Opportunities Threats
Client misbehavior/bad press
Feb. 2003
Contact local media to expand awareness of program High profile decomposition of clients. Clients dropping out of program
May 2003
Community perceptions; heightened awareness in the general community. Client success stories.
September 2003
No graduates have gone to jail
December 2003 March 2004 June 2004
Good success with other allied agencies Connection with community. Client success stories. Notoriety among treatment facilities. Positive publicity Community wants program continued; it has been well received Graduated clients have difficulty adjusting to less structure – drop services and are rearrested Publicity – media attention for the 10 graduates. Education of elected officials about FOREST clients. Community awareness- residential treatment programs are expanding to dual issues Potential of public event – bad press 24
Thank You!
25