Transcript Basic Steps for Beginning an Evaluation

Evaluation of Programs that promote
peer support, recovery & community
integration
Mark Salzer, P.h.D.
Matthew R. Federici, M.S, C.P.R.P.
[email protected]
www.mhrecovery .org
www.upennrrtc.org
Does it have to be so difficult?
Evaluation is everybody’s business



‘Plan with the end in mind'
-Stephen Covey
Program Evaluation as a recovery story
Speaking the Language:
–


INPUTS --> ACTIVITIES OR PROCESSES -->
OUTPUTS --> OUTCOMES
By describing work in this way, we have an easier
way to define the work and measure it.
Performance measures can be drawn from any of
the steps.
Evaluation: A Four Part Story
1.
2.
3.
4.
What stuff do have
What do you intend to
provide
What do you provide and
to who
What do you expect will
happen—in the shortterm and in the long-term
as a result of your
program
1.
2.
3.
4.
inputs (resources such as
money, employees, and
equipment) to
work activities, programs
or processes, to
the immediate outputs of
the work that are
delivered to customers, to
outcomes or results that
are the long-term
consequences of
delivering outputs.
What do you intend to provide
a.k.a. Mission and Purpose

The purposes of peer support services are to:
–
–
–
–
–
Provide opportunities for individuals receiving services to
direct their own recovery and advocacy process;
Teach and support acquisition and utilization of skills
needed to facilitate the individual’s recovery;
Promote the knowledge of available service options and
choices;
Promote the utilization of natural resources within the
community; and
Facilitate the development of a sense of wellness and
self-worth.
Inputs: What We Invest
–
–
List what is required to implement your program
Be careful not to list everything just the key
resources



What facilities do you use
Who works for your program
What funding do you have -and where do you get it
Work Activities
Define your activity
1.
2.
3.
4.
5.
Facilitate WRAP groups
Provide Peer Support
Utilize Community Resources
Reinforce the potential for recovery to
peers and community
Promote “consumer” advocacy
Outputs/Deliverables

What will you deliver/services provided
–
These are numerical or quantitative terms



Facilitate 30 support groups a year
Provide peer support to 100 people
Support 60 people to transition to independent housing
Outputs
The production of outputs is a better indicator
that something was delivered to customers,
but it is still possible that the output did not
really meet the customer's needs, was not
used, etc …
http://en.wikipedia.org/wiki/Logic_model
Outcomes

What kind of change is the program designed
to create?
–
–
–
Most programs in human services are designed to
create change in individual lives but can
sometimes serve whole systems/environments
You need to directly relate your outcomes to who
you are targeting
The change needs to be achievable
Key Ingredient: Outcomes




Key ingredient of the Evaluation Process is the importance of
measuring final outcomes
or results,
It is quite possible to waste time and money (inputs), "spin the
wheels" on work activities, or produce outputs without achieving
desired outcomes.
It is these outcomes (impacts, long-term results) that are the
only justification for doing the work in the first place.
For commercial organizations, outcomes relate to profit. For
not-for-profit or governmental organizations, outcomes relate to
successful achievement of mission or program goals.
http://en.wikipedia.org/wiki/Logic_model
Defining your Outcomes

List the community integration changes you
expect from your program?

What activities would promote these
outcomes?
Outcomes
Short Term (learning: awareness, knowledge, skills,
motivations)
–
What changes in the short-term






Participants feel less isolated
Attitudes and beliefs change
People feel understood
Helpers further their recovery
Knowledge is increased
Skills are acquired
More Outcomes
Medium Term (action: behavior, practice, decisions,
policies)
–
What changes in the medium-term








New friendships are formed
People feel hopeful
Newly gained skills are put to use
Employment status changes
A sense of community is created
Participants are empowered to do many new things
Life satisfaction increases
Your program grows
More Outcomes
Long Term (consequences: social, economic,
environmental etc.)
–
What changes in the long-term



Your program is replicated
More similar programs are funded
Your programs helps make:
Recovery becomes the norm
Difference between Outputs/Outcomes


Outputs are what you do ( Deliverables)
Outcomes are what happens because of what you
have done
–
Output:

–
500 consumers will engage in ongoing WRAP groups
Outcome:


75 percent of consumers who participate in ongoing WRAP
groups will report increased use of peer support in their life.
60 percent of the consumers who participate in ongoing WRAP
groups will report improvement in their quality of life.
Outcome Indicators
Knowledge
applied
Changes made
Satisfaction achieved
Empowerment demonstrated
Outcomes




Traditional Outlook
Decrease in
hospitalizations
Improved functioning
Symptom management
Increase medication
compliance




Recovery Outlook
Increase utilization of
nature supports
Increase involvement in
activities of interests
Increase sense of
control of one’s life
Improved quality of life
-Matthew R. Federici, M.S., C.P.R.P 8/08
Measuring performance



Performance measurement- an organized
way to describe the outcomes achieved by
systems, programs and service providers
Once outcomes are determined and clearly
stated, a performance measurement system
can be designed to measure the desired
outcomes
Domain-a large group or cluster of issues
that have something in common
How to Use What You’ve Got and What
You Can Easily Get

Archival Data
–
Sign-in sheets

–
–
–
–
Pseudonyms ok—as long they’re consistent
Monthly or quarterly reports to funders or
oversight entities
Old Satisfaction Surveys
Demographic Info
Contact Notes
Tools you can easily use


Surveys
Questionnaires
–
–
–
–
–
These are easily constructed
Can be one shot—or on a recurring basis (over time is
best)
Note how many people were asked to complete
How many people agreed instrument
How many actually completed instrument
New satisfaction surveys

Process indicators


Ascertaining what program participants know before
they participate in program
Ascertaining what program participants know after they
participate in program
Customer Surveys

The advantage of customer surveys is that they
are relatively quick and easy to implement, but
if properly designed they can give meaningful
feedback on customer perceptions, which are
relevant to outcomes
http://en.wikipedia.org/wiki/Logic_model
Standardized measures
These
have already been proven:
They have reliability and validity
 Hearth Hope Index
 Empowerment Scale
Self-designed measures
•
Drawback is that they are not tested for
reliability and validity
•
Ok to use, if not demonstrating an
evidence-based practice
Measure the performance
EXERCISE:
Peer Specialist Activity
1. Facilitate WRAP groups
2. Provide Peer Support
3. Utilize Community Resources
4. Reinforce the potential for recovery to peers
and community
5. Promote “consumer” advocacy
Motivation for collecting and
evaluating data
To determine what changes you
need to make!
Concerns about what you might find
Listen to your data – Stop playing
the same old tunes…
Resources


Matthew R. Federici M.S., Institute for
Recovery and Community Integration,
(www.mhrecovery.org) C.P.R.P
E-mail: [email protected]
Mark Salzer, Ph.D.
UPENN Collaborative on Community
Integration of Individuals with Psychiatric
Disabilities (www.upennrrtc.org)
E-mail: [email protected]
Resources


Usable Knowledge (2006.) www.usablellc.net

McLaughlin, J. A., & Jordan, G. B. (1999). Logic models: A tool for
telling your program's performance story. Evaluation and Program
Planning, 22(1), 65-72.

The University of Wisconsin-Cooperative Extension has an online
course entitled, Enhancing Program Performance with Logic Models.
The course contains two modules - Module 1, "Logic Model Basics," is
an introduction to logic models; and Module 2, "Introducing The
Community Nutrition Education Logic Model," is an application of logic
models to community nutrition education programs. Each module has
various interactive elements, including practice activities designed to
help students better understand the course content. The citation is:
Taylor-Powell, E., Jones, L., & Henert, E. (2002) Enhancing Program
Performance with Logic Models. Retrieved December 1, 2003, from the
University of Wisconsin-Extension web site:
http://www1.uwex.edu/ces/lmcourse/
References

Plotnick, Debbie (2007) Workshop on Steps for
Beginning an Evaluation

Salzer, Mark (2004) Workshop on Practical
Approach to Outcomes Assessment

Wetzel, RW (2004) “Outcomes, Measurable
Results, Benchmarks: Everyone Wants Them!”
American Society on Aging presentation, 2004