Transcript Program Evaluation Primer 2012 {PPT}

Driving Quality: Making the
Difference through Program
Evaluation Processes
Ben Kearney, PhD
Guidestone
“I think you should be more explicit here in Step Two.”
… thought Alice and she went on, “Would
you tell me, please, which way I ought to go
from here?”
“That depends a good deal on where you
want to get to,” said the Cat.
“I don’t much care where —” said Alice.
“Then it doesn’t matter where you go,”
said the Cat.
“— so long as I get somewhere,” Alice
added as an explanation.
“Oh, you’re sure to do that,” said the Cat,
“if you only walk long enough.”
(Carroll, 1865, p. 51)
Accountability
• What gets measured gets done
• If you don’t measure results, you can’t tell success
from failure
• If you can’t see success, you can’t reward it
• If you can’t reward success, you’re probably
rewarding failure
• If you can’t see success, you can’t learn from it
• If you can’t recognize failure, you can’t correct it.
• If you can demonstrate results, you can win public
support.
Osborne and Gaebler, 1992
Improvement Process
• Data
• Information
• Knowledge
• Wisdom
Inherent Questions
• Does the program work as designed—correlated
process measures with outcome measures?
• How does the program compare with other
programs and against
organizational/contractual/external benchmarks?
• If effectiveness indicators are met, what
efficiency changes can be made to maintain
quality, but increase client saturation?
A priori assumptions
– Therapy works
– Common factors help facilitate change
• Empathy, increasing awareness, therapeutic
relationship
– Specific interventions permits change to
happen even faster
– We must be responsible stewards
– Make an IMPACT
Components of Program
Evaluation
• Theory of Change
• Logic Models
– Process Measures (including Fidelity
Measures)
– Outcome Measures
• Short term—during treatment
• Mid term--At the end of treatment
• Long term—at some point following treatment—
depends on logic model and change theory
• Status reports
Theory of Change vs. a Logic
Model
• a theory of change defines how and why you
expect the desired outcomes to occur and
usually applies to several programs or the whole
organization
– theories of change clarify why you are doing what you
are doing
• a logic model visually presents your
understanding of the relationships among your
program’s resources, planned activities and
anticipated results and usually applies to a
single program.
– Logic models clarify what you are doing
Theory of Change vs. a Logic
Model
Theory of Change
• Links outcomes and activities to explain how and why
the expected change will occur
• Usually starts with a goal before deciding on
programmatic components
• Requires justification for program components; specifies
the hypothesis about why something will cause
something else
• Requires identifying indicators to measure outcomes
• Best used to design and evaluate a complex initiative
Theory of Change vs. a Logic
Model
Logic Model
• Graphically illustrates program components, identifies,
inputs, activities and outcomes
• Usually starts with a program and illustrates its
components
• Requires identification of program components, but
doesn’t show why activities are expected to produce
outcomes
• Sometimes includes indicators to measure outcomes
• Best used to demonstrate you have carefully identified
the inputs, outputs and outcomes of your work
Evaluation
• A logic model is the first step in evaluation.
Through evaluation, we test and verify the
reality of the program theory – how we
believe the program will work.
• A logic model helps us focus on
appropriate process and outcome
measures.
An Expanded Logic Model
Logic Model Components
• Situation
• Inputs
• Outputs—the measurement of:
– Activities
– Personnel
• Outcomes
The situation…
• …is the foundation for logic model
development.
• The problem or issue that the program is
to address sits within a setting or situation-a complex of sociopolitical,
environmental, and economic conditions.
Situational Questions
•
•
•
•
•
What is the problem/issue?
Why is this a problem? (What causes the problem?)
For whom (individual, household, group, community,
society in general) does this problem exist?
Who has a stake in the problem? (Who cares whether
it is resolved or not?)
What do we know about the problem/issue/people that
are involved? What research, experience do we have?
What do existing research and experience say?
Inputs
• Inputs are the resources and contributions
that you and others make to the effort, the
resources invested that allow us to
achieve the desired outputs. These
include time, people (staff, volunteers),
money, materials, equipment,
partnerships, research base, and
technology among other things.
• These inputs allow us to create outputs
Outputs
• Outputs are the activities, services, events, and
products that reach people (individuals, groups,
agencies) who participate or who are targeted.
• Outputs are "what we do" or "what we offer."
They include workshops, services, conferences,
community surveys, facilitation, in-home
counseling, etc.
• These outputs are intended to lead to specific
outcomes.
Outcomes
• Outcomes are the direct results or
benefits for individuals, families, groups,
communities, organizations, or systems.
Examples include changes in knowledge,
skill development, changes in behavior,
capacities or decision-making, policy
development. Outcomes can be shortterm, medium-term, or longer-term
achievements. Outcomes may be positive,
negative, neutral, intended, or unintended.
Outcomes
• In the past, we've tended to focus on what is
included in the outputs column - the "what we do
and who we reach."
• We are anxious to tell our clients, funders and
community partners what it is that we do, the
services we provide, how we are unique, who
we serve
• Now, the question is: "What difference does it
make?" This is a question about OUTCOMES.
• The newest question is: “How does what you do
produce the differences being made?” This
question links OUTPUTS to OUTCOMES.
Impact
• the ultimate consequence or effects of the
program--for example, increased
economic security, reduced rates of teen
smoking, improved air quality.
• the ultimate, long-term outcome or your
programmatic goal.
• Impact refers to the ultimate, longer-term
changes in social, economic, civic, or
environmental conditions.
Assumptions
• Assumptions are the beliefs we have
about the program and the people
involved and the way we think the program
will work. This is the "theory" we are
talking about: the underlying beliefs in how
it will work. These are validated with
research and experience. Assumptions
underlie and influence the program
decisions we make.
More about Outcomes…
• Outcomes often fall along a continuum
from shorter- to longer-term results. This
continuum is called an "outcome line"
(Mohr, 1995), the "outcome sequence
chart" (Hatry, 1999), or "outcome
hierarchy" (Funnell, 2000). This concept--a
series of outcomes that are connected--is
fundamental to a logic model.
More about Outcomes…
For our process:
• short-term—accomplished during the
program
• medium-term—accomplished at the end of
program
• long-term—accomplished at some point
after the program has ended
Starting at the End
• When planning, start where you want to end.
–
–
–
–
Identify the long-term outcome(s).
What is your end goal?
What will be different?
How will the community, producers, local citizens, the
environment, be different as a result of the program?
• Once you have that long-term outcome (end
result, goal) identified, then work backwards
across the logic model
Outpatient Counseling Fidelity
Checklist
•
•
•
•
•
The worker has utilized Nurturing Parenting principles and philosophies
to guide their interaction with the referred child and/or family.
The therapist has established an optimal therapeutic alliance with the
client and/or family by providing a secure base from which the client
can explore the various unhappy and painful aspects of his/her life, past
and present, through support, encouragement, sympathy, and
guidance.
The therapist has assisted the client in identifying important attachment
relationships; in recognizing the role they play in their current
functioning; to rework unhealthy attachment related behaviors/patterns;
and to promote stronger relationships between the client and their
children or loved ones in the present and for the future.
The therapist has promoted resiliency and generational change in the
client and/or family.
The therapist has utilized Cognitive Behavioral Therapy (CBT)
techniques to assist the client with monitoring and reduction of mental
health symptoms.
Outpatient Counseling Fidelity
Checklist
• The therapist has assessed for the client’s commitment to change;
applied appropriate intervention technique; and promoted client
belief in the notion that therapy will help.
• The therapist has facilitated the client’s awareness of the factors that
maintain his or her difficulties.
• The therapist has encouraged and assisted the client to consider the
ways in which he or she engages in relationships with significant
figures in his or her current life.
• The therapist has helped the client recognize and change images of
self and others that contribute to difficulties.
• The therapist has encouraged the client to engage in corrective
emotional experiences.
• The therapist has emphasized ongoing reality testing in the client’s
life.
• The therapist has worked to promote openness to empathy in the
client.
Helpful evaluation tools
• Website data entry
• Built-in specific reports
• Specific tools for analysis
•
Statistical change
•
Clinical change
•
Analysis tools
•
Run Charts
•
Pareto charts
•
Fishbone-cause and effect
•
Correlations/regressions
Fishbone
Pareto Chart
Pareto Chart-example
Pareto Chart by Cottage - December 2010
100.0%
20
100.0%
95.2%
90.0%
85.7%
80.0%
Number of Restraints
15
70.0%
66.7%
60.0%
50.0%
10
40.0%
7
7
33.3%
30.0%
5
4
20.0%
2
10.0%
1
0
0
0.0%
Open Residential Open Residential Girls SHARP 8-12 Boys
Boys 8-17
8-17 (ML)
(Hilton)
(Crossman/BF)
SHARP 13-17 Boys
(Burris)
Cottage
STC
Crisis/STEP
Run Chart--Example
Sharp 13-17 Boys 2003-2010.12 Restraint Numbers by Month
44.0
39.0
34.0
Number of Restraints
29.0
24.0
19.0 UCL
17.9
14.0
9.0 CL
7.7
4.0
-1.0
LCL
-2.4
-6.0
1
4
7
10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88 91 94 97
Month
Evaluation Potholes
• Be aware where the evaluator’s purpose is
different than the clinicians
• Be aware when the evaluator’s purpose is
the same as the clinicians
• Be clear about roles and boundaries, but
be willing to be flexible when needed as
well
ex: CCO, Evaluator
Evaluation Potholes
• Use well established instruments
• Computerization combines data gathering,
editing, and entering into one step
• Computerization permits automatic
analysis on predetermined factors
• Don’t be afraid to pilot—Pilot when you
can. Force field analysis for pilot—driving
and restraining forces.
Current work
Spirituality
• The openness to and the awareness of the
transcendent, the sacred, that which is
beyond us; and by this knowledge,
recognizing the connectedness and
responsibility we have to each other.
Current Work
• Moral emotional processes-build
communication and collaboration, transmit
culture
– Quietness, openess, mindfulness, awareness,
regulation, empathy
• Transformational emotional processes
– Hope, altruism, sympathy, truth (joy), narrative
(contentment)
References
• Beckworth, L., (2000). Prevention science and
prevention programs. In C. H. Zeanah, Jr. (Ed.),
Handbook of infant mental health (pp 439-456). New
York: Gilford Press.
• Breyfogle III, F. W. (2003). Implementing Six Sigma:
Smarter Solutions Using Statistical Methods (2nd ed.).
Hoboken, NJ: John Wiley & Sons, Inc.
• Brusse, W. (2004). Statistics for Six Sigma Made Easy!
New York, NY: McGraw-Hill.
• Carey, Ph.D., R. G., & Lloyd, Ph.D., R. C. (2001).
Measuring Quality Improvement in Healthcare: A Guide
to Statistical Process Control Applications. Milwaukee,
WI: ASQ.
References
• Cavanagh, R. R., Neuman, R. P., & Pande, P. S.
(2000). The Six Sigma Way: How GE, Motorola,
and Other Top Companies are Honing Their
Performance. New York, NY: McGraw-Hill.
• Cavanagh, R. R., Neuman, R. P., & Pande, P. S.
(2002). The Six Sigma Way. New York, NY:
McGraw-Hill.
• Hayes, R. A., & Stout, C. E. (Eds.) (2005). The
Evidence-Based Practice: Methods, Models, and
Tools for Mental Health Professionals. Hoboken,
NJ: John Wiley & Sons, Inc.
References
• Langley, G. J., Moen, R. D., Nolan, K. M., Nolan, T. W.,
Norman, C. L., & Provost, L. P. (2009). The Improvement
Guide: A Practical Approach to Enhancing Organizational
Performance (2nd ed.). San Francisco, CA: JosseyBass.
• Parlakian, P. and Seibel, N.L. (2002). Building strong
foundations: Practical guidance for promoting the socialemotional development of infants and toddlers.
Washington: ZERO TO THREE Press
• Stiffman, A. R. (Ed.) (2009). The Field Research Survival
Guide. New York, NY: Oxford University Press.