Transcript Writing Objectives, Identifying Outcomes & Impacts

Evaluation in the Field:
Putting Concepts into
Action
Janet Myers, PhD MPH
Richard Vezina, MPH
CAPS HIV Prevention Conference
April 21, 2006
Overview

Warm-up

Where does Evaluation fit?

Approaches to Evaluation

Examples

Q&A: Evaluating Your Programs
Warm-Up
Who here is …
Administrator?
Service
Provider?
Evaluator?
4 Questions:




What are the benefits of evaluating your
programs?
What are the challenges to evaluating
your programs?
What needs (besides $) do you have in
order to plan/conduct evaluation?
What resources do you have/use for
evaluation?
Where does Evaluation fit?
Mission
Program
&
Evaluation
PLANNING
Goals
Objectives
Activities
Outcomes
Impact
O
u
t
c
o
m
e
E
v
a
l
u
a
t
i
o
n
Program &
Process Evaluation
IMPLEMENTATION
Mission
 Provides
the vision
 How
this work makes a difference
in the world
 Broadest
scope
Goals (Ross & Mico, 1980; McKenzie & Smeltzer 2001)

“A future event toward which a
committed endeavor is directed”

Simple & concise

2 basic components


“Who will be affected”
“What will change as a result of the
program”
Objectives


Specific steps that contribute to a goal.
Often several objectives per goal.
Good objectives are SMART:
S – specific
M – measurable
A – attainable
R – realistic
T – time-bound
Good Objectives Show… (McKenzie & Smeltzer 2001)




What will change: Outcome that will be
achieved
When will it change: Conditions under
which the outcomes will be observed
How much change: Criteria for deciding
whether the outcomes has been achieved
Who will change: Target population
Activities



Internal: administrative, etc.
External: the services you provide
to clients
Based on your goals/objectives
Outcomes




Changes that occur in people being served
by your program
Attribution: To the best extent possible,
show that change is a result of your program
(but note…causality is difficult)
Standards are typically different for
evaluation (vs. research)
To assess, you need at least 2 time points
(pre- and post-) and/or a comparison group
Impact




The scope of the program’s effects, the
duration of its outcomes and the extent of its
influence on the broader context (for example,
HIV incidence)
Attribution: Can be more challenging to show
causality, because looking for more diffuse
effects
Usually broad and long-term
Typically not in the scope of program
evaluation
Approaches to
Evaluation
Why do we evaluate?

To determine if objectives are
being met

To improve quality of the program

To decide how to change content

To identify the effects of the
program
Process vs. Outcome Evaluation


Process
 Demographics (Who’s being trained?)
 Reaction to content (“Smile Sheets”)
 Service units delivered
Outcome
 Changes in knowledge/attitudes/beliefs
 Changes in behavior
 Impact on patients/clients
Process Evaluation can help us…

Create better learning environment

Improve presentation skills

Show accountability

Reflect the target populations

Track service units
Outcome Evaluation can help us…

Show the program’s effects

Allow for comparisons over time


Provide specific guide points for
improving programs
Show accountability
Planning Your Evaluation (1)

Figure out your questions: What will
this be used for?

Consider your Resources





Staffing
Time
Materials
$$$
Choose Methods


Quantitative: Surveys, pre/post tests, etc.
Qualitative: Interviews, focus groups, etc.
Planning Your Evaluation (2)

Select Indicator(s)




Relevant
Measurable
Improvable
Instrument/Tool Development

Don’t reinvent the wheel!

Analysis: Get answers to your questions

Reporting: Formal & Informal
Examples