Transcript sdfsdf

Program Evaluation
Needs Assessment Survey of Local
Coalitions
How many evaluators does it take to
change a light bulb?








One to do a needs assessment
One to do a feasibility study
One to do a qualitative study to find out what bulb to change
One to empower the bulb to change
One to tender a contract for further study
One to write performance indicators for success
One to do a cost benefit analysis of the best bulb to buy
One to do a meta-evaluation showing that all previous
studies have left everyone in the dark
SO.......how many evaluators does it
take?
None, actually. Evaluators don't change
bulbs, that's an implementation problem!
-Program Evaluation
gets a lot of flack!
But it can save your assets!
Objectives

Understand the purpose of program
evaluation

Review local coalition survey results
Most commonly asked PE question
How do I know I’m making a difference?
 Answer : Institute best practices…
 With fidelity, innovation, and critical
interpretation.
 Unfortunately, outcome evaluation is
expensive and rarely feasible with
community intervention.

Most commonly asked PE question
How do I know I’m making a difference?
 Answer : Institute best practices…
 With fidelity, innovation, and critical
interpretation.
 Unfortunately, outcome evaluation is
expensive and rarely feasible with
community intervention.

Process evaluation helps implement
best practices with fidelity.

Is my programming “evidence-based”?

Nicotine patches are great. Stick one over each eye and
you can't find your cigarettes.
Is the “best practice” appropriate for my
population?
 What does the tobacco science say about
adapting best practices to my population?
 Am I appropriately following the science?

Process vs. Outcome Evaluation

Process evaluation: Determines whether
best-practices were adopted properly.

Outcome evaluation: Determines whether
program was successful.
Process Evaluation
Assesses the delivery or implementation
of a program.
 What happened and why? How is this
different from what was planned?
 To be evaluated:

•
•
•
Activities
Materials
Delivery
•
•
•
Numbers of staff
Audience
Logistics
Process Evaluation
TATU Example:
 How many youth participated in TATU
trainings?
 What were the youths’ perceptions of
each training?
 What were the implementation problems?

Outcome Evaluation

Measures what progress the program has
made towards its goals.

To be evaluated:
•
•
Behaviors
Attitudes
•
•
Knowledge
Skills
Outcome Evaluation

Example goals to evaluate for TATU:
Short term – youth who participate in
TATU gain outreach skills.
 Intermediate – TATU youth reach fellow
students.
 Long term – Youth who TATU participants
reach actually use skills to abstain when
tobacco is available.

UW Tobacco Surveillance &
Evaluation Program
“It is now proved beyond doubt that
smoking is one of the leading causes of
statistics.”
http://www.medsch.wisc.edu/mep/
UW Tobacco Surveillance &
Evaluation Program





Indirectly provides outcome evaluation of
Wisconsin’s coordinated tobacco control
activities.
Fields surveys.
Analyzes data to monitor state trends.
Identifies progress and emerging challenges.
Future – more geographically specific
information; better disparities data.
http://www.medsch.wisc.edu/mep/
Conclusion :
Uses of program evaluation





Integral part of every successful organization.
Clarifies and re-focuses program goals and
strategies.
Verifies that current strategies are still
appropriate.
Refreshes staff expertise in best practices and
evidence-based programming.
Documentation for supporters, funders, and
stakeholders.
Survey Results
Results reported here are for fully-funded
coalitions ($20,000 or more).
 95% of fully-funded coalitions responded
(40 of 42 coalitions that were contacted).
 76% of coalitions responded in total
(55 of 72).
 94% of coalitions funded at $10,000 or more
responded (44 of 47).

Survey Results:
Is P.E. built into
your regular program functioning?
100%
85%
% of Coalitions
80%
60%
40%
13%
20%
0%
Some current
evaluation
No current
evaluation
Survey Results : The majority of coalitions
(85%) engaged in some program evaluation.
100%
85%
% of Coalitions
80%
60%
40%
13%
20%
0%
Some current
evaluation
No current
evaluation
Regional Distribution of
Coalitions
Distribution of fully-funded
coalitions by region
Population
37%
40%
30%
19%
20%
10%
% of Coalitions
% of WI population
40%
22%
14%
9%
0%
30%
20%
21% 21%
17%
24%
19%
10%
0%
N
W
S
NE
SE
N
W
S
NE
SE
Coalitions distributed geographically, rather
than proportionate with population.
Distribution of fully-funded
coalitions by region
Population
37%
40%
30%
19%
20%
10%
% of Coalitions
% of WI population
40%
22%
14%
9%
0%
30%
20%
21% 21%
17%
24%
19%
10%
0%
N
W
S
NE
SE
N
W
S
NE
SE
Number of full-time equivalent
employees (FTEs) in Wisconsin:
All fully-funded coalitions
35
30
25
20
15
10
5
0.82
0
FTEs in Wisconsin
33 FTEs statewide
 0.82 per coalition
(33 hours per week)
 About 40¼ total
staff.

33
Average FTEs per
coalition
FTEs by Region:
Number of residents per FTE
Total number of FTEs
10
8.5
8
6
274,948
300,000
6.9
5.1
7.5
250,000
200,000
4.7
150,000
4
100,000
2
50,000
0
0
N
W
S
NE
SE
176,440
162,405
125,844
97,678
N
W
S
NE
SE


Indication of urban-efficiency gains.
Building partnerships is vital.
Number of residents per FTE
Total number of FTEs
10
8.5
8
6
274,948
300,000
6.9
5.1
7.5
250,000
200,000
4.7
150,000
4
100,000
2
50,000
0
0
N
W
S
NE
SE
176,440
162,405
125,844
97,678
N
W
S
NE
SE


Most coalitions are funded entirely through a TPCP contract.
This raises the potential for instability given a potential change
in state government.
What steps are coalitions taking to diversify?
Grants Received by Funded Coalitions
(Select all that apply)
% of Coalitions

100%
80%
87%
71%
60%
40%
17%
20%
0%
Only TPCP Only TPCP
and TTA
contract
funding
funding
TTA
10%
7%
2%
7%
Community
Foundation
Federal
American
Legacy
Other
grants
Priority Populations
(Choose three)
Youth
General Public
Adults
Policymakers
low SES
Pregnant Women
Ethnic/racial minority
Employers
AODA/mental hlth
18-24
Vets
Elderly
Other Populations
90%

Targeting youth
has likely paid
off.

After rising
through 1999,
youth cigarette
use now drops
every year.
53%
46%
40%
28%
24%
12%
10%
8%
7%
0%
0%
3%
0%
20%
40%
60%
% of Coalitions
80%
100%
Priority Populations
(Choose three)
Youth
General Public
Adults
Policymakers
low SES
Pregnant Women
Ethnic/racial minority
Employers
AODA/mental hlth
18-24
Vets
Elderly
Other Populations
90%
53%

46%
40%
28%
24%
12%
10%
8%
7%
0%
0%
3%
0%
20%
40%
60%
% of Coalitions
80%
100%
More resources
needed for 1824 year olds.
Priority Populations
(Choose three)
Youth
General Public
Adults
Policymakers
low SES
Pregnant Women
Ethnic/racial minority
Employers
AODA/mental hlth
18-24
Vets
Elderly
Other Populations
90%
53%

Less than 50%
target
policymakers.

“Other”
populations
reported include
health care
providers.
46%
40%
28%
24%
12%
10%
8%
7%
0%
0%
3%
0%
20%
40%
60%
% of Coalitions
80%
100%
Coalition Objectives
(Check all that apply)
Youth prevention
90%
Smoke-free worksites
88%
Youth advocacy
73%
Treatment
68%
Disparities
58%
Smoke-free homes
38%
Smoke-free campuses
33%
Other Goals
20%
0%
20%
40%
60%
% of Coalitions
80%
100%
“Other” reported
goals:
 Reducing
tobacco use
among
pregnant
women
 Increasing the
tobacco tax.
Educate policymakers
93%
Wisconsin Wins
85%
Youth activities
83%
Media
83%
Educational materials
80%
Educate members on advocacy
80%
Recruit supporters
78%
Community partnerships
78%
Employers, smoke-free
(Check all that
apply)
 Coalitions
63%
FACT
60%
Employers, treatment
45%
Increase community members' skills
45%
TATU
43%
Train health professionals
43%
First Breath
selected an
average of 10
activities.
 All selected
40%
Pursue funding
23%
Other activities
more than one
activity.
8%
0%
20%
Major
Coalition
Activities
40%
60%
% of Coalitions
80%
100%
Educate policymakers
93%
Wisconsin Wins
85%
Youth activities
83%
Media
83%
Educational materials
80%
Educate members on advocacy
80%
Recruit supporters
78%
Community partnerships
78%
Employers, smoke-free
(Check all that
apply)
 Less than 100%
63%
FACT
60%
Employers, treatment
45%
Increase community members' skills
45%
TATU
43%
Train health professionals
43%
First Breath
recruit
supporters and
seek community
partnerships.
 Less than ¼
40%
Pursue funding
23%
Other activities
pursue
alternative funds.
8%
0%
20%
Major
Coalition
Activities
40%
60%
% of Coalitions
80%
100%
Educate policymakers
93%
Wisconsin Wins
85%
Youth activities
83%
Media
83%
Educational materials
80%
Educate members on advocacy
80%
Recruit supporters
78%
Community partnerships
78%
Employers, smoke-free
(Check all that
apply)
63%
FACT
60%
Employers, treatment
45%
Increase community members' skills
45%
TATU
43%
Train health professionals
43%
First Breath
40%
Pursue funding
23%
Other activities
8%
0%
20%
40%
60%
% of Coalitions
80%
Major
Coalition
Activities
100%
“Other” activities:
 working with state
partners.
 helping worksites
create individual
policies.
 asthma workgroups.
 the N-O-T program.
 Oral health
screenings in
disparate
populations.
Back to Program Evaluation…
Track # event attendees
Current
Program
Evaluation
Activities
68%
Lists of supporters
65%
Track earned media
60%
Track # meeting attendees
58%
Quitline Stats
58%
Pre and post tests
40%
Member survey
38%
Solicit event feedback
38%
Track results of paid media
30%
Survey of general public
30%
Youth survey
30%
Observation data
30%
Phone banking results
(Check all that
apply)
20%
Mail survey
18%
Survey of health professionals
15%
Worksite survey
15%
Other evaluation
15%
0%
20%
40%
% of Coalitions
60%
80%
 More than
50%
document
their basic
program
activities.
 Herein lies a
wealth of
experience to
share.
Track # event attendees
Current
Program
Evaluation
Activities
68%
Lists of supporters
65%
Track earned media
60%
Track # meeting attendees
58%
Quitline Stats
58%
Pre and post tests
40%
Member survey
38%
Solicit event feedback
38%
Track results of paid media
30%
Survey of general public
30%
Youth survey
30%
Observation data
30%
Phone banking results
(Check all that
apply)
20%
Mail survey
18%
Survey of health professionals
15%
Worksite survey
15%
Other evaluation
15%
0%
20%
40%
% of Coalitions
60%
80%
“Other”:
 focus groups
 police reports
 CDC’s SPF
indicators
 WI Wins data
 First Breath
statistics
 parent surveys
 oral health
surveys
 data from local
agencies
In your own words…

Coalitions’ answers to open-ended
survey questions:
1.
What questions would you want a program
evaluation to answer?
Please describe any sources of
information/data that might be available for
use in program evaluation.
2.
1. What questions would you want a
program evaluation to answer?




Open-ended responses were grouped
into:
process questions
outcome questions
technical needs
Process Questions:
 Is the program operating as intended?

Are training materials and meeting times best?
 Is the program structure optimal?

Are we reaching the intended population?
 Are our activities “best practice”?

Are we using best practices for helping pregnant women quit?
 Generally, what is the ranking of most- to leastvaluable practices?
 What is the need/demand for our program?
 How important are non-cigarette tobacco products?
Process Questions:
 Is the program operating as intended?

Are training materials and meeting times best?
 Is the program structure optimal?

Are we reaching the intended population?
 Are our activities “best practice”?

Are we using best practices for helping pregnant women quit?
 Generally, what is the ranking of most- to leastvaluable practices?
 What is the need/demand for our program?
 How important are non-cigarette tobacco products?
Outcome Questions:
 What difference has the program made?
 Are we changing attitudes about second hand
smoke?
 Have our education materials impacted attitudes
about smoke-free workplaces?
 What behavior changes resulted?
 Have pledge cards increased smoke-free homes?
 How many/who benefits?
 Progress on goals of multi-year action plan?
 Do results justify the costs?
Technical needs:
 Where can we find detailed local statistics?
 Where to find numbers of youth smokers, by school and by
grade?
 How can City of Milwaukee data be separated from
Milwaukee county data?
 How do we write survey questions that identify attitude and
behavior changes?
 Where can we find template surveys that can be changed to
fit the needs of our community?
 How can we use program evaluation data to promote tobacco
control?
 What are indicators of whether progress is being made?
2. Please describe any sources of
information/data that might be available
for use in program evaluation.
 Open-ended
into:


Qualitative
Quantitative
responses were grouped
Qualitative







Testimony from youth, school leaders, adults who work
with youth, partners, law enforcement.
Documentation of earned media.
Minutes of Board of Health meetings.
Supporter data base.
Mid-year and year-end reports on consolidated contract
tobacco objectives.
Community needs assessment (every 5 years).
Community Health Assessment data.
Quantitative

Statewide surveys/data:

YRBS, BRFS, UW Burden of Tobacco, UW survey of smoke-free municipal
and government-owned buildings, Worksite survey, WIC data on tobacco
use prevalence

Local surveys:

Dane County Youth Assessment, PRIDE survey in county schools, South
West Youth Survey (SWYS), evaluation of local smoke-free ordinance,
county smoke free air/tobacco use survey (400 residents surveyed),
community health assessment data, survey data collected by community
health improvement coalitions, oral health screening results

Program surveys/data:

Friends Helping Friends (youth peer group) survey data, WI Wins
compliance checks data, First Breath data, Quit-line and Fax to Quit
statistics, pre- and post-test data for participants, member survey
Next Steps…
Potentially, there may be regional
trainings in program evaluation methods
in each of the DHFS service areas.
 Show of hands:
 No PE training or experience
 Some PE training
 Some PE experience
 Some PE success

Thanks!

Local coalitions team:











Sue Marten, Chair
Dianna Forrester
Kathy Cahill
Themis Flores Ramos
Connie Olson
Heidi Foster
Deb Bruning
Paula Silha
Deb Gatzke
Karen Hagemann
Jody Moesch-Ebeling

T&TA








Lynn Habrik
Kathy Cahill
Deb Bruning
Mary Hilliker
Sue Marten
Randy Glysch
Cindy Musial
Sue Marten