Transcript PRESENTATION NAME - University of California, Davis

What Do Programs Want?
Jeanette Treiber, PhD
Project Director
Center for
Program Evaluation and Research
UC Davis School of Medicine
Objective
Determine local
agencies’ evaluation
capacity building
needs
Background

California Department of Public Health
(CDPH)’s Tobacco Control Program is
carried out by


61 Public Health Departments
Approximately 40 Competetive Grantees
More Background
CDPH Strategy
Norm Change
 Community base
 Community
Assessments
 3yr work plans
 Evaluation

Evaluation Components
Local Program Evaluation requirements





Comprehensive Evaluation Plan
Process and Outcome Evaluation
Internal or external evaluators
Progress Reports
Final Evaluation Reports
Tobacco Control Evaluation Center
(TCEC)
TCEC provides Evaluation Capacity Building
services to California TC programs through:
 TA
 Training
 Tools
 FER
scoring
and feedback
Rationale for TCEC Services
“…pressures have been accompanied by
increasing awareness (and often
frustration) on the part of funders as they
confront the limited ability of the
organizations they fund to meet the
challenge of evaluating programs in ways
that are likely to yield either genuine
outcome-oriented accountability or useful
information for formative improvement
programs” Stevenson et.al. 2002
Evaluation Capacity Building
Observed need and response in past
decade
 Various approaches to CB
 Various definitions
 Various approaches to
measuring effect of CB
 Still little research

Method
1. A 21 item electronic survey
(using the online service
surveymonkey.com) to a list
of 170 project directors and
evaluators.
2. Analysis of access database
for technical assistance from
2004 to 2009 (approx. 4.5
yrs)
Results (survey)
Need for training or materials
High or
Moderate
Need
Using culturally competent evaluation methods ……………….79.4 %
Writing evaluation plans ……………………………………….59.9 %
Adapting/developing data collection instruments ……………...57.1 %
Interpreting evaluation results…………………………………..54.1 %
As a reference during evaluation activities…………...................51.5 %
Writing evaluation reports……………………………………….50 %
Collecting evaluation data……………………………………….42.9 %
Projected Use of TCEC (survey)
Projected use
Response Percent
Finding, developing,
or adapting data collection
Instruments………………………………………..68.6
Writing an evaluation plan ……………………….57.1
As a reference during
evaluation activities ………………………………48.6
To build overall evaluation
capacity of the project…………………………….42.9
Preferred Format (survey)
Webinar
68 %
“How-to” evaluation
guides on website
57.1%
Individual consultation
by phone or e-mail 45.7%
Website strategy
exchange
42.9%
Requests for Assistance (TA Log)
Total number of TA requests between 11/2/04
and 7/16/09
512
Average: 11 requests/month
Up to 27 requests/agency total
Main Request types (TA Log)
Type of requests
300
250
200
150
100
50
0
Plans
tools
Analysis
Other
Type of Data Collection Instrument
189
200
180
160
140
120
100
80
60
40
44
39
20
0
Key informant
Interviews
Surveys
Observation tools
TA Demand
Number of projects that have requested TA
70
60
(79%)
50
(71%)
40
Total number of
projects
30
Total number of
projects
requesting TA
20
10
0
LLA
Compet
FER Score and TA requests
Final Evaluation report scores for Local Lead
Agencies were correlated with frequency
of TA request.
No statistically
significant correlation
was found.
Summary/Conclusion
Majority of local health departments AND
community organizations in CA TC require
evaluation assistance
 Greatest need is with DCIs, developing
Eval plans, and help conducting evaluation
with diverse populations.
 Most DCI dev. need is with surveys
 Webinars are the most desired mode of
training delivery
 Individual assistance requests vary greatly
in frequency, scope, and content

Limitations
Response rate to survey was only 20%
 TA assistance was not consistently entered
into log in the same manner
 Requests for TA are somewhat dependent
on marketing of TA
 TCEC team has changed personnel
throughout the 4.5 years

What do Programs Want?
•Flexible,
Varied
Evaluation
Assistance;
•Convenient
Delivery
References
California Department of Public Health Services,
Tobacco Control Program.
http://www.cdph.ca.gov/programs/tobacco/Page
s/default.aspx
Innovation Network – Transforming Evaluation for
Social Change. http://www.innonet.org/
Preskill, Hallie. (2005). Building Evaluation
Capacity. 72 Activities for Teaching and Training.
Thousand Oaks (Sage).
More Resources
Stevenson, John, Paul Florin, Dana Scott Mills, and
Marco Andrade. (2002). Building Evaluation
Capacity in Human Service Organizations: A Case
Study. Evaluation and Program Planning 25, 233243.
Tobacco Control Evaluation Center (TCEC). Center
for Program Evaluation and Research, UC Davis.
http://programeval.ucdavis.edu
University of Wisconsin Extension. Program
Development and Evaluation.
http://www.uwex.edu/ces/pdande/evaluation/ind
ex.html
Contact
Jeanette Treiber, PhD
Project Director
Center for Program Evaluation and Research
Tobacco Control Evaluation Center
UC Davis School of Medicine
[email protected]
http://programeval.ucdavis.edu
1616 Da Vinci Court
Davis, CA 95618