Transcript Developing an Evaluation Plan for a Comprehensive Tobacco

Developing an Evaluation Plan for
a Comprehensive Tobacco Control
Program
Jesse Nodora, DrPH
Michele Walsh, PhD
Joanne Basta, PhD
Learning Objectives
•
Describe what an evaluability assessment
is and the steps involved for conducting
one in order to inform an evaluation plan
•
Describe the key components of an
evaluation plan for a comprehensive
tobacco control program
•
Recognize the major challenges and
rewards of evaluation planning
Presentation Outline
•
Overview of TEPP, TEPP Evaluation
History, and Arizona Population
(background & context)
•
TEPP Evaluability Assessment
•
Key Findings
•
TEPP Evaluation Plan Components
Arizona Demographics
•
•
Population
Race/Ethnicity
•
Non-Hispanic White
Hispanic
American Indian
African American
Asian & P. Islander
•
U.S. Census 2002 Estimates
•
•
•
•
5,456,453
64%
25%
5%
3%
2%
Percent
Arizona Adult Smoking Prevalence
Estimates: 1990-2002
100
90
80
70
60
50
40
30
20
10
0
AZ BRFSS
ATS
National
1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002
AZ BRFSS 20.6
ATS
National
23.8
19.2
20.9
23
22.9
23.7
21.1
21.8
20
18.5
21.5
23.4
-
-
-
-
-
-
23.8
-
-
18.8
-
-
20.1
25.5
25.7
26.5
25
25.5
24.7
24.7
24.1
23.5
23.3
22.8
23
Arizona High School Students
Smoking Prevalence Estimates: 1991-2003
100
90
80
70
60
50
40
30
20
10
0
1991 1993 1995 1997 1999 2000 2001 2002 2003
AZ Substance Abuse Survey
Arizona Youth Tobacco Survey
National Youth Risk Behavior Survey
Arizona Youth Survey
Legacy Foundation National YTS
TEPP Values
TEPP is committed to excellence through integrity and
inclusiveness, honoring individual, cultural and religious
practices.
We value the communities we serve:
Through excellence in developing quality programs and services
that are determined by community needs and are data
driven.
By acting with integrity, being truthful and showing compassion.
By seeking and integrating community input through creating
partnerships, collaborations and accessible services.
By giving respect and dignity with a commitment to honoring
individual, cultural and religious practices.
TEPP Mission
Based upon a belief in excellence,
inclusiveness and integrity that honors
individual culture and religious practices,
the TEPP mission is to protect and
improve the health and quality of life of
all Arizonans by reducing tobacco use
through prevention and treatment, by
denormalizing tobacco use and by
reducing exposure to environmental
tobacco smoke.
TEPP Goals
•
TO REDUCE MORBIDITY AND MORTALITY
RELATED TO TOBACCO USE
•
TO PREVENT AND REDUCE TOBACCO USE AMONG
ALL ARIZONANS
•
TO PROVIDE AND SUPPORT ACCESSIBLE,
AFFORDABLE AND EFFECTIVE CESSATION
SERVICES AND SYSTEMS
•
TO REDUCE ALL ARIZONANS' EXPOSURE TO
SECOND HAND SMOKE (SHS)
TEPP Goals-Cont.
•
TO ENSURE INCLUSITIVITY OF
POPULATIONS WITH TOBACCO CONTROL
DISPARITIES IN PLANNING AND
SERVICES
•
IMPLEMENT AN INTEGRATED,
COMPREHENSIVE EVALUATION MODEL
THAT PLANS, SURVEYS AND EVALUATES
PROGRAM PROCESSES AND OUTCOMES
TEPP Program Components
•
•
•
•
•
Community Programs
Marketing and Public Relations
State-wide Programs
Evaluation
Administration
TEPP Expenditures
•
•
•
•
•
•
•
•
FY
FY
FY
FY
FY
FY
FY
FY
1995/96
1996/97
1997/98
1998/99
1999/00
2000/01
2001/02
2002/03
$9.7M (76% M&PR 17% CP)
$18.2M (64% M&PR 17% CP)
$24.2M (55% M&PR 26% CP)
$32.5M (43% M&PR 33% CP)
$34.5M (41% M&PR 38% CP)
$36.6M (33% M&PR 40% CP)
$31.3M (28% M&PR 44% CP)
$28.6M (23% M&PR 35% CP)
M&PR - Marketing and Public Relations
CP – Community Programs
TEPP Evaluation History
•
1996-1998
•
•
•
•
•
1999-01
•
•
•
•
Adult and Youth Tobacco Surveys
Open Systems Process Evaluation
Media Tracking and Surveys
LP Assessments (adult cessation, prevention)
Adult and Youth Tobacco Surveys
Open Systems Process Evaluation End
Policy Surveys (Schools, Worksites, Ordinance
Tracking, Policymaker Perceptions)
2002-03
•
•
Adult and Youth Tobacco Surveys
TEPP Evaluation Unit
TEPP Evaluation Unit & ADHS
•
•
•
•
Semi-independent Unit
Addressing comprehensive
evaluation (Links among data)
Addressing all TEPP programs
Evaluation as a developmental
process
Perspective on Program Evaluation
Evaluation is a developmental process rather than a single
event or periodic report card
Program planning
Learning
Program implementation
Evaluation
Evaluability Assessment - Defined
“An evaluability assessment is a
process for clarifying program
designs and exploring a program’s
context in order to determine if it
can be meaningfully evaluated.”
(Wholey, Hatry & Newcomer, 1994)
Five-Tiered Approach
I
Needs Assessment
II
Monitoring and Accountability
III
Quality Review & Program Clarification
IV
Achieving Outcomes
V
Establishing Impact
Recognizes that programs, even at same site, are at different
levels of development
Organizes evaluation around these
developmental levels
Adapted from Jacobs et al., 2000
Allows Mapping to Evaluation Types
•
Needs/Resource Evaluation
•
•
Process Evaluation
•
•
Linking program to short-term changes
Impact Evaluation
•
•
Knowing what programming is happening
Outcome Evaluation
•
•
Knowing what programming is needed
Linking program to longer-term changes
Cost Analysis Evaluation
•
Are the results we are producing likely to be
cost efficient?
Evaluability Assessment
Answers these basic questions:
What level and type of evaluation best
suits program and stakeholder needs?
What evaluation
activities best support program enhancement?
Evaluability Assessment - Steps
•
•
•
•
•
•
Examined program documents
Reviewed past evaluations
Distributed pre-site visit
questionnaire (community pgms)
Conducted site visits
Statewide project interviews
Identified existing data sources
General Findings
•
•
Considerable variability in
programming across the state
Considerable amount of existing
data
#
Data Source
Name
Overview Description
Owner
Data
Who
Collection Collects
Dates
Data
How
Respondents
are Selected
Method of
Data
Collection
Where
Data
Reside
Disproportionate stratified
sampling, random digit dialing
that over-represents rural
areas. Respondents must be:
AZ resident, 18 or older;
speak English, and have a
phone. 1996 and 1999
surveys had English and
Spanish versions. There will
be a Spanish language survey
launched in 2003.
Disproportionate stratified
sampling using random digit
dialing. Residents, 18 years
and older, noninstitutionalized, have phones.
Computer Assisted
Telephone
Interviewing
(CATI)
ADHS-TEPP
Evaluation Unit
Computer Assisted
Telephone
Interviewing
(CATI)
ADHS
For 2003-The YTS survey
sample design is developed by
the CDC and is a 2-stage
cluster design. In the first
stage, schools are selected
randomly within the grade
range specified with a
probability proportional to
enrollment size. At the second
stage, classes are randomly
selected from within the
selected schools and all the
students within a selected
class are surveyed. Included
grades 6-12 in both public and
charter schools. Data are
weighted to provide state
prevalence estimates.
2003 survey
administered to
youth in schools.
1997 was a phone
survey of youth
ages 10-17. 2000
survey was
administered in
schools to grades
6-8, and 9-12 in
public and charter
schools. Grades 912 data were not
released due to low
response rate.
ADHS-TEPP
Evaluation Unit
has 2000, and
2003 data.
Survey
administered in
schools.
Individual
Arizona tribal
nations and
Inter-Tribal
Council of
Arizona
Surveillance
1
Adult Tobacco
Survey--ATS
Was initiated in 1996 by the Arizona
Department of Health Services (ADHS) to
collect detailed surveillance of adult tobacco
use and attitudes. The survey has since been
repeated in 1999 and 2002 to provide
evaluation data for the Tobacco Education
and Prevention Program (TEPP) through
continual monitoring of tobacco use among
Arizona adults.
ADHS
1996 (baseline)
1999, 2002. Every
three years.
Social Research
Laboratory at
Northern Arizona
University; ADHS
Telephone Survey
Center (1996, 1999)
2
Behavioral Risk
Factor Survey-BRFS
ADHS &
CDC
Annually since
1982
Private firm
contracted by ADHS
3
Youth Tobacco
Survey--YTS
A joint effort between CDC and ADHS to
collect annual data on health risk behaviors
of non-institutionalized residents 18 years of
age & older. Used to plan, implement, and
monitor health promotion and disease
prevention efforts in AZ. Core questions
include items about tobacco use.
Developed by the CDC for state surveillance
and initiated by ADHS in 1997 to collect
detailed surveillance of youth tobacco use
and attitudes. The survey has since been
repeated in 2000 and 2003 to provide
evaluation data for the Tobacco Education
and Prevention Program (TEPP) through
continual monitoring of tobacco use among
Arizona adults. Developed by the CDC for
state surveillance.
Arizona
Department of
Education;
ADHS/TEPP
1997 (baseline),
2000, 2003.
Every 3 years.
For 2003-ORDMacro contracted by
Arizona Department
of Education
4
Tribal Youth
Tobacco Survey
Developed by the CDC for state surveillance
specifically focusing on Native American
youth and initiated by ADHS and Inter
Tribal Council of Arizona to collect detailed
surveillance of Native American youth
tobacco use and attitudes.
Varies
Individual Arizona
tribal nations or
Native American
organizations with
assistance from
CDC.
5
Youth Risk
Behavior Survey-YRBS
2003—first time
Arizona
participated in
YRBS.
Conducted every
three years.
ORD-Macro
contracted by
Arizona Department
of Education
9th-12th grade in Arizona
public and charter schools are
administered the survey using
CDC sampling design
protocol.
Survey
administered in
public and charter
schools to 9th-12th
grade students.
Arizona
Department of
Education
6
Arizona Youth
Survey--AYS
(formerly the
Arizona Youth
Substance Abuse
Survey)
Developed in 1990 by CDC to monitor
priority health risk behaviors that contribute
to leading causes of death, disability, and
social problems among youth and adults in
U.S. Includes items about tobacco use.
Provides comparable national, state and local
data, and among subpopulations among
youth. Monitor progress toward achieving
Healthy People 2010 objectives. Voluntary
state participation.
Survey of 8th, 10th, and 12th grade students in
Arizona, to fulfill the legislative
requirements to conduct a survey to measure
both the attitudes and actual prevalence and
frequency of substance abuse by children
and adults. The survey is a partnership
between the Arizona Criminal Justice
Commission (ACJC) Arizona Department of
Health Services, and Governor’s Division of
Individual
tribal nations
in Arizona
and Native
American
Urban Health
Centers
Arizona
Department of
Education
(ADE)
Arizona
Criminal
Justice
Commission
(ACJC)
Biennial
administration.
Last survey
conducted in
2002.
Southwest Center of
Prevention at
University of
Oklahomacontracted by ACJC
Stratified random sample
based on school size, and a
weighting procedure to
accurately represent student
composition in 8th, 10th, and
12th grades. Schools can
voluntarily participate if not
selected for sample, but data
are not reported in official
Survey
administered in
public and private
schools.
Arizona
Criminal Justice
Commission
(ACJC)
Youth Access
Study that assesses the impact of the
CounterActs Program on tobacco merchants.
Maricopa County intervention site and Pima
County control site. Baseline and post survey
using cross-section sample to assess public’s
perceptions about youth access. Random
compliance checks conducted by law
enforcement agents and post-compliance
check questionnaire administered to store
clerks.
ADHS/TEPP
Attorney General
Compliance
Inspection Reports
A database that holds descriptive data about
each individual inspection and its outcome
that was conducted by the Attorney General
regarding merchant compliance with the youth
tobacco access law.
ADHS/TEPP
Synar Database
A database that holds information about the
outcome of the Synar inspections. By law,
states must conduct random unannounced
inspections to determine buy rate of tobacco
products sold to youth under the age of 18.
State rate must not exceed 20% otherwise
state may lose 40% of its substance abuse
block grant.
ADHS
Counter Acts Pilot
Study Data
2002 to 2003
Contractor for
ADHS supervised by
Eusebio Alvaro, U of
A.
For cross-sectional survey:
Random sample of adult
Maricopa County residents to
assess perceptions about youth
access to tobacco.
Attorney General
Special Agents
For compliance checks:
merchants and/or clerks who
were inspected.
Ongoing since
2002
Attorney General,
Special
Investigations
Section Agents
Decision-making and criteria
determined by Attorney
General’s office and
ADHS/TEPP.
Annual since
2000
Contractor for the
State to conduct
inspections
Pre-post crosssectional survey
Computer Assisted
Telephone
Interviewing
(CATI)
For compliance
checks: merchants
and/or clerks who
were inspected
given short
questionnaire to
complete.
Special Agent
completes the
inspection form
after each merchant
inspection.
Attorney
General’s office;
ADHS/TEPP
Evaluation Unit
Random,
unannounced
inspections of
retailers who sell
tobacco.
Attorney
General’s Office;
ADHS TEPP
Evaluation Unit
School Based Prevention & Other Youth Data
Information about Arizona Schools such as
resources that each school has; indicators of
risk at the school such as free/reduced lunch,
attendance rates, test scores, expenditures;
safety and healthy environment
Part of the Comprehensive Health Education
Standards Surveillance System (C.H.E.S.S.S.)
Purpose is to account for prevention dollars at
the site/school level, and is part of a collection
system to fulfill data reporting requirements to
state and federal agencies. Includes data
about tobacco prevention programming and
policy enforcement.
Assesses the status of school health programs
and policies among samples of middle/junior
high and senior high schools in CDC funded
states and selected cities. For example, what
health topics are covered, teaching methods
used, school policies on health-related topics,
such as tobacco.
To assess the outcomes of the TEPP Local
Project school based prevention programs.
Different surveys developed by various local
projects to assess changes in knowledge,
attitudes, intentions, and behaviors of program
participants who received intensive schoolbased prevention.
ADE
Arizona School
Tobacco Policy
Survey
Two surveys conducted of Arizona public
schools to document whether schools had
written policies or rules regarding tobacco
use. The 1998 survey was done prior to
passage of ARS 36-798.03 prohibiting
tobacco products on school grounds, in
buildings, etc. The 2000 survey was designed
to assess compliance with the new law.
ADHS/TEPP
Arizona Inter-
Data from outcome and satisfaction survey
ADHS/TEPP
Arizona Department
of Education –School
Report Cards
Arizona Department
of Education—Safe
and Drug Free
Schools Survey and
Report (formerly
CHAPPS)
Arizona Department
of Education—
School Health
Profiles (AZ SHEP)
TEPP Local Project
School Based
Prevention Pre/Post
Surveys
ADE
ADE
Annual since
1998
ADE, Research and
Policy Division
Administrative personnel at all
public and public-charter
schools must complete the
survey.
Web-based survey
hosted on ADE
website.
ADE
ADE
Annual since
1996?
ADE
School principal and lead
health educator at the sampled
middle/junior high and senior
high schools in Arizona.
Self-administered
questionnaires.
Voluntary and
confidential.
ADE
TEPP Local Project
Staff and/or School
Teachers.
Participants in TEPP schoolbased prevention programs.
Administered to
students in schools.
Individual Local
Projects. Some
data housed in
TEPP Evaluation
Unit (e.g. Pima
County,
Coconino
County
1998 and 2000
Arizona Cancer
Center, The
University of
Arizona
All public, alternative, and
charter schools in Arizona
Ongoing
AIA staff
All participants in the AIA
Computer Assisted
Telephone
Interviewing
(CATI). Two stage
calling: 1) to set
appointment with
appropriate person;
2) to conduct
interview
Pre-post program
Various TEPP
Local Projects
ADHS/TEPP
General Findings
•
•
Need for a phased approach to develop
meaningful evaluation plan
Steps required
•
•
•
•
Establish evaluation infrastructure
Create structured process evaluation to clarify
and quantify program
Identify appropriate outcomes at different
levels
Design reporting that supports program
planning
Evaluation Infrastructure
Components
•
Develop coordination model
Coordinating Evaluation Efforts with Local Projects
Outcomes
Input
Local
project
staff
Activities
Outputs
Consistent
communication with
local project staff
ADHS
TEPP
staff
Short-term
Reports and presentations
Increased understanding of
roles and responsibilities
in evaluation of comprehensive
tobacco control
Evaluation
Unit
Staff
Workshops
Education regarding
evaluation methods
and procedures
LP
plans
LP
Logic
models
Needs
and
resource
assessment
ADHS
TEPP
strategic
plan
Evaluation
Standards
Measurement
development
Data management
Data analysis
Long-term
Increased knowledge of
science-based practice for
tobacco control
Synthesis of scientific
literature
Links with
statewide
projects
Intermediate
Increased recognition
of evaluation
utility
Refining program
goals and objectives
Effective comprehensive
tobacco
control programs
Integration of evaluation
and program planning
and development
Training materials
Increased knowledge about
and skill in evaluation
methods and procedures
Positive partnership
between
evaluation unit
and local projects
Reduced smoking prevalence
Database development
Adherence to improved data
quality control procedures
Appropriate
data interpretation
Reduced tobacco-related
morbidity and mortality
Implementation
of
standardized
measurement procedures
High data quality
Decreased tobacco-related
disparity
Measurement tools
Evaluation Infrastructure
Components
•
•
Develop coordination model
Assess feasibility of centralized IT for
evaluation
Process Evaluation Components
•
•
•
Develop measures of program
quantity
Develop measures of program
quality
Coordinate with other state and
national efforts
•
•
Promote comparisons across state
Develop system with an eye
towards linking to outcome
Outcome Evaluation Components
•
Identify appropriate outcomes at different
levels
•
•
•
•
Use appropriate measures
•
•
•
•
Work from state and local program models
Use existing data where possible to build
historical picture
Examine proximal, rather than distal,
outcomes
Standardize instruments used
Align with national and other state instruments
where possible
Go beyond surveillance data alone as outcome
Use appropriate data analytic techniques
•
Incorporate multivariate models
Key Challenges
•
Recognizing and accepting appropriate level of
evaluation
•
•
•
Identifying appropriate outcomes
•
•
•
•
Developing complementary measures to supplement
surveillance data
Coordinating needs of local project evaluations with
broader, statewide evaluation
•
•
Process evaluation not as “sexy” as outcome
But need to know the “program” in program evaluation
local: short term outcomes, less rigorous, quick
turnaround
statewide: standardization of data elements and
collection procedures, rigorous design, intermediate to
longer term outcomes, and a relatively long timeframe
Creating (and retrofitting) evaluation systems
Longer process than anyone wants it to be
Key Rewards
•
•
Unpacking the “comprehensive program”
black box
Clarifying program planning
•
•
•
•
Supports clearer thinking about goals and how
attempting to reach them
Encouraging better understanding of how
programs are linked to outcome
Anchoring a baseline for evolution of
program and evaluation planning
Facilitating intra-state and interstate
communication
References
U.S. Census Bureau 2002, State and County Quick Facts.
(http://quickfacts.census.gov/qfd/states/04000.html; http://www.census.gov/statab/ranks/rank01.html)
Campaign for Tobacco-Free Kids, "A Broken Promise to Our Children: The 1998 State Tobacco Settlement Five
Years Later,” November 2003.
(http://www.tobaccofreekids.org/reports/settlements/2004/fullreport.pdf)
Jacobs, F., Kaouscik, J. Williams, P., & Kates, E. (2000). Making it Count: Evaluating Family Preservation
Services. Tufts University Press.
Wholey J, Hatry H & Newcomer K. (1994) Handbook of Practical Program Evaluation. San Francisco, :JoseyBass Inc.
For further information contact:
Jesse Nodora, Dr.P.H.
Evaluation Administrator
Arizona Department of Health Services
Tobacco Education and Prevention Program
1740 West Adams
Phoenix, Arizona 85007
(602) 364-0837 Fax (602) 364-0844
[email protected]
www.tepp.org
Or
Michele Walsh, PhD
Senior Evaluator
Tobacco Education and Prevention
Program Evaluation Unit
The University of Arizona
PO Box 210462
Tucson, Arizona 85721
(520) 318-7259 x 123
[email protected]