Transcript Journey of a Survey – Roadmap to Usable Data

Frequently Asked
Evaluation Questions
Tobacco Control Evaluation Center
Jeanette Treiber, Robin Kipke and
Travis Satterlund
April 28, 2011
Where Do We Start?
Each evaluation activity is derived from the plan:
 Objective plan type  outcome/process
data
 How will data be used to move the objective
forward?
 Who should the data inform or convince?
 Map out what pieces of information are
needed, and how best to collect it (method,
from whom, where, when)
2
What Is End-use Strategizing?
 Identify your evaluation goals
 Know the kind of information needed and
specific purpose it should serve
 Understand the best source for your data
 Figure out when the data should be collected
 Recognize what type of method is most useful
 Decide an appropriate sample size
3
How Can We Use Our Data?
Your data can inform project staff by:
 Measuring how your message is being perceived
by various stakeholders (subgroups)
 Indicating whether further community education
efforts are needed
 Assessing which policy options to pursue
 Identifying where your campaign might encounter
opposition
 Signaling that perhaps it is not yet the right time to
pursue policy
4
Are There Other Uses For Our Data?
Through a variety of reporting strategies,
programs can use data to:
 Indicate public/stakeholder readiness for a
policy to decisionmakers
 Generate “buzz” by sharing results in
meetings, presentations, fact sheets, press
releases
 Build a groundswell of support and
motivate more people to join the effort
 Counteract critics
5
Should We Create New Instruments?
Adapt existing instruments (based on end use
strategizing):
 Not always necessary to reinvent the wheel
 Adapt other data collection instruments for
your project’s purposes
 Start with high quality instrument
 Add questions you need data on
 Delete unnecessary questions
 Make sure it will collect what you need
6
Where Can We Find Instruments?
 TCEC website has model instruments on
limited number of objectives
 Contact TCEC to access repository of
instruments created/used by other projects
 Ask other projects working on similar
objectives to share their instruments
7
What Other Sites Have Instruments?
Download instruments from statewide sites
 CA Health Interview Survey –
http://www.chis.ucla.edu/questionnaires.html
 CA Healthy Kids Survey –
http://www.wested.org/cs/chks/view/chks_s/17?xlayout=surveys
 CDC –
http://cdc.gov/tobacco/data_statistics/surveys/
 YTPS on STORE Website –
http://www.tcsstore.org/stages/1_document/t_yps.
pdf
8
How Can We Reach a Large Sample?
 Utilize high traffic venues/locations
 Back-to-school packets
 Church service programs
 Grocery store flyers
 Classroom teachers
 Entertainment tickets (e.g. movie theaters)
 Organizational newsletters
 Fairs and local events
9
What Are Other Distribution
Avenues?
 Online surveys (SurveyMonkey)
 Invite participation via email, social
networking media (Facebook, Twitter)
 Your organization’s website (and those of
coalition members, agencies that serve
target population)
10
What Are the Pros of Online Surveys?
 Cost effective
 Faster delivery
 Quick response time
 Better at addressing sensitive issues
 Ease of layout design and analysis
11
How Can We Increase Response Rates
for Online Surveys?
Some tips for maximizing the response rates:

Request participation in advance

Include a cover letter

Allow sufficient time frame to complete the survey

Provide survey instructions

Design a clear survey

Send reminders to those who still have not completed the
survey

Offer incentives for completion
12
Do We Need to Pilot Test the
Instrument?
 Evaluate instrument’s effectiveness
 Get a sense of how long it takes to complete
 Create checklist of concerns to address with
the pre-test




Do respondents/data collectors understand the
instrument?
Is the wording clear to all segments of sample
population?
Do any items produce irritation or embarrassment?
Have important issues been overlooked?
13
What Should Our Key Informant
Interview Protocol Look Like?
 Identify your goals beforehand
 Know the kind of information needed and
specific purpose it should serve
 Create a structure to your interview
 Probe, probe, probe (write probes into your
interview protocol)
14
What Kind Of Outcome Data Do We
Need For A MUH Objective?
Depending on the objective:
 Observed presence and placement of signage
 Observation data of smoking and tobacco litter
 Confirmed smoke-free language in leases
 Tabulated count of signed leases with smoke-
free language
15
How Can We Tap Into Volunteer
Labor?
 Develop relationships with local universities,
high schools, programs or clubs
 Recruit students looking for research
experience to do data collection
 Offer training, internships, course credit,
other incentives
 Entice coalition members to join in the
activities
16
How Do We Make Use Of Our
Volunteers?
 To field test instruments/materials for
appropriateness to data sources
 To translate instruments/materials into other
languages
 To collect data
 To provide access to other distribution points
17
How Do We Write Up the Results
From Our Data Collection?
 Summarize as you go
 Write “mini-analyses”
 See if the data can help you meet your
objective
 Incorporate your summaries and analyses
into your reports
18
How Can We Utilize TCEC’s
Resources?
 Online coalition satisfaction survey administration
and analysis service
 Additional model data collection instruments on a
variety of topics will be developed in coming
months
 Model instruments translated into other
languages (Spanish, Vietnamese, Hmong and
Russian)
 We’re working with other statewide projects to
develop tools that will help you use and share
your data more effectively for your policy work
19
When Should We Contact TCEC?
For individualized assistance with:




evaluation planning
development of data collection instruments
advice on analyzing & interpreting data
guidelines for reporting & sharing info
20
Resources You Can Tap
 TCEC archive of data collection instruments
 Evaluation associates at TCEC who can advise




you on developing instruments
Recorded webinars on survey development,
sampling decisions, conducting POS
Tips & Tools on creating interview questions,
sampling and end-use strategizing
Electronic quarterly newsletter
Your colleagues working on similar objectives
21