Transcript Sampling Decisions in your TC Program Evaluation

Sampling Decisions
in Your TC Program Evaluation
TCEC
Overview








Why sampling?
Terms
Sampling methods
Sample size
Sample size and analysis
Answers to program questions
Resources
Announcements
Why sampling?



Evidence from data drive the
program
Census (collecting data from each
entity such as stores, tenants,
population) may not be possible
A representative sample can do
the trick
Terms



Population = Entire group from
which to collect data (e.g. county
population; stores in a certain
area, all city parks, etc.)
Census = all members of the
population
Sample = a portion of the
population
More terms


Sampling unit = the objects,
people, timeslots, etc. that are
being sampled (stores, residents,
casino patrons, etc)
Sampling frame = How you derive
your sample (phone book, street
corner traffic…)
Sampling Methods





Simple Random
Stratified Random
Cluster
Convenience
Purposive
Simple Random
Each member of
the population has
the same chance of
being in the sample
Stratified Random
A sample of equal size is drawn from
different sub groups of the population
Use example:
TRL: different neighborhoods that
have different foreseeable
consequences if TRL has passed –
you want equal representation
of all groups
Cluster
Census of
All members of
the population in
a group, e.g.
all tobacco retailers
A
B
C
A = rural
B = urban
C = college town
Purposive


Who or what you sample depends on
the purpose of the use for the data
Examples



KIIs generally
POP: only those who smoke at the fair
Observation: only parks with tot lots
Convenience


Reason: limited time and capacity (give
reason)
Examples:



Intercept public opinion surveys
YTPS only when youth are available to do stings
Only housing residents who come to a housing
association meeting
Sample Size



Sample size = The number of population
members you will use to collect data
Confidence level = The level of certainty
(usually set at 95%) that your sample
represents the whole population
Confidence interval = The percentage of error
you expect in your results
Population Confidence
size
level
150
150
150
95%
95%
95%
Confidence
interval
10
6
4
Sample
size
59
96
120
Question
What is the minimum POS sample size to
make statistically valid decisions for a
population of 1.8 million?
Go to http://surveysystem.com
Research Aids
Sample Size Calculator
Sample size calculator
Determine Sample Size
Confidence Level: _x_ 95% __99%
Confidence Interval: 5 (2)
Population: 1,800,000
Sample size needed: 384 (2398)
What it means for analysis
Example
Survey Question:
Please say if you agree or disagree with the following
statement:
“Smoking can shorten a person’s life.”
__ Agree
__ Disagree
Let’s say 75% of those asked said “agree.”
Analysis at confidence interval 5: We can say with 95% certainty
that between 70 and 80 % (75 plus/minus 5) of the population
in the county agree that smoking can shorten a person’s life.
Analysis at confidence interval 2: We can say with 95% certainty
that between 73 and 77 % (75 plus/minus 2) of the population
in the county agree that smoking can shorten a person’s life.
How to report
In evaluation plan
A public opinion survey will be conducted with a sample
size of 384 county residents randomly selected from
the phone book (or “through a convenience
sample”), for a 95% confidence level and a + 5 %
confidence interval for a total of 1.8 million county
residents.
(Note 1: you have to start out with a bigger sample size since a large
percentage will decline to participate)
(Note 2: if you analyze sub-groups of your survey, let’s say an ethnic
groups’ collective responses, your sample size is much smaller)
Determining statistical power
afterwards
Resource:
http://www.greatbrook.com/survey_statis
tical_confidence.htm
Requirements
Q: Do I need to do sample size calculations?
Reviewers like to see it but will not require it.
An estimated sample size is acceptable if it is
at an “acceptable” rate, with acceptable
referring to reasonableness with regards to
available resources (in this case “200” will
usually satisfy reviewers).
Other sample sizes (ss)
KII policy makers: ............(5-6)
 POS w/MUH residents: ……use calculator/ estimate ss
 Observation people
at event:……………………………estimate total event
participants/ calculate or
estimate ss
 Tobacco litter …………………See “Sampling Plan” in Tips
and Tools # 8, Observation

Representativeness
Sampling method determines
representativeness
Less likely in
Convenience sample (e.g. intercept
survey)
More likely in
Random samples
Question
How do you deal with small sample sizes
in a survey?
If the survey is not yet completed – add
to the sample
If survey is completed – report results,
add limitations explanation; acceptance
of results is still quite possible
Planning versus Analysis
What to focus on during
Planning
“population” size, sample size, sampling
method,
Analysis
Representativeness, limitations
Resources
TCEC website:
 Tips and Tools # 8: Observation
 Tips and Tools # 9: Sampling Decisions
 OTIS project plans and reports
Individual assistance:
 Call TCEC at 530-752-9951
 E-mail at [email protected]
Announcements



New Evaluation Associate
TCEC will be on facebook
A webinar on Cost-effective TC
Evaluation this Thursday at 10 (same
phone number and website)