SURVEY DESIGNS PPT - LISA

Download Report

Transcript SURVEY DESIGNS PPT - LISA

SURVEY DESIGNS AND ANALYSIS
BY
BENEDICTO KAZUZURU
WHY SURVEY DESIGNS?
Data
Surveys
Knowledge
Understanding
Decisions
Survey Design
The key steps of survey design
1. Cleary define your research problem
2. Clearly define your population of study
3. Develop a sampling plan
• Sampling methods
• Sample size calculation
• Data capture methods
4. Designing a questionnaire to minimize bias and errors
5.Pretest of the questionnaire
Step1:Clearly defined research problem
(i) State the overall research goal
(ii) State the specific questions
Need to ensure that the research goals are reflected at
every stage of the design as well as their specific
questions. For example the overall research goal could
be
“ Assessing the wealth status of the inhabitants around
the three altitudes of mount Kilimanjaro”
The specific questions could be:
• What is the average value of the assets of an individual
household
• What is the average annual income of an individual
household
Step 2: Define the population to be sampled
Who will you interview to answer your research questions?
Who has the answers to your questions?
• Subject: Any object we measure
Plants, persons, pupils, etc.
• Sample: subset of the
population to be measured (i.e.,
a group of subjects that
represent the population)
• Population: representation of all
the possible outcomes or
measurements of interest.
Step2:More Concepts on Population
• Target population: A complete collection of all the observations we
want to study . It is always hard to define a target population.
Sampled population: The collection of all possible observation units that might
have been chosen in a sample. The population from which a sample was taken
• Observation unit/An element: An object on which measurement is taken. In
studying human populations Individuals are the observation units
• Sampling unit: The unit we actually sample. For example our observation unit
could be individuals in a community, but we actually do no have a list of them
and consequently a household becomes a sampling unit
• Sampling frame : A list of all the elements/units of a population
Step 3: Develop a sampling plan
• Once the target population has been identified, next the sampling plan
must be devised
• Goal: Randomly select a small percent of the population that will in
turn represent the ideas of the population as a whole.
• The sampling plan involves:
A. The technique used to select the subjects for your study
Probability sampling
•
•
•
•
Simple random sampling(SRS)
Systematic sample
Stratified random sampling
Cluster sampling
Non Probability Sampling
Convenience, purposeful, quota, snowballing
B. The number of people needed for your study
• Sample Size Calculations(Sample enough for precise results )
C Methods of Data Capture
Simple random sampling
• Each subject has an equal chance of being in the study
• With population of size N, and sample of size n there  nN  different
 
possible samples without replacement
• Easiest to summarize BUT most tedious to implement in the field
Example:
Randomly select 10
students from the Stat 3005
class roster to ask a
question.
Systematic sampling
• Choose a selection interval k=N/n where N is the population size and n is the sample size
• Select the first subject at random from the first k subjects in a given list of all possible subjects(sampling frame)
• Select every kth subject from a list of all possible subjects counting from the first subject until n subjects are
reached
Example:
Telemarketers randomly
sample every 10th phone
number on the Yellow
Book to make marketing
calls.
Systematic sampling
•
•
•
•
•
Advantages over simple random sampling
It is easy and simple to implement
It gives assurance that the population will be evenly sampled
Disadvantages over simple random sampling
If it happens by chance that the process of selection of an item
coincides with a hidden periodic trait within the population
then the sample is no longer a random sample.
For example in a household survey seeking individuals’ opinions
on crime activity in their community you have sampling interval
k=1000 such that after every 1000th houses there is a police
station. In such a scenario the sampled individuals could give you
biased information regarding crime activity.
Stratified random sampling
• Both simple random sampling and systematic sampling assumes that the
population contains homogenous units which is not necessarily the case
• First divide population into strata (groups) based on similarity
• Strata are subgroups in a population which are of interest to the
researcher .For example “ a survey to investigates students’ opinion on
students’ mass failure in a University”, different degree programs or
different years of study could form different strata.
• Then randomly select subjects within each strata
• For example consider a population with 40,000 adults in which there
are 30,000 males and 10,000 females and one seeks individuals' opinions
on the use of contraceptives
• If the proposed sample size is 400. Then the allocation would be such
that:
• 300 males out of 30,000 males and 100 females out of 10,000 females.
• This is called proportional allocation
Stratified random sampling
• Advantages over simple random sampling
• A stratified sample can guard against an
"unrepresentative" sample (e.g., an all-male sample
from a mixed-gender population).
• A stratified sample can provide greater precision than a
simple random sample of the same size.
• Because it provides greater precision, a stratified
sample often requires a smaller sample, which saves
money.
• We can ensure that we obtain sufficient sample points
to support a separate analysis of any subgroup
• We could apply different methods of data capture in
different strata.
• Disadvantages over simple random sampling
• The main disadvantage of a stratified sample is that it
may require more administrative effort than a simple
random sample
Cluster sampling
• Simple random sampling and stratified sampling assume that
a sampling frame exists, which is not always the case.
• A population can have many clusters
• First randomly select a number of clusters
• Then sample a units within each cluster
• Require clusters to be representative of population
Example 1: Finding students opinions
at Virginia Tech
Population: All students (attending class)
at Virginia Tech
1. Randomly select a certain number of
classes
2. Ask all students in each class their
opinion
This is called stage one cluster sampling
Note: Cluster sampling is often NOT as
*
Cluster sampling
• Example 2: Consider a survey on opinions by Catholic
church members in USA
• No list for all church members exists
• One could considers the catholic churches as clusters
• Take a random sample of the churches in USA
• Then interview all the members within in each churchthis will be a case of first stage cluster sampling
• Alternatively he could interview some few members
within each selected church using for example simple
random sampling –this would be the case of two stage
cluster sampling
Cluster sampling
• Advantages over stratified random sampling
• Can be done even if there is no sampling frame as long as the
clusters exist or can be created
• It is easy, cheap and quick much as it involves fewer
groups/strata/clusters
• Disadvantages over stratified sampling
• If the studied clusters are very limited may not represent the entire
population which may lead to large sampling error.
Multistage cluster sampling
• This is an extension of single and two stage cluster
sampling
• It can be done for several stages
• For example one could consider several nations and
pick one
• Then consider several regions/states and pick one
• Consider several districts/counts
• Consider several divisions and pick one
• Consider several ward and pick one
• Consider several villages/streets and pick one
Non probability Sampling
•It can be used when the research does not aim to generate
results that will be used to create generalizations pertaining
to the entire population.
•It is also useful when the researcher has limited budget,
time and workforce.
•This technique can also be used in an initial study which
will be carried out again using a randomized, probability
sampling.
Convenience sampling
• In here you sample what is available. For example
in hospitals or in clinics the arriving patients could
all be included in the sample
• The only problem here is that you are unsure
whether those individuals do really represent the
entire population
Purposive sampling
• This is a deliberate selection of individuals of particular type.
For example someone studying cattle diseases could
deliberately go to a particular region where people keeps cattle.
Advantages
• Saves, time, money
Disadvantages
• The problem here again is that your sample may not be a
representative of the entire population and most likely you may
end up with a selection bias problems
Snowball sampling
• In snowball sampling, you begin by identifying someone who meets the
criteria for inclusion in your study. You then ask them to recommend
others who they may know who also meet the criteria
• Advantages
• It could be the only possible method in circumstances where your
research issue is a sensitive one in a society.
• Disadvantages
• It faces the same problem that the individuals you interview may not be
the representatives of the population.
Quota Sampling
In a stratified sampling when one picks units from
each stratum instead of using simple random
sampling he could just pick by convenience
• You could have either proportional quota sampling
• Non proportional quota sampling
In either way the pick is by convenience and through
a random procedure
Sample size calculation
• There many ways to get a sample size
• Essentially it requires three key inputs
i. The level of significance
ii. The margin of error
iii. The standard deviation of the measured quantity
• The standard deviation of the measured variable could
be obtained through pilot survey or through previous
study similar to the current study
Sample size calculation for a proportion
• Consider a proportion P whose estimate is pˆ
•
pˆ (1  pˆ )
ˆ
p

z
• (1-α)% confidence interval for P is
 /2
n
•
•
•
•
•
pˆ (1  pˆ )
z
Solving for n we get n= z pˆ (1  pˆ ) where d=
n
d
  5% so that z  1.96
For example if d=0.05, and pˆ  0.5
n becomes 384.
d is called the margin of error which decreases as n get large
Practically with n=1000 or more d remains more or less constant
2
 /2
 /2
2
 /2
Methods of Data Capture
Methods of Data Capture
• After we know what data we want to collect, who we want to survey,
and how we ask the questions we must determine the best instrument
for collecting the survey data
• Data collection options:
Personal Interviews—either paper or PDA/Smartphone
Telephone Interviews
Mail Surveys
Email Surveys
Measurement
Observation
Secondary sources
• For more discussion of data collection options see
http://www.surveysystem.com/sdesign.htm
Personal interviews
• A face-to-face encounter between the interviewer and the
subject
• Advantages:
• People usually respond when confronted face-to-face
• Can get a better sense of the reaction of the subject
• Prevent misunderstandings
• Disadvantages:
• More costly
• Interviewers who are not trained properly may introduce
bias into the sample
Telephone interviews
• Most popular instrument for survey in the United States since
most of homes have telephones
• Advantages:
– Less expensive than personal interviewing
– Random phone numbers can be dialed
– Fast results
• Disadvantages:
– People are reluctant to answer phone interviews
– Phone calls can usually only be made from around 6pm-9pm
– Phone surveys normally need to be shorter in length than
personal interviews
– Can rarely works in developing countries
Mail surveys
• Advantages:
– Cheap
– Questionnaire can include pictures
– People are able to answer on their own time
• Disadvantages:
– It may take time for respondents to respond
– Response rates have a tendency to be low
– It may take time to compile the responses
Email surveys
• Advantages
• Cheap
• Fast
• You can attach pictures or sound files
• Disadvantages
• People may respond multiple times
• People who have email may not be representative of the
population as a whole-especially in developing countries
Measurements
• Advantages
-could work even if the population consists of
insensible human beings (e.g. measuring children
health under five years)
-Likely to provide the most accurate results
Disadvantages
-most expensive method
-Might also be harmful(e.g testing a drug quality)
Secondary Sources
• These are readily available information collected by institutions such as
national bureau of statistics, ministries and scholars, often found in
libraries, internet, and in books.
• Advantages
• -Easy to do it
• Less costly in time and money
• Disadvantages
-may be hard to find the exact information needed
Step 4: Designing a questionnaire to
minimize biases and errors
Three major types of biases and errors:
1. Selection bias or coverage error:
This would occur when part of the target population is not in
the sampled population
- e.g. target population could be all eligible voters while sample
population could be a population reachable by telephone.
2. Nonresponse bias:
This occurs when one fails to get responses from all the chosen
sample. Often case those who respond to your survey are
different in important ways from those who choose not to
respond.
3. Measurement error: Inaccurate answers to survey questions
(sometimes due to lack of clarity in writing).
Possible causes of selection bias
• Misspecfying the target population. For example in presidential
opinion polls what should be the target population? Eligible voters?
Registered voters? Voters reachable by telephone?
• Deliberately or purposeful selecting a representative sample. For
example in estimating students scores at Virginia Tech you choose say
students in BSc. Agriculture.
• Substituting a convenient member of a population for a designated
member who is not readily available.
Possible causes of non response bias
• Respondents not understanding how important is
the survey
• Lack of incentives by the respondents to respond
• Boring questions
• Too long a questionnaire
Dealing with non responses
• Keep the questionnaire as short as possible
• Categorize questions into 3 groups:
– Must Know
– Useful to Know
– Nice to Know
• If the questionnaire seems too long, start omitting the “nice to
know” questions.
• Don’t keep asking questions that are not pertinent.
• Explain the importance of the study to the respondents in the
beginning of the questionnaire
• Provide the incentives to the respondents. For example in Tanzania
in some rural surveys a matchbox/a packet of salt is provided to a
respondent.
Possible causes of measurement errors
• People sometimes do not tell the truth. In agricultural surveys farmers in areas
with food aid programs may under report crop yields
• People sometimes do not understand the questions
• People sometimes forget . For example asking a farmer’s yields in the past five
years
• People respond different to different interviewers. For example a female and male
interviewers asking a female respondent “ do you think men could be trusted?
• An interviewer may affect the accuracy of the response by misreading questions or
record responses inaccurately.
• Certain words could mean different things to different people. For example, in a
household survey a question such as do you own a car, may have YES or NO
answer depending on a respondent’s understanding.
• Questions wording and ordering may have a large effect on responses obtained.
Dealing with measurement error bias
• Train the enumerators before going to field about how
to ask questions and presenting themselves
• Provide your enumerators with necessary tools for
recording the data and assessing answers’ accuracy
• Explain the importance of the surveys to the
respondents
• Carefully check the questions' wordings and order of
the questions
• Write clear, concise questions.
• Be aware of social factors that may influence
responses.
• Check also on the types of questions(open
ended/closed ended questions)
Types of questions
1. Open Ended
• Designed to prompt the respondent to
provide you with more than just one or two
word responses.
• These are often How or Why questions.
• Drawback: It’s harder to compile their
results.
2. Closed Ended
• Also known as forced-choice questions.
• Specific questions that elicit YES or NO
responses.
• e.g. Do you eat apples?
• Easier to analyze.
Type of Questions Cont’d
3. Multiple Choice
• Allow respondents to choose one or more answers from a few
possible choices.
• Elicit more details than close-ended questions.
• Results can be compiled more easily than in open-ended questions.
4. Likert Scales
• Respondents are asked to rate items on a response scale.
• You might sum a respondent’s ratings for all of the items.
• Can be done without a neutral category.
e.g. The police have done enough to prevent crime in Virginia.
Strongly Agree() Agree() Strongly Disagree() Disagree()
More Tips on Questionnaire Design
• Always include preambles.
• It should contain what you want to do and why.
• Address sensitive questions as discretely as possible.
• e.g.: Are you infected with any STD?
• Avoid words that provoke bias or emotional response.
• e.g.: Why do you believe in gay marriage?
• Place similar questions together logically.
• Avoid putting too much into a single page.
More Tips on Questionnaire Design
• Ensures that respondents will not feel that they are losing their
anonymity when answering the rest of the questions.
• Choose the most important questions for your survey to be asked
at the beginning of the survey.
• Ensure anonymity as much as possible.
• Refrain from having two concepts embedded in one question.
Example:
“Do you have time to read the newspaper every day?”
• Notice you are asking about “time” and “reading the newspaper every day”.
Revision:
“Do you read the newspaper every day?”
• If the answer is no, you can create a question to determine the reasons the
person does not read the newspaper
Step 5: Pilot test (and retest)
your survey
• You should pretest the survey on a smaller sample
whenever possible!
• This pilot test can:
- Allow you to revise the questionnaire if needed.
- Allow you to create a closed question from the responses for an
open question.
- Help you estimate the variability in the responses to your
questions and determine the necessary sample size.
Analysis of Survey data
• Survey data must be processed, analyzed, and
reported
• Code or input your data onto a computer
• Clean your data—start with the most important
variables:
1. Ensure all data are in the correct format.
2. Decide what to do with missing data or outliers.
3. Detect outliers and coding errors by visual or graphical
inspection.
The quickest statistical analysis is often just a plot or
graph of your data
• Summarize your data one variable (e.g. height) at a time.
Histograms show the distribution of the data points
4
3
2
1
0
Number of respondents
Histogram of Height
10
20
30
40
50
Height in inches
60
70
Plot two quantitative variables
on a scatter plot
• The relationship between two quantitative variables can be
visualized in a scatter plot and quantified by correlation or
regression.
Scatter plot of Height and Weight
220
180
●
●
140
●
●
●
●
●
●
●
●
●
100
Weight in pounds
●
●
●
●
●
●
50
55
60
65
Height in inches
70
75
Some Tips
• Different statistical procedures are appropriate for different types
of data (more on next slides).
• Questionnaire derived data are generally likely to require nonparametric techniques (does not require normality assumption).
• Exceptions exists when you ask the respondents to fill in their
height, weight, scores, or income (continuous variables).
• The mean of a categorical variable is meaningless!
• Instead, use mode, frequency tables, and cross tabulations to
summarize categorical or ordinal variables.
• You can also use bar charts or pie charts!
What kind of data do you have?
Data Type
Description
Examples
Summary
Statistics
Nominal
Data with no intrinsic
relative meaning behind
labels
Black, White,
Hispanic
Mode
Ordinal
Data with an ordered
structure
Small, Extra
Large, Likert
Scale*
Median and
Percentiles
Interval
(continuous
or discrete)
Data with meaningful
difference relations
Degrees in
Celsius,
Birthdates, GPS
Coordinates
Mean, Standard
Deviation,
Correlation
Ratio
(continuous
or discrete)
Data with scale relations
Weight, Income,
Length
Mean, Standard
Deviation,
Correlation
Example of data exploration using SPSS from a
survey on commercialization of maize farmers in
Tanzania
• Histogram of rate of commercialization
• Boxplots of farmers commercialization by sex
• Scatter plot of commercialization rate against age
Histogram
Histogram of Maize Commercialization
40
30
20
10
Std. Dev = 28.05
Mean = 44.3
N = 212.00
0
0.0
20.0
10.0
40.0
30.0
60.0
50.0
80.0
70.0
100.0
90.0
commercialization of maize by ahousehold
Maize commercialization by gender
120
100
80
60
40
20
0
-20
N=
47
162
Female
Male
sex of the head of the household
Scatter plot
Maize Commercialization Versus Age
120
100
80
60
40
20
0
-20
0
20
40
Age of the head of the household
60
80
100
References
• Dillman, Don A., Jolene D. Smyth, and Leah Melani Christian. Internet,
Mail, and Mixed-Mode Surveys: The Tailored Design Method. 3rd ed.
Hoboken, NJ: John Wiley & Sons, Inc, 2009.
• Lietz, P. (2010) Research into Questionnaire Design. International Journal
of Market Research, 52, 2, pp. 249-272.
• Lohr, Sharon L. “Sampling Designs and Analysis”, International
Thompson Publishing Company
• Scheaffer, Richard L., William Mendenhall III, and R. Lyman Ott.
Elementary Survey Sampling. 6th ed. Belmont, CA: Duxbury, 2006.
•
•
•
•
•
http://en.wikipedia.org/wiki/Likert_scale
http://www.surveysystem.com/sdesign.htm
http://www.csudh.edu/dearhabermas/sampling01.htm
http://www.youtube.com/watch?v=53mASVzGRF4
LISA, http://www.lisa.stat.vt.edu/, Eric Vance short course.