Transcript Surveys! - University of Northern Iowa

 What are the differences between surveys, interviews,
scales, and questionnaires?
 What makes surveys great?
 Determine content and purpose of question
 Choose the response format
 Figure out how to word it
 Figure out where to put it
 Pilot test!
 Ask for feedback from participants (at least have a
comments box)
 Is the question needed? At that level of detail?
 Is there a double-barreled question? Do you need to




ask more than 1 question?
Do p’s have the info needed to answer the question?
Do you need to be more specific or more general?
Are there biases in the question?
Will people answer the question honestly?
 Hypothetical projective respondent
 I feel completely secure in facing unknown new
situations because I know that my partner will never
let me down.
 When would you want to use dichotomous vs.




nominal, ordinal, or interval data? Advantages and
disadvantages?
In a Likert scale, how many options should you have?
How should they be set up?
Should there be a neutral point?
How can the responses you offer affect results?
 When would fill-in-the-blank be good or bad?
 When should you give multiple options?
 Do you have all the alternatives without going into too
much detail? At what point do you quit adding
categories?
 When are unstructured response formats good?
How often do you exercise?
Infrequently 17%
Occasionally 48%
Often
35%
In the last six months, how often have you engaged in at least 20 minutes of
aerobic activity?
Almost never
17%
Less than 1x/week 13%
1x/week
12%
2x/week
15%
3x/week
4x/week
>4x/week
15%
15%
13%
 Will people misinterpret the question?
 Does the question include assumptions or need a time






frame specified?
Does the question fit the population you’re sampling?
How personal is the wording?
Is the wording too direct or not direct enough?
Are there words people wouldn’t know?
Are the alternatives clear?
Is the wording unbiased?
 in 2005
 When I graduated from college
 When I was 18
 In the past 30 days, were you able to climb the stairs
with no difficulty?
 On days when you drink alcohol, how many drinks do
you usually have?
 How many miles are you from the nearest hospital?
Survey wording
• Don't you agree that campus parking is a
problem?
• There are many people who believe that
campus parking is a problem. Are you
one of them?
• Do you agree that campus parking is a
problem and that the administration
should be working diligently on a
solution?
• What do you think about parking?
 n
I oppose raising taxes.* (willing to pay)
I would be willing to pay a few extra
dollars in taxes to provide high-quality
education to all children.
The primary task of the government
should be to keep citizens safe from
terrorism and crime.
The primary task of the government
should be to preserve citizen’s rights and
civil liberties.
2.88
4.82
3.81
4.50
I regularly perform routine maintenance
on my car.
Sometimes I don’t change the oil in my
car on time.*
I make it a practice to never lie.*
4.07
3.27
2.65
Like all human beings, I occasionally tell 4.63
a white lie.
Monogamy is important to me.
5.02
Sexual freedom is important to me.
4.43
People should wait to have sex until
they are in a committed relationship.
3.97
Sex can strengthen a new relationship.
2.40
My partner and I always use protection. 4.08
Although I know it is important,
sometimes I don’t practice safe sex.*
5.22
 Do you approve of the government temporarily taking
over major banks in danger of failing?
54%
approved
 Do you approve of the government temporarily
nationalizing major banks in danger of failing?
57% disapproved
Another polling example
In any health care proposal, how important do you feel it is to give
people a choice of both a public plan administered by the federal
government and a private plan for their health insurance--extremely
important, quite important, not that important, or not at all
important?
77% extremely or quite important
•
Would you favor or oppose creating a public health care plan
administered by the federal government that would compete
directly with private health insurance companies?
•
43% favor













opinioncenter.com
1. If a bottle of water costs you three times more, would you continue buying it?
2. If not, what would you do?
3. Are you aware of the steps use to process a generic bottle of mineral water?
4. Do you believe that bottled water can be more expensive than oil?
5. When travelling overseas, do you find it easy to find your preferred bottled water
brands?
6. How much importance do you give to drinking water on daily basis?
7. How concerned are you that there may be water problems, including water
shortages, around the world?
8. Please specify any global water issues or concerns you are aware of?
9. Have you taken any steps to help alleviate any water problems around the world?
“The student makes the statement that Darwin killed religion”.
“The student makes the statement that finding out whether religion is real is not a big
deal in their life”.
‘The Store’ represents good value for the money.”
 What should be early vs. late in the survey?
 What else affects placement?
 What about layout issues?
 Thank them to start
 Keep it short
 Be alert to discomfort
 Thank them at the end
 Send a copy of the results if they want them
 What is the role of the interviewer?
 How should they be trained?
 Should they be “blind”?
 Any examples of good/bad interviewers?
 What should an interviewer bring/do?
 How do you decide which person to interview?
 What are good techniques for
 Getting entry
 Introduction
 Explanation of study
 Asking questions
 Probing for more information
 Recording the interview
 Ending the interview








Mail survey
Group-administered questionnaire
Household drop-off survey
Electronic survey
Focus group
Telephone interview
Face-to-face interview
Computerized options:






CATI
CAPI
ACASI
IVR
Online panels
Knowledge networks
 Mixed mode
 Population issues
 Sampling issues
 Question issues
 Content issues
 Bias issues
 Administrative issues
 Table 4-1
 http://fivethirtyeight.blogs.nytimes.com/2012/11/10/w
hich-polls-fared-best-and-worst-in-the-2012presidential-race/
 Who paid for the survey
 Who are the participants
 What was the sampling frame and method
 How were the questions worded
 What is the margin of error
 Coverage error
 Nonrespondent error
 Measurement error
 Processing error
 Should you have one or multiple pages?
 Should you have a line telling people how far along
they are?
 What other things may affect responses?
 Can you get representative samples?
 30s-60s
 Developed methods
 60s-90s
 Phone interviews, research on response rates, missing
values
 90s Harder to get respondents
 Computer methods
 Future?
 Big data
 What are advantages and disadvantages of surveys
overall?
 How can the disadvantages be dealt with?
 How have cell phones affected surveys?
 How can you deal with unlisted numbers?
 What are issues with RDD?
 Mitofsky-Waksberg method
 List-assisted
 Why are response rates decreasing?
 What are reasons for nonresponse and how can you
deal with them?
 Call back rates?
 How can survey researchers increase rates?
 Are lower response rates a problem? When?
 When is missing data likely to be a problem?
 What are the different types of missing data you can have?
 MCAR vs. MAR vs. MNAR
 In reality, a continuum between MAR and MNAR
 What problems do they cause?
 What are the “old” methods and when are they okay?
 Listwise deletion
 Pairwise deletion
 Mean substitution
 Missingness dummy variable
 Regression-based single imputation
 Sidebar: What is an eigenvalue and where would you
find one?
 EM algorithm (expectation maximization)
 Goes through values one at a time. If there is a value, it’s
added to the model. If not, then the best guess based on
predicting it in regression with all the other variables is
put in. This continues until it becomes stable.
 Good for: mean, variance, covariance estimates,
correlation matrices, coefficient alpha, exploratory FA
 No standard errors produced, so not good for hypo
testing
 Use SAS, NORM, EMCOV, maybe SPSS
 MI (multiple imputation):
 Better because it doesn’t assume that responses lie on
the regression line—it adds in random error
 Use NORM, SAS, Splus
 PAN program—uses growth curves or clustered data
 FIML:
 Does it all in one step
 SEM software: Amos, LISREL, Mx
 Is it okay to “make up data”?
 Inclusive variable inclusion strategies
 Include variables that are correlated, even if not in
model
 Reduces bias
 Helps with power
 How to deal with missing scale items:
 If just part of scale, can use partial data if at least ½
variables and good alpha and all item-total correlations
are about the same
 Otherwise, impute at the individual level
 Issues with MI:
 Works even with n = 50, 18 predictors and 50% missing
 Works with nonnormal data
 Don’t round
 Use lots of imputations (e.g., 40 for 50% missing)
 Consider whether to impute separately for groups (e.g,
gender)
 Can use with clustered data (dummy code)
 Can use with categorical data
 Can use with lots of variables
 3 form design
 All get some questions, others just some get
 2 method measurement
 Get data from everyone on cheap measure and from a
sample on “expensive” measure
 Include good predictors of the missing values in your
data set
 Measure p’s plans to drop out
 Follow up and try to get measures for some drop outs
 Why isn’t looking at difference between stayers and
leavers a cure?
 Longer history in sociology
 Sociograms, centrality
 Clustering, social influence, selection
 Experimental design and mediators/moderators
 Make sure you understand the difference between them
and how to test for them
 2 chapters and 2 articles
 Class is on Tuesday!
 Outline for paper due (whole thing)