Transcript Lecture 7

EMR 6500:
Survey Research
Dr. Chris L. S. Coryn
Lyssa N. Wilson
Spring 2015
Agenda
• Stratified random sampling for
proportions
• From questions to a questionnaire
• Budgeting survey research
Stratified Random Sampling
Estimate of Population
Proportion
1
1 L
N1 pˆ1  N 2 pˆ 2    N L pˆ L    N i pˆ i
pˆ st 
N
N i 1
1 é 2
V̂( p̂st ) = 2 ë N1 V̂ ( p̂1 ) + N 22V̂ ( p̂2 ) + + N L2V̂ ( p̂L )ùû
N
L
1
2
= 2 å N1 V̂ ( p̂i )
N i=1
L
æ ni öæ p̂i q̂i ö
1
2
= 2 å Ni ç1- ÷ç
÷
N i=1 è Ni øè ni -1 ø
Example for a Population
Proportion
N
n
n+
p̂1
Town A
155
20
16
0.80
Town B
62
8
2
0.25
Rural
93
12
6
0.50
1 é
p̂st =
ë(155) ( 0.80) + ( 62) ( 0.25) + ( 93) ( 0.50)ùû = 0.60
310
Example for a Population
Proportion
æ ni öæ p̂1q̂1 ö æ 20 öé ( 0.80) ( 0.20) ù
V̂( p̂1 ) = ç1- ÷ç
÷ = ç1- ÷ê
ú
19
è Ni øè ni -1 ø è 155 øë
û
= ( 0.871) ( 0.008) 0.007
V̂( p̂2 ) = 0.024
V̂( p̂3 ) = 0.020
Example for a Population
Proportion
3
1
2
V̂( p̂st ) = 2 å Ni V̂ ( p̂i )
N i=1
1 é
2
2
2
ù
=
(155)
(0.007)+
(62)
(0.024)+
(93)
(0.020)
û
2ë
(310)
= 0.042
p̂st ± 2 V̂ ( p̂st ) = 0.60 ± 2 0.0042 = 0.60 ± 2 ( 0.065) = 0.60 ± 0.13
Selecting the Sample Size for
Estimating Population
Proportions
Sample Size for Estimating
Population Proportions
L
n=
åN
p q / ai
2
i i i
i=1
L
N D + å N i pi qi
2
i=1
The forumula for sample size n (for a given bound B on the error of estimation)
is the same as for a mean except that s 12 becomes pi qi
B
D=
4
ai = fraction of observations allocated to stratum i
pi = population proportion for stratum i
2
Sample Size for Estimating
Population Proportions
• As with allocation methods for means
and totals, allocation methods for
stratified random samples for
proportions can be equal,
proportional, or optimal depending
on the study’s purpose
From Questions to a
Questionnaire
General Premises
• The design of a questionnaire should
consider how to motivate the
recipient to respond
• It should also avoid measurement
errors, ranging from order effects to
item nonresponse
Guidelines for Ordering
Questions
Ordering Questions
• General guidelines
1. Group related questions that cover
similar topics, and begin with
questions likely to be salient to nearly
all respondents
2. Choose the first question carefully
3. Place sensitive or potentially
objectionable questions near the end
4. Ask questions about events in the
order that they occurred
5. Avoid unintended order effects
Guidelines for Creating a
Common Visual Stimulus
Visual Stimulus
• General guidelines
1. Establish consistency in the visual presentation of
questions (across pages and screens) and use
alignment and vertical spacing to help
respondents organize information on the page
2. Use color and contrast to help respondents
recognize the components of the questions and
the navigational path through the survey
3. Visually group related information in regions
through the use of contrast and enclosure
4. Use visual elements and properties consistently
across questions to emphasize or deemphasize
certain types of information
5. Avoid visual clutter
6. Minimize the use of matrixes and their complexity
First name/initials
(optional)
Does this relative live
in Kalamazoo? . . . . . . .
If no: about how far
away from Kalamazoo
do they live? . . . . . . . .
About how old is this
relative? . . . . . . . . . . . .
Approximately, how
often do you
communicate with this
relative? . . . . . . . . . . . .
Relative 1
Relative 2
Relative 3
(name)
(name)
(name)
Miles
Miles
Miles
Years
Years
Years
Every day
Every day
Every day
Every week
Every week
Every week
Once a month
Once a month
Once a month
Less than
once a month
Less than
once a month
Less than
once a month
Yes
No
Guidelines for Mail
Questionnaires
Mail Questionnaires
• General guidelines
1.
2.
3.
4.
5.
6.
Determine whether keypunching or optical imaging
and scanning will be used, and assess the limitations
imposed on designing and processing questionnaires
Construct paper questionnaires in booklet formats, and
choose physical dimensions based upon printing and
mailing considerations
Decide question layout and how questions will be
arranged on each page
Use symbols, contrast, size, proximity, and pagination
effectively when designing branching instructions to
help respondents correctly execute them
Create interesting and informative front and back
pages that will have wide appeal to respondents
Avoid placing questions side-by-side on a page so that
respondents are asked to answer two questions at
once
Guidelines for Web
Questionnaires
Web Questionnaires
• General guidelines
1. Decide whether an electronic alternative is
appropriate
2. Choose how the survey will be programmed
and hosted, commensurate with needs, skills,
and sponsorship
3. Take steps to ensure that questions display
similarity across platforms, browsers, and
user settings
4. Decide how many questions will be presented
on each page and how questions will be
presented
5. Develop a screen format that emphasizes the
respondent rather than the sponsor
Web Questionnaires
• General guidelines
6. Use a consistent page layout across screens and
visually emphasize questions information that
respondents will need to complete the survey
while deemphasizing information not essential to
the task
7. Do not require responses unless absolutely
necessary
8. Design survey-specific and item-specific error
messages to help respondents troubleshoot
9. Evaluate carefully the use of interactive features,
balancing improvements in measurement with the
impact on respondent burden and the
implications with mixed-mode surveys
Web Questionnaires
• General guidelines
10. Use audiovisual capabilities sparingly, and
evaluate the differential effect they have on
respondents
11. Allow respondents to stop the survey and finish
completing it at another time
12. Whenever possible, collect paradata that provide
feedback on how respondents interact with
questionnaire
13. Test the survey using a variety of platforms,
connection speeds, browsers, and user-controlled
settings, and test the database to ensure that
items are collected and coded accurately
14. Take screenshots of each page of the final
questionnaire for testing and documentation
Guidelines for Pretesting
Questionnaires
Pretesting Questionnaires
• General guidelines
1. Obtain feedback on the draft of the questionnaire
from a number of people, each of whom has
specialized of some aspect of questionnaire
quality
2. Conduct cognitive interviews of the complete
questionnaire in order to identify wording,
question order, visual design, and navigational
problems
3. When the stakes are high, consider experimental
evaluations of questionnaire components
4. Conduct a small pilot study with a subsample of
the population in order to evaluate
interconnections among questions, the
questionnaire, and implementation procedures
Budgeting Survey Research
General Cost Frameworks for
Survey Research
• Fixed-price
– All costs are subsumed under a single
dollar value, no matter time, effort, and
other expenditures
• Cost-reimbursable
– Reimbursement for actual expenditures
Direct Costs
• Personnel
– Hourly staff (+7.65% FICA)
– Salaried staff (+ 48.75% for full benefits)
– Academic staff (+ 22.25% for full benefits)
• Travel
– Local transportation, flights, per diem, lodging
• Other costs
– Materials and supplies (e.g., office and operating
supplies, telephone and conference calls, postage and
shipping)
– Publication and dissemination (e.g., standard printing,
copying, production)
– Participant costs and/or incentives
– Consultants or subawards
– Computer services
Indirect Costs
• F&A (facilities and administration)
– 49% is WMU’s current HHS approved
rate
– 24% is WMU’s current off-campus rate
(requires rental space elsewhere)
– Other rates can be negotiated based on
institutional policies (e.g., existing policy
of funders)
Typical Budgeting Methods
• Per task
• Per day (e.g., %FTE)
• In budgeting, both are often
necessary
Consider…
• A mail survey to 500 randomly selected
recipients
– Printing of surveys = $2.00/survey
– Postage = $0.45/survey for delivery +
$0.45/survey for return
– Processing of surveys = 1 hour/survey @
$18/hour
– $2/recipient incentive
• What is the total cost for just the
above, which does not include other
costs?