Transcript Using Surveys for Assessment

SURVEY RESEARCH
What is a Survey?
A scientific social research method that involves:
 Selecting a random sample of people to answer some
questions
 Designing & administering a standardized
questionnaire to get information about the research
questions
 Analyzing the results to provide descriptions about the
people in the sample and find relationships between
different responses
 Generalizing the results to the population from which
the sample was drawn
2
Basic survey formats




Telephone interview
Face-to-face interview
Computer assisted interview
Self-administered “paper & pencil” survey (scannable
or not)
 Self-administered Web survey
 Self-administered Email survey
3
Why do a survey?
 To collect information not available from other
sources
 To have comparable information
 To be able to generalize results to the
population in which you’re interested
4
Before you plan to do a your
own survey…
Think about alternative sources


Information from non-survey sources
Information from existing survey data
5
Still think you need to do a
survey?
You will have to









Identify the goals and objectives
Develop the research design
Design the questionnaire
Identify the population and select the sample
Pre-test the survey
Administer the survey
Analyze and interpret the results
Write up the results
Present the findings
6
Still think you need to do a
survey?
Think about:
 Resources in terms of TIME (Time up front in planning
stage is especially important) and MONEY(Labor
costs, but also, in some cases, material costs (e.g.,
stamps, envelopes, scannable forms, etc
 How will you USE the results?
7
And if you still want to do a survey
Helpful hints
Familiarize yourself with survey research ‘best practices’
and code of ethics (see www.aapor.org)
 Get approval from your Institutional Review Board if
needed
 Organize a survey team. Include all those working on
any part of the survey project. Get input from all team
members at each step along the way
8
So you still want to do a
survey …
Develop a detailed overall research design





What?
Why?
When?
How?
Who?
Each decision has implications for other steps!
9
What is the purpose of the
survey?
 Explicitly state your objectives
Example:
Improve effectiveness of simulation module developed for modeling
DNA.
 Identify and list factors related to objectives
Examples of factors related to “effectiveness”



Student learning outcomes
How students feel about using the simulation
What they don’t use, etc.
 Make an exhaustive list of everything you want to be able to
measure related to your objective.
10
Who do you need to survey?
Identify your population(s)
Survey everyone in a specific population.
If you
 Have a relatively small population
 Have reason to expect a relatively low response
rate
 Want to have enough respondents to do subgroup analyses for small groups
11
Who do you need to survey?
A sample of the population
If you
 Understand sampling procedures
 Can convince others that it’s okay to sample
 Want/need to limit time in the field
 Want to enhance ability for quality control over
entire process
 Want to keep costs down
12
Select the sample
Probability sampling methods
Simple random sample


List elements in the sampling frame and use random
number chart or computer program to select elements
Systematic sampling


List elements in the sampling frame and take every Nth
element, based on how many needed in sample
Warning: Be attentive to any inherent patterns in how the
sampling frame is ordered.
Stratified sampling


Draw appropriate proportion of elements from
homogenous groups you are interested in
13
Select the sample
Sample size? Depends on:

Cost

Anticipated response rate
Number of responses needed per subgroup
interested in

Level of precision want (“margin of error”)

See Appendix A for more information on sampling.
14
Select the sample
Sample Warnings
BAD SAMPLE = BAD DATA
 Survey results are only representative of the
sampling frame from which the sample was
selected.
 Sample survey results are generalizable only if
respondents are randomly selected.
15
Select the sample
A Warning About Response Rates!
A high response rate
does NOT mean good data
if those who responded
are ‘biased’ in some way
16
Select the sample
A Warning about Web Surveys!
If you can not control who
accesses and responds to a web survey
the results are NOT
generalizable to the population.
17
Design the questionnaire
REMEMBER YOUR OBJECTIVES!
 What do you want to learn?
 How will you use the information?
18
Design the questionnaire
Don’t reinvent the wheel!!
Develop (or borrow*) questions/items to
measure your specific outcomes
*In general it is perfectly acceptable to borrow questions
from other surveys, but always get permission and/or
check for copyright restrictions.
19
Design the questionnaire
To have confidence in your survey
results your questions must be

Reliable
Questions consistently convey the same meaning
to all people in the population being surveyed.

Valid
Questions measure what they are intended to
measure
20
Design the questionnaire
Warning!
Bad Question = Bad Data
The way a question is worded
and the response options offered
determine the nature of the data received.
21
Design the questionnaire
Open-end questions
 Pros

Stimulate free thought

Solicit suggestions

Clarify positions

Richer information
 Cons

Respondent burden

Incomplete, irrelevant, uninterpretable responses

Coding, analyzing, reporting
22
Design the questionnaire
Suggestion for
open-end questions!
Limit the number of open-end questions
BUT always include
at least one
for additional comments.
23
Design the questionnaire
“Forced-choice” (closed-ended) questions

Pros

Less demanding on respondent

Determine levels of intensity, frequency of
participation, etc.

Use to form scales

Easy to analyze and make comparisons
 Cons

Limits richness of data
24
Good Questions…
1)
2)
3)
4)
5)
6)
7)
8)
9)
10)
Are clear and use simple language
Are concise
Are specific
Are possible to answer
Don’t overly tax the respondent’s memory
Are not overly sensitive
Are relevant to the respondent
Do not use double negatives
Avoid biased terms
Have only 1 part (not “double-barrel”)
25
Good Questions…
Most importantly…
Can be connected to your objectives
and

Provide usable information

26
Good Questions…
Good Questions:
Use simple language...
My professor gives me clear feedback on my written
work.
Is better than :
My professor provides lucid feedback when
transmitting her thoughts regarding my written
work.
27
Good Questions…
Good Questions:
Are concise...
How interested would you be in a flexible study
program in which you could take some courses at
home, online?
Is better than:
How interested would you be in a flexible study
program that would still require you to complete
the same credit hours, but for some courses you
could not physically come to the campus but work
from home online?
28
Good Questions…
Good Questions:
Are specific...
In which semester did you first enroll as a full-time
student at NC State University?
Is better than:
When did you start studying here?
29
Good Questions…
Good Questions:
Are possible for the respondent to answer...
NO:
The time I spend in the library has a direct effect
on my grade.
30
Good Questions…
Good Questions:
Do not overly tax the respondent’s memory...
NO:
How many of your peers said hello to you on the
first day of class?
31
Good Questions…
Good Questions:
Are not overly sensitive…
NO:
How often do you take time out to pray during the
course of a normal day?
32
Good Questions…
Good Questions:
Are relevant...
NO:
My professor should get a better haircut.
33
Good Questions…
Good Questions:
Do not use double negatives...
The lack of flexibility in my school hours makes it
difficult for me to spend the amount of time I
would like with my family.
Is better than:
The inflexibility in my school hours does not make it
difficult for me to spend the amount of time I
would like with my family.
34
Good Questions…
Good Questions:
Do not use biased terms or suggestions...
To what extent do you agree or disagree that all
students need to be on campus during regular
hours?
Is better than:
Don’t you agree with the Chancellor when he says
that all students need to be on campus during
regular hours?
35
Good Questions…
Good Questions:
Do not ask two questions in one (“doublebarrel”)...
My professor is knowledgeable about xxxxx
My professor takes the time to make sure I
understand xxxxxx
Is better than:
My professor is knowledgeable about xxxx and takes
the time to make sure I understand them as well.
36
Response Options
Make sure your response options:
 Reflect the concepts you are trying to measure
 Fit with the question wording
 Get at level of precision/intensity needed
 Allow for comparisons between items
 Use “undecided/don’t know’ sparingly
 Are used consistently
 Use balanced scales
 Are mutually exclusive
 Are exhaustive
37
Response Options
Most importantly

Relate back to your objectives

Correspond to how want to use the results
38
Response Options
Must use

Balanced scales
Example: NOT balanced...
How would your rate your professor’s lab classes?
Outstanding
Excellent
Very good
Good
Example: balanced...
How would your rate your professor’s lab classes?
Excellent
Good
Fair
Poor
39
Response Options
Must be:

Mutually exclusive (can select only one appropriate answer)
Example: NOT mutually exclusive...
Q. On most days, what time do you leave your house for
school?
6:00 AM or earlier
7:30-8:00 AM
6:00-6:30 AM
8:00-8:30 AM
6:30-7:00 AM
8:30-9:00 AM
7:00-7:30 AM
9:00 AM or later
40
Response Options

Mutually exclusive continued
Example: IS mutually exclusive...
Q. On most days, what time do you leave your house
for school?
Before 6:00 AM
7:30-7:59 AM
6:00-6:29 AM
8:00-8:29 AM
6:30-6:59 AM
8:30-8:59 AM
7:00-7:29 AM
9:00 AM or later
41
Response Options
Must be:

Exhaustive
all possible answers are listed (including e.g., “other, “
“don’t know,” etc.)
Example: NOT exhaustive
Q. On most days, what time do you leave your house
for campus?
7:00-7:29 AM
7:30-7:59 AM
8:00-8:29 AM
8:30-8:59 AM
42
Response Options

Exhaustive continued
Example: IS exhaustive
Q. On most days, what time do you leave your house
for campus?
Before 7:00 AM
7:00-7:29 AM
7:30-7:59 AM
8:00-8:29 AM
8:30-8:59 AM
9:00 AM or later
43
Response Options
Some common scales:






Excellent, Good, Fair, Poor
Strongly agree, Agree, Disagree, Strongly disagree
Very useful, Somewhat useful, Not very useful, Not
at all useful
Definitely, Probably, Probably not, Definitely Not
All of the time, Most of the time, Some of the time,
Seldom, Never
Strongly agree, Agree, Neither agree nor disagree,
Disagree, Strongly disagree
44
Questionnaire format








As short as possible
Spread out & uncluttered
Attractive & professional looking
Broken into logical sections
Have a good flow
Uncomplicated & easy to follow
Include simple instructions for answering questions
Should have clear skip patterns for contingency
questions
45
Questionnaire format
Question order

Start with interesting, easy, non-threatening
questions
Remember –
‘Question order effects’
Earlier questions provide information and context that
influences responses to later questions
46
Pilot
A pilot or pre-test is
important!
Test for:




Question clarity
Questionnaire format
Variance in responses
Etc.
47
Data collection
Include (in a perfect worlds)
1.
Preliminary announcement
2.
Cover letter, questionnaire w/ return envelope
and/or survey URL
3.
Reminder (with copy of questionnaire and return
envelope and/or survey URL)
4.
2nd reminder (with contact info to request copy of
questionnaire and/or URL)
5.
Thank you card/page
48
Data Collection
Instructions should




Emphasize confidentiality (if applicable)
Provide contact information
Briefly explain

purpose of survey

how results will be used

why respondent was selected

why participation is important

how to access questionnaire
THANK respondent for participating
49
Clean, analyze, & interpret
the data



Remember your objectives
Remember your report outline
Remember your audience
 Keep it simple…
 You can always do more analyses…
50
Interpreting the results
Everything is relative!
Suggestions for what to look for:





Do responses/ratings meet pre-defined goals?
(e.g., % giving a positive rating)
How do responses/ratings for individual items compare to
each other?
What factors/characteristics are related to ratings?
How do responses/ratings from one group of respondents
compare to another?
(How have ratings changed over time?)
51
Prepare written report(s)
Generally include




What you did (methods)
Why you did it
What you learned
How it can be used
52
Want to learn more about
surveys?
American Association of Public Opinion Research, “Best Practices for Survey
and Public Opinion Research.” http://www.aapor.org/ethics/best.html
Babbie, Earl (2002), The Basics of Social Research 2nd ed. Wadsworth
Thomson Learning: CA.
Dillman, D.A. (2000), Mail and Internet Surveys 2nd ed. Wiley: NY.
Fink, Arlene and Jacqueline Kosecoff (1998), How to Conduct Surveys. Sage:
CA.
Fowler, Floyd J. (1993), Survey Research Methods. Sage: CA.
Fowler, Floyd J. (1995), Improving Survey Questions. Sage: CA.
National Council on Public Polls, “20 Questions a Journalist Should Ask About
Poll Results.” http://www.ncpp.org/qajsa.htm
Sudman, Seymour and Bradburn, Norman (1982), A Practical Guide to
Questionnaire Design. Jossey-Bass: San Francisco.
Tourangeau, R., and Smith, T.W. (1996), “Asking Senstive Questions: The
Impact of Data Collection Mode, Question Format, and Question Context,
“ Public Opinion Quarterly, 60:275-304.
53
Appendix A
Response Rate
# surveys completed and returned
net sample size*
*net sample size = sample size - undeliverable surveys
54
Appendix A
Margin of Error
Based on
# people in the sample (or population)
# people who respond
the response distribution




e.g., 75% said yes, 25% said no
how sure you want to be of you data (“confidence
interval”)


e.g., a 95% confidence interval means that you can be
sure that 95 out of 100 times the responses to a given
question would be within some calculable percentage
points of the actual number you got.
55
Appendix A
Margin of Error
Examples
(at 95% confidence interval)
Population Size # Respondents
100
75
40
30
10
7
Margin of Error
+ 2.8
+ 4.5
+ 11.1
E.G. If 50% of the 7 respondents (from your population of 10) said
“yes,” you can be sure that 95 out of 100 times between 38.9%
and 61.1% of those in your population would say “yes.”
56