Transcript Psy301 - Lecture 1 - Outline

Research Methods & Design in
Psychology
Lecture 2
Survey Design 2
Lecturer: James Neill
Overview
•
•
•
•
•
Survey construction - nuts & nolts
Sampling
Ethics
Levels of measurement
Measurement error
What is a Survey?
• A standardised stimulus
• A measuring instrument
• A way of converting fuzzy psychological stuff
into hard data for analysis
Survey Construction – Nuts & Bolts
•
•
•
•
•
Constructing questions
Modes of response
Response formats <-> LOM
Measurement error
Survey formatting
Constructing questions
• Define target constructs
• Check related research & questionnaires
• Draft items
(aim to have multiple indicators)
• Pre-test & revise
When drafting questions aim to:
•
•
•
•
•
Focus directly on topic/issue
Be clear
Be brief
Avoid big words
Use simple and correct grammar
Bias in questions
•
•
•
•
•
•
•
Inapplicable
Over-demanding
Ambiguous
Double negatives
Double-barrelled
Leading
Loaded
Bias in responding
•
•
•
•
Social desirability
Acquiescence or Yea- and Nay-saying
Self-serving bias
Order effects
Modes of Survey Administration
Interview
• high demand characteristics
• can elicit more information
Questionnaire
• lower demand characteristics
• information may be less rich
Objective vs. subjective
Objective:
How times during 2000 did you visit a G.P.?
Subjective:
Think about the visits you made to a G.P. during
2000. How well did you understand the medical
advice you received?
perfectly
very well
reasonably
poorly
not at all
Open-ended vs. close-ended
Open-ended
•
•
•
•
rich information can be gathered
useful for descriptive, exploratory work
difficult and subjective to analyse,
time consuming
Close-ended
•
•
•
•
important information may be lost forever
useful for hypothesis testing
easy and objective to analyse
time efficient
Open-ended questions Examples
What are the main issues you are currently
facing in your life?
How many hours did you spend studying this
week? _________
Close-ended questions – Example 1
What are the main issues you are currently facing in
your life? (please all that apply)
•
financial
•
physical/health
•
academic
•
employment/unemployment
•
intimate relations
•
social relations
•
other (please specify)
________________________________
Close-ended questions – Example 2
How many hours did you spend studying this
week?
•
less than 5 hours
•
> 5 to 10 hours
•
> 10 to 20 hours
•
more than 20 hours
Close-ended rating scales
•
•
•
•
•
Likert scale
Graphic rating scale
Semantic differential scale
Non-verbal scale
Frequency scale
Likert Scale
Pick a number from the scale to show how much
you agree or disagree with each statement:
1
2
3
4
strongly
disagree
disagree
neutral
agree
1
2
3
4
strongly
agree
agree
neutral
disagree
5
strongly
agree
5
strongly
disagree
Graphic Rating Scale
How would you rate your enjoyment of the
movie you just saw? Mark with a cross (X)
not enjoyable
very enjoyable
Semantic Differential Scale
What is your view of smoking?
Tick to show your opinion.
Bad
___:___:___:___:___:___:___
Strong
___:___:___:___:___:___:___
Masculine ___:___:___:___:___:___:___
Unattractive ___:___:___:___:___:___:___
Passive
___:___:___:___:___:___:___
Good
Weak
Feminine
Attractive
Active
Non-verbal Scale
Point to the face that shows how you feel
about what happened to the toy.
Verbal Frequency Scale
Over the past month, how often have you
argued with your intimate partner?
1. All the time
2. Fairly often
3. Occasionally
4. Never
5. Doesn’t apply to me at the moment
Sensitivity & Reliability
• Scale should be sensitive yet reliable.
• Watch out for too few or too many options
Scale of measurement guidelines
General aim:
Maximise sensitivity
Maximise reliability
(i.e. more options)
(i.e. less options)
How many measurement options?
• Minimum
=2
• Average
= 3 to 7
• Maximum
= 10?
FEELING ABOUT SOMETHING
EXTREMELY POSITIVE
EXTREMELY NEGATIVE
2-Categories
GOOD
NOT GOOD
3-Categories
GOOD
FAIR
POOR
4-Categories
VERY GOOD
GOOD
FAIR
POOR
5-Categories
EXCELLENT
VERY GOOD
GOOD
FAIR
POOR
Watch out for too many or too few
responses
“Capital punishment should be reintroduced for
serious crimes”
1 = Agree
2 = Disagree
1 = Very, Very Strongly Agree 7 = Slightly Disagree
2 = Very Strongly Agree
8 = Disagree
3 = Strongly Agree
9 = Strongly Disagree
4 = Agree
10 = V. Strongly Disagree
5 = Slightly Agree
11 = V, V Strongly Disagree
6 = Neutral
Sampling
•
•
•
•
•
Sampling Terminology
What is Sampling?
Sampling Techniques
Example: Shere Hite’s Sex Survey
Summary of Sampling Strategy
Sampling Terminology
•
•
•
•
Population
Sampling Frame
Sample
Representativeness
What is sampling?
“Sampling is the process of selecting units
(e.g., people, organizations) from a
population of interest so that by studying
the sample we may fairly generalize our
results back to the population from which
they were chosen.”
- Trochim, 2002
Sampling Techniques
• Probability sampling
– Random
– Systematic
– Cluster
• Multi-Stage Cluster
• Non-probability sampling
– Quota
– Convenience
– Snowball
Representativeness of sample
depends on:
•
•
•
•
adequacy of sampling frame
selection strategy
adequacy of sample size
response rate – both the % &
representativeness of people in sample who
actually complete survey
• Note: It is better to have a small, good
sample than a large, poor sample.
Sampling Example:
Shere Hite
‘American Sexology’
Male-Female Relations
• Shere Hite ‘doyenne of sex polls’
• Media furors & worldwide attention
• 127-item questionnaire about marriage &
relations between sexes
• 4500 USA women, 14 to 85 years
• Society and men need to change to improve
lives of women
Some of Hite’s findings....
• 70% married for 5 years having affairs...
(usually more for ‘emotional closeness’ than sex)
•
•
•
•
•
•
76% did not feel guilty
87% had a closer female friend than husband
98% wanted “basic changes” to love relationships
only 13% married for 2+years were still in love
84% were emotionally unsatisfied
95% reported emotional & psychological harassment
from their men
Some of the critical comments....
• “She goes in with prejudice & comes out with a
statistic.”
• “The survey often seems merely to provide an
occasion for the author’s own male-bashing
diatribes.”
• “Hite uses statistics to bolster her opinion that
American women are justifiably fed up with
American men.”
Response rate & Selection bias - 1
100,000 questionnaires
Sent to a variety of women’s groups
- feminist organisations, church groups,
garden clubs, etc.
4,500 replied
(4.5% return rate)
Response rate & Selection bias - 2
“We get pretty nervous if respondents in our
survey go under 70%. Respondents to surveys
differ from nonrespondents in one important
way: they go to the trouble of filling out what in
this case was a very long, complicated, and
personal questionnaire.”
- Regina Herzog, University of Michigan Institute for Social
Research
Summary of sampling strategy
•
•
•
•
Identify target population and sampling frame
Selection sampling method
Calculate power and required sample size
Maximise return rate
Survey Format Checklist
• Introduction/covering letter or verbal
introducation
– e.g. Who are you? Are you bona fide? Purpose of
survey? Ethical approval? How results will be used?
Confidentiality? Further info? Complaints?
• Instructions
– Sets the “mind frame”, but be aware few people will
read it without good prompting and being easy-to-read
• Group like questions together
• Consider order effects, habituation, fatigue,
switching between response formats
Survey Format
• Font type / size, number of pages, margins,
double vs. single-siding, colour, etc.
• Demographics - single section, usually at
beginning or end of questionnaire, only use
relevant questions
• Space for comments?
• Ending the questionnaire – say thanks!
• Pre-test the questionnaire & revise/refine
Pre-test & Revise
• Pre-test items and ask for feedback
• Revise:
–
–
–
–
–
items which don’t apply to everybody
redundancy
skewed response items
misinterpreted items
non-completed items
• Reconsider ordering & layout
Ethical issues: How to treat
respondents
•
•
•
•
•
•
Minimise risk/harm to respondents
Informed consent
Confidentiality / anonymity
No coercion
Minimal deceit
Fully debrief
Other ethical issues
• Honour promises to provide respondents
with research reports
• Be aware of potential sources of bias/
conflicts of interest
• Represent research literature fairly
• Don’t search data for pleasing findings
• Acknowledge all sources
• Don’t fake (or unfairly manipulate) data
• Honestly report research findings
Levels of measurement = type of
data
Levels of Measurement
=
Type of Data
4 levels of measurement
•
•
•
•
Nominal/Category
Ordinal
Interval
Ratio
Levels of measurement – discrete vs.
continuous
•
•
•
•
Categorical / Nominal
Ordinal / Rank
Interval
Ratio
(Discrete)
(Discrete)
(Discrete?)
(Continuous)
Each level has the properties of the preceeding
levels, plus something more!
Categorical / Nomimal
• Arbitrary assignment of #s to categories
e.g. male = 1, female = 2
• No useful information, except as labels
Ordinal /Ranked Scales
• #s convey order, but not distance
e.g. in a race, 1st, 2nd, 3rd, etc.
• Often must be treated as categorical
Interval Scales
• #s convey order & distance, 0 is arbitrary
e.g. temperature (degrees C)
• Usually treat as continuous for >5 intervals
Ratio Scales
• #s convey order & distance, meaningful 0
e.g. height, age
• ratios - e.g. 2 x old, 3 x high
Why do levels of measurement
matter?
different analytical procedures
are used for different
levels of data
More powerful statistics can be applied to
higher levels
Measurement scales -> Analysis
categorical & nominal
-> non-parametric
interval & ratio
-> parametric
What are parametric stats?
= procedures which estimate PARAMETERS of a
population, usually based on the normal
distribution
• any procedure which uses M, SD
–
e.g. t-tests, ANOVAs
• any procedure which uses r
–
e.g. bivariate correlation, linear regression
Practice Exam Question
Level
Nomimal
/Categorical
Ordinal /
Rank
Interval
Ratio
Properties Examples Descriptive Graphs
Statistics
What are non-parametric stats?
(Distribution-free Tests)
= procedures which do not rely on estimates
of population PARAMETERS
• any procedure which uses frequency
–
e.g. sign test, chi-squared
• any procedure which uses rank order
–
e.g. Mann-Whitney U test, Wilcoxon matchedpairs signed-ranks test
Parametric vs. non-parametric stats?
parametric statistics are more
powerful
but are also more sensitive to
violations of assumptions
Measurement error
• Observed score
= true score + measurement error
= true score + systematic error + random error
• Measurement error is any deviation from
the true value.
Sources of Error
Non-sampling
Paradigm
Sampling
Personal
Sources of measurement error
•
•
•
•
Paradigm
Personal researcher bias
Sampling
Non-sampling
To minimise measurement error
•
•
•
•
Use well designed measures
Reduce demand effects
Maximise response rate
Ensure administrative accuracy
Summary
•
•
•
•
•
Survey construction - nuts & bolts
Sampling
Ethics
Levels of measurement
Measurement error
Respond to UC’s student survey and
win an iPod Video valued at $380
Features: 30GB, music
videos, home movies and
video podcasts, new iPod
games, audiobooks, photo
albums and, of course, an
entire library of up to
20,000 songs.
• Logon to OSIS before
14 March, fill in the
student survey & you’ll
enter the draw!