Transcript Part - time MSc course Epidemiology & Statistics Module

The following lecture has been approved for
University Undergraduate Students
This lecture may contain information, ideas, concepts and discursive anecdotes
that may be thought provoking and challenging
It is not intended for the content or delivery to cause offence
Any issues raised in the lecture may require the viewer to engage in further
thought, insight, reflection or critical evaluation
Questionnaires & Surveys
Delivery
Design
Development
Improving Response rates
Prof. Craig Jackson
Head of Psychology Division
Education Law & Social Sciences
BCU
Are Postal Questionnaires Dead Yet?
IT predicted death of postal surveys
Use of IT at home and work increases survey methods
Comparison of surveys using
WWW
or
EMAIL
or
POSTAL
Subjects – UK university staff
200 email questionnaires
200 emails with www url
100 postal questionnaires
Asked 3 questions:
teeth cleaning
fruit
walking
Jones, R. and Pitt, N. 1999
Are Postal Questionnaires Dead?
Results
Days after sending
email
n=200
www
n=200
post
n=100
numbers responding
3
5
9
10
17
17 day response rate
25
59
61
63
68
34%
31
34
35
35
37
19%
0
16
53
60
72
72%
cost per reply
Actual cost
With 100% response
35p
19p
41p
7.5p
92p
72p
What this means for surveys
Golden age of communication?
Postal methods still much better
Novelty value of email is dead - therefore lower response rates
Junk mail perceptions
Email Filters are improving
Postal letters demonstrate & emphasize their importance
www surveys & email allow immediate data processing (software)
Email & www have potential for low cost regular user surveys
Intranet users benefit from e-surveys
Introduction
Questionnaire is a fundamental component of most research projects
Most MSc / PhD projects use questionnaire methods
Largest reason for criticising projects - weak / dubious questionnaires
Can be very efficient
A Questionnaire does:
Take planning to get right
Reward time spent on it
Capture information
A Questionnaire is not:
Constructed in 10 minutes
An easy option
A collection of simple questions
Basics of Measurement
Measurement tools must be appropriate
Psychometrics -- personality, attitudes, stress, symptoms, Physical
measurements -- working, environment, symptoms
Exposure assessments -- hazards, risks, ppm3, duration
When measuring...
Take multiple measurements (and take the mean)
Under same conditions, but if not….
Statistical remedies to adjust e.g. age, time of day etc.
Reliable & Validated tool
Defined and Regular variables
Well-defined standard of reference
Dilemma
Achieving a high response rate to a questionnaire is vital
But does not promise a normal distribution of responses?
Postal questionnaires rarely get a response rate > 40%.
Unless respondents have a vested interest in the outcome.
Bias?
Most efficient (best) response rates usually happen when respondents have to
do very little to take part in the study.
Multiple phase projects see a depletion in numbers at every stage.
Quick “in and out” one-stop approach is best
http://onlinestatbook.com/stat_sim/sampling_dist/index.html
Know your enemy
Diminishing returns of multi-stage recruitment
Researcher
Potential Sample
1000 people
540 consents
540 questionnaires
Under-powered study
n = 210
Response rate of 21%
210 questionnaires
Structure
Identifying Items
Title
Preamble
Branch Instructions
Research Items
does not need to be honest
some deception is necessary
shorter is better
Identifying items
Preliminary questions
Collecting info necessary for screening:
recording keeping
tracking
tracing
data manipulation
Ask only for relevant info - unethical
Fewer items minimizes chance of alienating respondent
The need for rich info
to improve the study
Participant’s need for
privacy & anonymity
Encryption devices
Steganography
Secret communication of a message by hiding it’s existence
Steganos, meaning covered. Gk
Graphein, meaning to write, Gk
If message is discovered it is easily read because of no encryption
Cryptography
Secret communication of a message by hiding it’s meaning
Kryptos, meaning hidden. Gk
Message established using a known protocol, to be decrypted by the receiver
Steganography & Cryptography can be combined together if needed
Steganogrpahy arouses less suspicion in questionnaire respondents
Encryption devices
Steganography
“This example of steganogrpahy may not work very well when projected onto a
large screen, but it works very well on paper, such as questionnaires.
Hopefully many of you will not notice the method of steganography used in
this piece of text. Gosh how clever I am……”
Cryptography
drbjh kbdltpo, jotujuvuf pg pddvqbujpobm ifbmui
The above text (containing an encrypted name and address) looks suspicious
and may be obliterated by the respondent
drbjh kbdltpo, jotujuvuf pg pddvqbujpobm ifbmui
Preamble
An important introduction
Frame of reference
Without a FOR a respondent may base their answer on a wrong context
“We would like to know if you have had any medical complaints
and how your health has been in general, over the past few weeks.
Please answer ALL the questions on the following pages by ticking
the answer which best applies to you. Remember that we want to
know about present and recent complaints, not those that you had
in the past. It is important that you try to answer all questions”
Title
Not to be underestimated – try to be user-friendly
Sets the tone for the respondent
The General Health Questionnaire vs The GHQ 28
“Simple language for simple people”
Beware of abbreviated titles - may alienate respondents
e.g.
Agricultural Satisfaction Scale
LEI
BDI
MMPI
HAD etc
Research items
Set(s) of items to collect the “real” research data
Take many forms:
Item Type
Data type
Fixed option items
(ordinal or nominal)
Rating scale items
(likert)
Yes / No items
(binary)
True / False items
(binary)
Likert scale items
(likert)
Diagrams
binary, ordinal, or nominal
Branch Instructions
Guide the respondent through the items
Maximises the efficiency of respondent’s efforts
Avoids redundant items
Focuses on worthwhile items
Instructions can sometimes confuse the respondent so think carefully
e.g
“If you answered ‘NO’ to question 9, ignore 9b, 9c, and 9d, go to question 10.”
Ah, the Truth
Are your respondents telling the truth?
Use a sprinkling of LIE DETECTOR items to assess reliability
e.g
“Have you ever taken anything without asking permission?”
“Did you ever lie to your parents as a child?”
“Have you ever visited a pornographic web-site?”
Respondent
“Outrage”
increases
with item
intimacy
Decide what to do with any respondent who “fails” the lie detector
Keep them in?
Exclude immediately?
Retain but “flag” them?
Funnel Items
Set funnel questions to try to narrow down a respondent to specific details
e.g
6a “Did you write essays in college?”
Yes / No
6b “What type of essays were they mostly?”
Narrative / Descriptive / Persuasive
6c “How often did you write them?”
___ per term
1 per term / 2-3 per term / 3 or more
Response Set Bias
Enables collection of detailed data easily
Build a detailed data set from a simple binary item
Randomise the layout / order of responses “true / false”
e.g
True
True
True
True
True




False
False
False
False
False
Respondents ticking the same items
Swap the “True & False” responses sequence
Likert scales
A visual linear scale for rating purposes
happy
X
Swap around the words to avoid response-set bias
Avoid using numbers on the scale -- can be (mis)leading
sad
Likert scales
XX
X
happy
X
XXX
XX
X
X
X
X
X
XX X
X
X
X
X
sad
Likert scales
XX X
happy
XX
X
X
XX X X
X
sad
Design Pointers
Avoid colloquialisms or abbreviations
Beware of local expressions
Avoid words with double meanings
e.g.”Fair....Dip....Lie....Well”
Set a definition of specific terms
e.g. “OK.....Average”
Avoid long questions
Specify exact time, place and context
e.g “At school, did you ever...”
Phrase items to make denial impossible e.g “When did you first...”
Avoid numbers on your responses / scales
Have items seemingly related to the research topic
Design Pointers
Clearly phrase items
Make items unambiguous
Avoid leading items
e.g. “Many people think...”
Ask only what a respondent is qualified to answer
Avoid socially loaded items
e.g. “beggars” or “junkies”
Do not use socially / religiously biased items
Avoid phrases, clichés or sooth-sayings
Ask for ages as a true number / D.O.B - - NOT in age groups
Font
Pictures
Most common errors
“In your organization, do women have the same
responsibilities as men, and should women have more?”
“Preventing accidents in the workplace is vital, and more
money should be spent on prevention.”
“Training in risk assessment is something I would like to do,
and I would like to see my colleagues do it to.”
Only ask one question per item
Most common errors
What is your age ? (please tick)
16 - 26
26 - 36
36 - 46
46 - 56
What is your marital status ? (please tick)
Married
Single
How many blood splashes have you had ? (please tick)
1-5
6 - 10
11 - 15
15 - 20
20 plus
Are life preservers and flares essential on-deck equipment ?
agree
disagree
Factorial approach
The GHQ 28
A self-completion questionnaire assessing mental health
How?
28 items
7 about Anxiety
7 about Severe depression
7 about Dysfunction
7 about Somatic symptoms
Anxiety score
Depression score
Dysfunction score
Somatic score
Global score
By summing the 4 factors there is a Global Mental Health score
Statistics are performed on the factor scores and the global score
NOT on each individual item
3
4
5
2
14
Designing factors
Piloting items
Set down as many items as
possible concerning the topic
Limit the questionnaire to 30
items
Give the questionnaire to at
least 20 people
Score responses and place
into SPSS spreadsheet
Factor Analysis
use statistics to “group” items together into factors
(will do this as a group exercise in the SPSS lecture)
Creating factors
Factor Analysis
Results in manageable number of factors instead of 30 items
Each factor comprised of a number of items that had similar answers
from your pilot sample
The similarity of responses to particular items is what SPSS uses to
group items together into factors
Identifying Items
As a matter of routine:
Get as much information as possible
may be vital in later (unforeseen analyses)
Sex
DOB or Age (not age groups)
Marital status / Domestic arrangements
Increasing Response Rates
Incentives
Appearance
Delivery
Origin
Contact
Content
Communication
Increasing Response Rates - Incentives
Money
Vouchers
Prize draw
Ethical aspects
Bias
O.R
Monetary incentive vs. None
2.02
Incentive with Q. vs. Incentive on return
1.71
Non-monetary incentive vs. No incentive
1.19
Edwards et al. 2002
Increasing Response Rates - Appearance
O.R
Shorter format vs. Longer format
1.86
Brown envelope vs. White
1.52
Coloured ink vs. Black
1.39
Folder / Booklet vs. Stapled pages
1.17
Personalised vs. Not personalised
1.16
ID feature on return vs. No ID
1.08
Coloured Q vs. White Q
1.06
Edwards et al. 2002
Increasing Response Rates – Delivery methods
O.R
Recorded delivery vs. Standard
2.21
Stamped return envelope vs. Business reply / franked
1.26
Q sent to work vs. Q sent to home
1.16
1st class outward mail vs. Other class
1.12
Pre-paid return envelope vs. Not pre-paid
1.09
Stamped outward envelope vs. Franked
0.95
Commemorative stamp vs. Ordinary stamp
0.92
Edwards et al. 2002
Increasing Response Rates – Origin & Contact
Origin
O.R
University vs. Other organisation
1.31
Sent by senior persons vs. Juniors
1.13
Ethnically ambiguous name vs. Non-white name
1.11
Contact
O.R
Pre-contact vs. No contact
1.54
Follow up vs. No follow up
1.44
Postal follow up with Q vs. Without Q
1.41
Mentioning follow up vs. None
1.04
Pre-contact by telephone vs. Postal pre-contact
0.90
Edwards et al. 2002
Increasing Response Rates - Content
O.R
More interesting vs. Less interesting
2.44
User-friendly vs. Standard
1.46
Factual items only vs. Factual & attitude items
1.34
Relevant items first vs. Other items
1.23
Demographic items first vs. Other items
1.04
“Don’t know” boxes vs. no “Don’t know” boxes
1.03
Sensitive items vs. No sensitive items
0.92
General items first vs. Last
0.80
Edwards et al. 2002
Increasing Response Rates - Communication
O.R
Explain drop-out required vs. Not
1.32
Stresses benefit to respondent vs. Others
1.06
Stresses benefit to sponsors vs. Others
1.01
Stresses benefit to society vs. Others
1.00
Response deadline given vs. No deadline
1.00
Instructions given vs. No instructions
0.89
Choice to opt out given vs. No opt out
0.76
Edwards et al. 2002
Questionnaire Summary
1. Postal questionnaires widely used in data collection
2. Mark pre-pay or addressed envelopes
3. Use steganography over cryptography
4. Perform analyses on factors not individual items
5. Identifying items may be useful in later analyses
6. Think about scoring items before rushing ahead with the questionnaire
7. Store questionnaires securely until passing any viva voce
8. Collect as much info at source, parse it down later at discretion
Questionnaire Summary cont.
9. Perfect for epidemiological studies and health research
10. Non-response to postal questionnaires reduces sample & introduces bias
11. Identification of effective ways to increase postal response rates
12. Use existing metrics - pilot items if making new questionnaire
13. Use factor analysis
14. Keep it all as brief as possible
15. Don’t alienate respondents
16. Alternate types of items