Transcript Part - time MSc course Epidemiology & Statistics Module

The following lecture has been approved for
University Undergraduate Students
This lecture may contain information, ideas, concepts and discursive anecdotes
that may be thought provoking and challenging
It is not intended for the content or delivery to cause offence
Any issues raised in the lecture may require the viewer to engage in further
thought, insight, reflection or critical evaluation
Survey Research
Samples and Populations
Response rates
Data Types
Honesty
Delivery
Validity
Dr. Craig Jackson
Senior Lecturer in Health Psychology
Faculty of Health
UCE Birmingham
Brief Research History
Fatigue in short-haul budget airline pilots 2002-2004
(Web-based Survey)
Mental Health of UK Farmers using OP Pesticides (X2) -- 1997-2000
(Epidemiological Surveys)
Neurobehavioural Performance of desert-based Oil Drillers -- 1998-2000
(Clinical assessment)
Temporary Hearing Loss in Student Bar Staff – 2000-2002
(Epidemiological Survey)
Benefits of Occupational Health Advice in Primary Care Settings -- 2001-2004
(Randomised Controlled Trial)
Smaller-Scale projects – (Tri-Services, NHS Personnel, NHS Patients)
(Cross-sectional Surveys, Clinical Trials)
Multiple roles of psychologist, statistician, and methodology designer
Quantitative Research Designs
Patients
Staff
Healthy
Laboratory
Experimental
RCT
Approach
Case - control
Epidemiology
Cohort study
Observational
Survey
Postal questionnaire
Are Postal Surveys Dead Yet?
IT predicted death of postal surveys
Use of IT at home and work increases survey methods
Comparison of surveys using
WWW
or
EMAIL
or
POSTAL
Subjects – UK university staff
200 email questionnaires
200 emails with www url
100 postal questionnaires
Asked 3 questions:
teeth cleaning
fruit
walking
Jones, R. and Pitt, N. BMJ1999
Are Postal Surveys Dead Yet?
Results
Days after sending
email
n=200
www
n=200
post
n=100
numbers responding
3
5
9
10
17
17 day response rate
25
59
61
63
68
34%
31
34
35
35
37
19%
0
16
53
60
72
72%
cost per reply
Actual cost
With 100% response
35p
19p
41p
7.5p
92p
72p
The Postal Survey is Alive and Well . . . .
Communication age?
Postal surveys still much better than e-surveys
Novelty value of email is dead
“Junk mail” perceptions
Email filters are improving – fewer emails get through
Postal letters demonstrate & emphasize their importance
www surveys & email allow immediate data processing (software)
Email & www have potential for low cost regular user surveys
Intranet use of e-surveys seems ok
Survey Research
Questionnaire is a fundamental component of most surveys
Most MSc / MPhil /PhD projects use survey methods
Can be very efficient
Weaknesses
weak / dubious questionnaires
non-valid questionnaires
biased samples
biased responses
poor response rate
Wide range of applications
Psychometrics -- personality, attitudes, stress, symptoms
Physical measurements -- working, environment, symptoms
Exposure assessments -- hazards, risks, ppm3, duration, chemicals
Necessary Requirements
Take multiple measurements (and take the mean)
Under same conditions, but if not….
Statistical remedies to adjust e.g. age, time of day etc.
Reliable & Validated tool
Defined and Regular variables e.g. problem with stress research
Well defined standard of reference e.g. how far back are you “surveying”?
Validity
Does the survey measure what it says it is measuring
Reliability
Does the survey yield stable
data over time
Quantitative Research Requires . . . .
Numerical / Quantifiable data
Probability-based
Nomothetic (group data not individual data)
Sufficiently large sample size (to detect statistically significant effects)
Randomised sampling of a population (cannot guarantee a random sample)
Statistical analyses of data
Population Samples
Achieving a high response rate to a questionnaire is vital
as helps ensures a normal distribution of responses?
Postal questionnaires rarely get a response rate > 40%
Unless respondents have a vested interest in the outcome
Bias?
Most efficient (best) response rates usually happen when respondents have to
do very little to take part in the study
Multiple phase projects see a depletion in numbers at every stage
Quick “in and out” one-stop approach is best
N of population
A “Normal” Sample
5’6”
5’7”
5’8”
5’9”
5’10” 5’11”
Height
RANDOM sampling
OPPORTUNISTIC sampling
CONSCRIPTIVE sampling
QUOTA sampling
6’
6’1”
6’2” 6’3”
6’4”
Being Practical – how many is a sample?
Student
Pop
I.D
Forces yachting training schools
E.M
Companies using stress counselling
S.M
Divers and ear barotrauma
N.O
Solvent exposure in Myanmar
V.W
Routine flu vaccinations
A.F
Dermatitis in hairdressers
S.M
O.H needs of NHS staff
T.R
NIHL in student employees
I.C
Blood tests in British Army pilots
O.Y
Upstream oil company deaths
A.A
Renal colic in flight deck crew
A.C
Hepatitis B in army regulars and territorials
N
300
150
142
80
900
102
23
14
408
161
254
476
indepth
yes
yes
The Importance of Sample Size
• Forgotten in many studies
• Appropriate size needed to confirm / refute hypotheses
• Small samples can’t detect anything but the grossest difference
• Too large a sample – unnecessary waste of resources
• Ethical considerations – waste of patient time, inconvenience, discomfort
Essential to make assessment of optimal sample size before
starting investigation
Non-Responders just as Important
Postal surveys may accrue poor response rates (e.g. 20%) from pop.
May need to re-write to pop. to re-recruit bigger sample
Inefficient to write to all pop. again
Need to re-write to non-responders and NOT responders
Impossible in anonymous studies with no linkage
Can be done with confidential studies
Diminishing returns of multi-stage recruitment
Researcher
Potential Sample
1000 people
540 consents
540 questionnaires
Under-powered study
n = 210
Response rate of 21%
210 questionnaires
Encryption devices
Steganography
Secret communication of a message by hiding it’s existence
Steganos, meaning covered. Gk
Graphein, meaning to write, Gk
If message is discovered it is easily read because of no encryption
Cryptography
Secret communication of a message by hiding it’s meaning
Kryptos, meaning hidden. Gk
Message established using a known protocol, to be decrypted by the receiver
Steganography & Cryptography can be combined together if needed
Steganogrpahy arouses less suspicion in questionnaire respondents
Encryption devices
Steganography
“This example of steganogrpahy may not work very well when projected onto a
large screen, but it works very well on paper, such as questionnaires. Hopefully
many of you will not notice the method of steganography used in this piece of
text. Gosh how clever I am……”
Cryptography
drbjh kbdltpo, jotujuvuf pg pddvqbujpobm ifbmui
The above text (containing an encrypted name and address) looks suspicious
and may be obliterated by the respondent
drbjh kbdltpo, jotujuvuf pg pddvqbujpobm ifbmui
Selection Bias
Sampling properly is Crucial
Samples may be askew
Specialist publications attract a specialist response group
Exists a self-selection bias of those with special interests
Bird Flu
Call
Centres
Controversial topics, or litigious areas
Gulf War
Syndrome
Depleted Uranium Weaponry
Organophosphate Pesticides
Stress
Telecomms
THIS IS AN INHERENT PROBLEM WITH
HEALTH RESEARCH
COMBAT IT WITH LARGE SAMPLES
AND CLEVER METHODOLOGY
Structure of Surveys
Identifying Items
Title
Preamble
Instructions
Research Items
does not need to be honest
some deception is necessary
shorter surveys better than long surveys
Identifying items & Demographics
Preliminary questions
Collecting info necessary for screening:
recording keeping
tracking
tracing
data manipulation
Ask only for relevant info – unethical to “harvest data”
Fewer items minimizes chance of alienating respondent
The need for rich info
to improve the study
Subject’s need for
privacy & anonymity
Preamble
An important introduction
Frame of reference
Without a FOR a respondent may base their answer on a wrong context
“We would like to know if you have had any medical complaints
and how your health has been in general, over the past few weeks.
Please answer ALL the questions on the following pages by ticking
the answer which best applies to you. Remember that we want to
know about present and recent complaints, not those that you had
in the past. It is important that you try to answer all questions”
Title
Not to be underestimated – try to be user-friendly
Sets the tone for the respondent
The General Health Questionnaire vs The GHQ 28
“Simple language for simple people”
Beware of abbreviated titles - may alienate respondents
e.g.
ASS -
Agricultural Satisfaction Scale
LEI –
Life Events Inventory
BDI –
Beck Depression Inventory
MMPI – Minnesota Multiphasic Personality Inventory
HAD – Hospital Anxiety Depression scale
Are Respondents Telling the Truth?
Use a sprinkling of LIE DETECTOR items to assess reliability
e.g
“Have you ever taken anything without asking permission?”
“Did you ever lie to your parents as a child?”
“Have you ever visited a pornographic web-site?”
Respondent
“Outrage”
increases
with item
intimacy
Decide what to do with any respondent who “fails” the lie detector
Keep them in?
Exclude immediately?
Retain but “flag” them?
Response-Set Bias
Enables collection of detailed data easily
Build a detailed data set from a simple binary item
Randomise the layout / order of responses “true / false”
e.g
I eat ice cream
I like ice cream
I enjoy ice cream
I love ice cream
I hate ice cream
 True
 True
True
True
True
False
False
False
False
False
Respondents ticking the same items
Swap the “True & False” responses sequence
Likert scales
A visual linear scale for rating purposes
happy
sad
X
happy
option 1 (measure from start)
sad
option 2 (measure from median)
Swap around the words to avoid response-set bias
Avoid using numbers on the scale can be (mis)leading
Likert scales
XX
X
happy
X
XXX
XX
X
X
X
X
X
XX X
X
X
X
X
sad
-.75
2
1
0
-1
-2
Likert scales
Bend your data and avoid fence-sitters
XX X
happy
XX
X
X
XX X X
X
sad
Design Pointers for Surveys
Avoid colloquialisms or abbreviations
Beware of local expressions
Avoid words with double meanings
e.g.”Fair....Dip....Lie....Well”
Set a definition of specific terms
e.g. “OK.....Average”
Avoid long questions
Specify exact time, place and context
e.g “At school, did you ever...”
Avoid impossible denial e.g “When did you first...”
Avoid numbers on your responses / scales
Have items seemingly related to the research topic
Design Pointers for Surveys
Clearly phrase items
Make items unambiguous
Avoid leading items
e.g. “Many people think...”
Ask only what a respondent is qualified to answer
Avoid socially loaded items
e.g. “beggars” or “junkies”
Do not use socially / religiously biased items
Avoid phrases, clichés or sooth-sayings
Ask for ages as a true number / D.O.B - - NOT in age groups
Easy Fonts
Pictures (not smiley faces!)
Some Common Errors in Surveys
“In your organization, do women have the same responsibilities as
men, and should women have more?”
“Preventing accidents in the workplace is vital, and more money should
be spent on prevention.”
“Training in risk assessment is something I would like to do, and I
would like to see my colleagues do it to.”
Only ask one question per item
Some Common Errors in Surveys
What is your age ? (please tick)
16 - 26
26 - 36
36 - 46
46 - 56
What is your marital status ? (please tick)
Married Single
How many blood splashes have you had ? (please tick)
1-5
6 - 10
11 - 15
15 - 20
20 plus
Are life preservers and flares essential on-deck equipment ?
agree
disagree
Factorial approach
The General Health Questionnaire (GHQ 28)
A self-completion questionnaire assessing mental health
How?
28 items
7 about Anxiety
7 about Severe depression
7 about Dysfunction
7 about Somatic symptoms
Anxiety score
Depression score
Dysfunction score
Somatic score
Global score
3
4
5
2
14
By summing the 4 factors there is a Global Mental Health score
Statistics are performed on the factor scores and the global score
NOT on each individual item
Increasing Response Rates
Incentives
Appearance
Delivery
Origin
Contact
Content
Communication
Edwards et al. 2002
Incentives
Money
Vouchers
Prize draw
Ethical aspects
Bias
O.R
Monetary incentive vs. None
2.02
Incentive with Q. vs. Incentive on return
1.71
Non-monetary incentive vs. No incentive
1.19
Edwards et al. 2002
Appearance
O.R
Shorter format vs. Longer format
1.86
Brown envelope vs. White
1.52
Coloured ink vs. Black
1.39
Folder / Booklet vs. Stapled pages
1.17
Personalised vs. Not personalised
1.16
ID feature on return vs. No ID
1.08
Coloured Q vs. White Q
1.06
Edwards et al. 2002
Delivery methods
O.R
Recorded delivery vs. Standard
2.21
Stamped return envelope vs. Business reply / franked
1.26
Q sent to work vs. Q sent to home
1.16
1st class outward mail vs. Other class
1.12
Pre-paid return envelope vs. Not pre-paid
1.09
Stamped outward envelope vs. Franked
0.95
Commemorative stamp vs. Ordinary stamp
0.92
Edwards et al. 2002
Origin & Contact
Origin
O.R
University vs. Other organisation
1.31
Sent by senior persons vs. Juniors
1.13
Ethnically ambiguous name vs. Non-white name
1.11
Contact
O.R
Pre-contact vs. No contact
1.54
Follow up vs. No follow up
1.44
Postal follow up with Q vs. Without Q
1.41
Mentioning follow up vs. None
1.04
Pre-contact by telephone vs. Postal pre-contact
0.90
Edwards et al. 2002
Content
O.R
More interesting vs. Less interesting
2.44
User-friendly vs. Standard
1.46
Factual items only vs. Factual & attitude items
1.34
Relevant items first vs. Other items
1.23
Demographic items first vs. Other items
1.04
“Don’t know” boxes vs. no “Don’t know” boxes
1.03
Sensitive items vs. No sensitive items
0.92
General items first vs. Last
0.80
Edwards et al. 2002
Communication
O.R
Explain drop-out required vs. Not
1.32
Stresses benefit to respondent vs. Others
1.06
Stresses benefit to sponsors vs. Others
1.01
Stresses benefit to society vs. Others
1.00
Response deadline given vs. No deadline
1.00
Instructions given vs. No instructions
0.89
Choice to opt out given vs. No opt out
0.76
Edwards et al. 2002
Human Nature Then . . . . .
Cash incentives work
Want cash upfront not contingent
Like to feel important
Little time to spare
Not motivated by benefit to any other people
Lazy
Survey Summary
1. Surveys widely used in data collection
2. Mark pre-pay or addressed envelopes
3. Steganography better than cryptography
4. Perform analyses on factors not individual items
5. Identifying items / demographics usually useful in later analyses
6. Think about scoring before rushing ahead with survey
7. Store questionnaires securely until passing any viva voce / publication
8. Collect as much info at source, parse it down later at discretion
Survey Summary
9. Perfect for epidemiological studies and health research
10. Non-response to postal questionnaires reduces sample & introduces bias
11. Identification of effective ways to increase postal response rates
12. Use existing metrics - pilot items if making new questionnaire
13. Use factor analysis
14. Keep it all as brief as possible
15. Don’t alienate respondents
16. Alternate types of items
Final Points – Bias does not ruin everything
Bias
Avoiding bias is a good aim to have
Existence of some bias in a sample does not ruin a project entirely
Mostly leads to underestimation rather than overestimation of main effects
Spector et al, (2000) shows the “inflating effect” of self-report bias may not be
so prominent
Spector PE, Chen PY, O’Connell BJ. A longitudinal study of relations between
job stressors and job strains while controlling for prior negative affectivity
and strains. Journal of Applied Psychology 2000; 85: 211-218.
Final Points – Generalizability is not everything
Basic principle:
Internal validity is always more important than its generalizability
Never appropriate to generalise an invalid finding
Mant et al. (1996)
Mant J, Dawes M, Graham-Jones S. Internal validity of trials is more important
than generalizability. British Medical Journal 1996; 312: 779.