Transcript Slide 1

Planning and carrying out (or commissioning) a local survey
Principles and Practice,
Processes and Pitfalls
Steve Wisher
Information by Design
E: [email protected]
T: 01482 467 411
1
Overview
 Planning Surveys
 Survey Design
 Sampling
 Further Information
2
Planning Surveys
Practicalities

What to include

What survey method to use

How big a sample, and how to select it

Should we do it (all) internally or commission it out

How much will it cost

How fast can it be done

What about ethical approval

Other aspects
4
Postal Survey Process
Finalise Research Objectives
Clear well defined research objectives
improves research clarity
Design Sample
Identify who is to be sampled, and what
sampling method to use
Draw Sample
Create a mailing file of all target groups to be
mailed with postal questionnaire
Fieldwork Period
Mailing phase, administer
questionnaires to sample selected
from mailing file
Mailing 1 – First questionnaires
Data capture on going
as questionnaires arrive
1st Reminder
(4 weeks after initial Mail out)
2nd Reminder
2/3 weeks after first reminder)
Clean and Prepare data for analysis
Reporting Phase
Report on findings of survey
5
Contracting Out - Preparing a Specification
To obtain useful and comparable quotations and proposals you
should prepare a specification for the survey. This should include:
 Background to your organisation and its activities
 Background to the survey and your needs
 Objectives of the survey and how the findings will be used
 Work required from the organisation
 Methodology
 The outcomes you expect to be achieved
 (Possibly) an estimate of the budget involved
 Who will be responsible for the project within your organisation
 Timescales
6
Survey Design
Deciding on the Survey Method



Main methods used in Health and Lifestyle surveys……
Postal, Face-to-face, Telephone, Web
Compare and contrast…..Group exercise 1
8
POSTAL
Arguments For
Arguments Against
FACE-TO-FACE
Arguments For
Arguments Against
TELEPHONE
Arguments For
Arguments Against
WEB
Arguments For
Arguments Against
9
Postal Surveys
Arguments For
Arguments Against
Cost – they are cheaper than face-to-face or telephone surveys,
but more expensive then web-based.
Response Rates – well-designed, well-managed postal surveys
can deliver good response rates. However, response rates
in recent years seem to be falling.
Resource – less resource needed for data collection than face-toface and telephone. No need for trained team of
fieldworkers. Data entry team needed.
Response Bias – certain groups tend to have lower response
rates eg. young adults, younger men, some BME groups.
Ensure you act to reduce the impact of this at the design
and analysis stage.
Time – quicker than face-to-face interviewing; about the same as
telephone; web-based is fastest. Time needed for data
entry phase.
Question Wording – needs to be simple and easily understood
(no opportunity for consistent clarification as in interview
surveys).
Geography – can be inclusive (in a cost effective manner) for
remote communities. Allows geographical stratification
more easily.
Literacy – not suitable for residents with low levels of literacy or
where there are language barriers. Appropriate use of
transitions and telephone help-lines can assist those with
low levels of English literacy.
Honesty – potential for more open responses than face-to-face /
telephone. (Be careful to limit personal questions or to
invade privacy)
Respondent – there is no guarantee that the named resident has
actually completed the questionnaire.
Respondent – method is less intrusive than face-to-face or
telephone. Allows respondent to complete questionnaire in
their own time.
Design – need to be simple to complete, limited chance to use
additional materials.
10
Two Primary Objectives in Conducting a Reliable Lifestyle Survey

Increasing precision and reducing bias

Precision is about how precise or accurate your results are; how
narrow the confidence intervals are for estimates from your survey

Bias is about whether your survey really is representative of the
population of residents.
11
Designing the Questionnaire

Qaire is to translate information needed into questions that
the respondent can and will answer

Qaire should motivate and encourage the respondent to
become involved in the interview – to cooperate, to complete
the interview

Qaire should minimise response error (e.g. when the
respondent gives inaccurate answers, or when answers are
mis-recorded or mis-analysed
12
Researcher Vs Respondent
What does Respondent want?
 Reward
 Confidentiality
 Interesting subject
 Interesting experience
 Personal benefits from completing
the survey
 Social benefits from competing the
survey
 Being ‘chosen’ as respondent
with expertise on the subject
 Research organisation with
excellent reputation
 Rapport and trust
What does the researcher want?
 Honesty
 Cooperation – accepts the reason
for the research
 Follows the instructions and
completes the survey
 Respondent thinks through the
issue before responding
 Respondent feels good from
completing the interview or qaire
 And wants to do it again!
13
Qaire Design Process
Specify the information needed
Specify the data collection method
Determine content of individual questions
Overcome the respondents inability and unwillingness to answer
Choose question structure
Choose question wording
Arrange questions in proper order
Check layout
Pilot the qaire
Test Reliability
and Validity
14
Reliability and Validity












Reliability = The stability of the measurement scale
How far will it give the same results on separate occasions?
….concerned with the ‘measurement error’
Stability – the extent to which the same results are obtained on repeated use
of the instrument
Internal Consistency – are the sub-parts of the scale measuring the same
characteristic?
Equivalence – do different ‘raters’ get the same answer?
Validity = Does the scale measure what you intend it to measure?
Concerned with the meaning and interpretation of the scale
Content validity – has it included all relevant items?
Face validity – does the scale look reasonable?
Criterion Validity – how does the scale compare with an established
measure?
Construct Validity – how does the scale correlate with the construct under
investigation?
15
Group Exercise 2

Design a question to measure smoking prevalence….
16
Measuring Well-being
For this question, we ask you to tell us how satisfied or dissatisfied you are with different parts of your
life, by using a scale of 0 to 10.
On a scale of 0-10, with 0 being very dissatisfied and 10 being very satisfied, how would you rate
each of the following statements? Please Tick  One Box For Each Statement
Very
Dissatisfied
0
1
Very
satisfied
2
3
4
5
6
7
8
9
10
Your life as a whole
Your personal relationship with family and friends
Your work situation
The area where you live
Your leisure time
17
Overcoming Inability or Unwillingness to Answer

Is the respondent informed?

Can the respondent remember?

Is the respondent able to articulate?

Effort required by respondent?

Legitimate purposes?

Sensitive information?
18
‘Rules of Thumb’ – Postal Qaire











Use a booklet format if possible
'Brand' the questionnaire on the front cover. Give the survey a name that is
likely to appeal to the respondent.
Do not crowd the questions
Use large type
Number all questions
Split the questionnaire into sections - give a short introduction to each
section
Put clear instructions on the front of the questionnaire. Give examples of
how to complete a question if relevant.
Put questions that are easy to answer and ‘non-challenging’ to the
respondent at the beginning of the questionnaire.
It is best to get the respondent interested as early as possible in the
questionnaire – put interesting topics early – it will result in better cooperation from the respondent.
Questions should be presented in a logical order to the respondent.
If filter questions are used, make sure the routing to later questions works.
19
Postal Surveys - Qaire Length and Response Rates





Subject matter – an interesting range of topics gets better response rates.
Style and layout of the questionnaire – good formatting gives better
response rates generally
Question ordering – keep more difficult or personal questions until later;
start with topics that interest the respondent
Question style – avoid too detailed questions, or questions that require too
much recall. Long lists tend to be boring to respondents
Questionnaire length – this is important, but other factors are probably more
important.
20
Sampling
Sample Designs

Most Lifestyle Surveys are based on a sample rather than a census
– notable exceptions…. Lifestyle of Older People in West Sussex

Most use some form of stratified random sampling

Little detail provided in most Lifestyle Survey reports….
22
Practical Example – Stratified Random Sampling

Step 1: Decide on the stratification factors you think are most
appropriate in your survey. These might be electoral ward, age
or gender; or some combination of these.

Step 2: Say you have chosen to stratify by age. Order the
database (think of it as a long list) by age or age group –
youngest at the top, oldest at the bottom.

Step 3: Select systematically down the list. For example
(depending on the required sample size), select every 20th
person from the list. The sample selected will then have the right
proportion of residents from each age (or age band). So for
example, if 20% of the database is over 65, then 20% of the
sample will be over 65
23
How Big a Sample?

The big question!

Use a formula?
2
N=(1.96) (p(1-p)
2
B

Sample size is usually determined by the need for analysis at subgroup level
24
Sample Size and Sampling Error
Sampling error(%)
Sample size
1
10,000
1.5
4,500
2
2,500
2.5
1,600
3
1,100
3.5
816
4
625
4.5
494
5
400
5.5
330
6
277
6.5
237
7
204
7.5
178
8
156
8.5
138
9
123
9.5
110
10
100
25
Estimating Sampling Errors
Percentage found by the survey
Sample Size
5% or 95%
10% or 90%
20% or 80%
30% or 70%
50%
100
4.4
6.0
8.0
9.2
10.0
200
3.1
4.2
5.7
6.5
7.1
500
1.9
2.7
3.6
4.1
4.5
1000
1.4
1.9
2.5
2.9
3.2
2000
1.0
1.3
1.8
2.0
2.2
5000
0.6
0.8
1.1
1.3
1.4
10000
0.4
0.6
0.8
0.9
1.0
26
Example – 5% precision in Northumberland Young Peoples Survey
B
9 & 10
Alnwick
Berwick
Blyth Valley
Castle Morpeth
Tynedale
Wansbeck
G
119
112
229
163
185
184
Total:
121
105
221
165
181
182
11 & 12
B
G
119
105
225
167
176
186
115
105
226
166
180
175
13 & 14
B
G
118
98
236
169
183
183
119
100
234
164
186
182
15 & 16
B
G
127
108
237
173
187
185
118
103
240
154
187
177
7879
Samples for analysis by district and by age:
9 & 10
Alnwick
Berwick
Blyth Valley
Castle Morpeth
Tynedale
Wansbeck
Total:
246
233
326
289
300
301
6679
11 & 12
13 & 14
15 & 16
248
266
245
228
200
169
332
320
297
289
291
280
303
305
293
301
319
299
Samples for analysis by District and sex
B
G
Alnwick
Berwick
Blyth Valley
Castle Morpeth
Tynedale
Wansbeck
305
274
350
330
337
341
Total:
Sample size
increases as more
‘cells’ for analysis
(with same level of
precision in results)
304
271
350
327
338
340
3866
Samples for analysis by Dsitrict:
Alnwick
Berwick
Blyth Valley
Castle Morpeth
Tynedale
Wansbeck
340
319
366
354
359
361
Total:
Sample for total population:
379
2099
27
Getting the Sample
Various Options…..

Exeter Data

PAF

Names and Numbers
28
Exeter Data
 Patient Records – from GP records
 Variables include:
name, address etc
 DOB, gender
 GP

‘Out of area’ registrations – e.g.
from those who work in the area


• Can be used to Stratify the Sample
Posttown
Valid
AYLESFORD
CHATHAM
DARTFORD
FAVERSHAM
FOLKESTONE
GILLINGHAM
GRAVESEND
HERNE BAY
LONGFIELD
MAIDSTONE
RAMSGATE
ROCHESTER
SHEERNESS
SITTINGBOU
SNODLAND
WEST MALLI
Total
Frequency
165
78459
1
6
1
96358
271
1
6
89
1
91836
14
3597
215
1
271021
Percent
.1
28.9
.0
.0
.0
35.6
.1
.0
.0
.0
.0
33.9
.0
1.3
.1
.0
100.0
Valid Percent
.1
28.9
.0
.0
.0
35.6
.1
.0
.0
.0
.0
33.9
.0
1.3
.1
.0
100.0
Some issues with accuracy –
e.g. moved house but not
registered with new GP

Cumulative
Percent
.1
29.0
29.0
29.0
29.0
64.6
64.7
64.7
64.7
64.7
64.7
98.6
98.6
99.9
100.0
100.0
Confidentiality concerns from
PCT
Ethical issues – sharing the data
with an external organisation


Need for confidentiality/data
processing agreement to be
established
29
What Response Will We Get?

Response rates have fallen in recent years

Previously (<2002) response rates of 60% or higher. Now 30% is common

2 short case studies….
30
Case Study – Kent 2005 Survey of Health and Lifestyle
Sampling Frame
Sampling Method
GP registration list from Kent Primary Care Agency
Random – one-in-fifty of population
Data Collection
Postal Survey. 3 Reminders: post card at 4 weeks, additional questionnaire at 6
weeks, additional postcard at 8 weeks. The survey was also available online
– very few people responded this way.
Sample Size
Total returns
Achieved response
22,861 aged 16+
5,800
27%
Best response rates were achieved from people in their middle years, with younger people, especially men under 45,
being less likely to complete the questionnaire
31
Case Study – Southampton Health & Lifestyle Survey 2006
Sampling Frame
Sampling Method
Patient and Practitioner Services Authority (PSA) register (residents registered with
a GP).
Stratified random sampling.
Data Collection Method
Postal, 2 reminders including questionnaires.
Sample Size
Target
Total returns
Achieved response
5,020 residents aged 16+ (in 2 phases).
100 returns in each of 16 wards.
1,777
38%
Response gives city-wide estimates to within +/- 2.3% (95% confidence level).
Response also allows analysis for priority areas within the city.
32
General Issues…..




If I use a postal survey method, how will I maximise the
response?
How should I avoid bias - in which certain groups fail to
respond?
How should I ensure 'hard-to-reach' groups are
represented in the survey response?
Are postal survey methods appropriate where literacy
levels are low?
33
Monitor the Response – Helps Improve Response
Daily Returns
120
100
80
60
103
96
40
83
78
72
70
66
20
32
7t
h
6t
h
0
21
17 19 16
23
23 21
12
16
11
6
9
14
5
19
24 25
29
23
6
12 11
10
2
6
3
2
4
8
4
1
2
3
2
3
8t
h
11
th
12
th
13
th
14
th
15
th
18
th
19
th
20
th
21
th
22
nd
25
th
26
th
27
th
28
th
29
th
2n
d
3r
d
4t
h
5t
h
6t
h
9t
h
10
th
11
th
12
th
13
th
16
th
17
th
18
th
19
th
20
th
23
rd
24
th
25
th
26
th
27
th
20
37
32
34
Balancing Sample Size and Cost
Small Scale
Objective
Larger Scale
To obtain Local Authority level estimates of key
lifestyle measures
To obtain ward level data on lifestyle
measures
Sample size (mailing)
3,000
15,000
Questionnaires returned
1,000
5,000
50
250
£10,000
£40,000
(£10 per response)
(£8 per response)
±3%
±1.5%
Approximate returns per Ward
(assuming 20 wards)
Cost
Approximate sampling error at LA
level
35
Sources of Error
Total Error
Random sampling error
Non-sampling errors
Response errors
Non-response errors
Proxy Information
Researcher errors
Measurement/Qaire
Population definition
Sampling frame
Data analysis
Interviewer errors
Respondent selection
Questioning
Recording
Cheating
Respondent errors
Inability
Unwillingness
Pleasing the interviewer
36
More Information
The Updated SEPHO Lifestyle Survey Toolkit
Soon to be found at http://www.lifestylesurvey.org.uk/
38
North West PHO Work on core questions http://www.nwpho.org.uk/documents/
39
PH INTELLIGENCE TRAINING COURSE Day 7:
Surveys and Modelling
Local Case Studies
Steve Wisher
40
Postal Surveys Case Studies
Case Study 1 – Leicester Lifestyle Survey
Name of Lifestyle
Survey
Who
commissioned it?
When?
Who conducted it
(if different from
commissioners)?
Sample sizes
used? Include total
mailout size;
response rate etc.
Sampling method
used?
Survey method(s)
used?
Leicester Lifestyle
Survey
Leicester City Health
Action Zone
Gfk NOP
Almost 9,500
mailed, 3,340
returns (38%
response rate)
Random sample
drawn from electoral
register
Self-completion
postal questionnaire
42
Leicester Lifestyle Survey - The Response…weighting
required to correct for response bias
43
Case Study 2 - Bradford Lifestyle Survey 2007-08
Name of Lifestyle
Survey
Who
commissioned it?
When?
Who conducted it
(if different from
commissioners)?
Sample sizes
used? Include total
mailout size;
response rate etc.
Sampling method
used?
Survey method(s)
used?
Bradford Lifestyle
Survey
Bradford and
Airedale PCT;
2007/08
Information by
Design
15,000 mailout,
response c5,000
Stratified random;
stratified by ward,
gender and age
Postal, with
telephone and postal
reminders
HEALTH
and
LIFESTYLE
In Bradford and Airedale
We are asking people in the whole of Bradford and Airedale to
give us a picture of their health and lifestyle.
This will help us to plan local services which meet your needs.
Tell us about you!

The person who is named on the letter enclosed should complete this.

Please follow the instructions for each question carefully. Some of them ask you to tick one
box, and some more than one box.

Everything you tell us will be treated in the strictest confidence – we definitely won’t pass
your details on to anyone else.

Please return your completed questionnaire to Information by Design in the envelope
provided. They are collecting the information for us.

If you have any questions please telephone Information by Design on Freephone
0800 0112932.
Return this and your name will be entered into a prize
draw to win a top prize of a £100 voucher
and a further twenty £10 high street store vouchers
44
Extract from the Questionnaire
About You
This part just asks you to tell us a little bit about yourself.
1
Are you male or female? Please Tick  One Box
Male
2
Female
How old are you?
 Please write your age in years in the box
3
Are you: Please Tick  One Box to the right of the answer you mean
Single
(never
married)
Married
(first
marriage)
Re-married
Divorced
Separated
(but still
legally
married)
Widowed
Living
with a
partner
About Your General Health
The following section asks some general questions about your health.
4
In general, would you say your health is Please Tick  One Box to the right of the answer you mean
Very Good
5
Good
Fair
Bad
Very Bad
Do you have any long-term illness, health problem or disability which limits your daily activities or the work you
can do? Please Tick  One Box
Yes
No
6
Go to Question 7
If yes, what is this, and how does it limit your daily activities or the work you can do? Please write the
condition and how it limits your daily activities in the boxes below
Condition
How it limits your daily activities
45
Extract from the Data File
46
School-based Surveys
Schools Health Education Unit (SHEU) Surveys
www.sheu.org.uk
48
Example Data
60%
40%
20%
0%
1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005
M 12-13y 0.26
0.32
0.31
0.31
0.3
0.29
0.29
0.26
0.26
0.25
0.27
0.26
0.25
0.21
0.21
0.24
0.2
0.23
0.18
0.24
0.26
0.5
0.5
0.53
0.54
0.51
0.43
0.42
0.39
0.38
0.37
0.37
0.34
0.35
0.3
0.31
0.34
0.34
0.33
0.27
0.32
0.34
M 14-15y 0.19
0.21
0.21
0.2
0.16
0.19
0.18
0.16
0.15
0.16
0.18
0.15
0.14
0.16
0.14
0.15
0.14
0.15
0.12
0.16
0.16
0.35
0.35
0.38
0.34
0.34
0.31
0.3
0.26
0.25
0.25
0.24
0.2
0.21
0.19
0.2
0.22
0.2
0.21
0.18
0.21
0.2
F 12-13y
F 14-15y
49