Transcript An introduction to surveys – the basics

An introduction to surveys – the basics
U3A Cambridge Summer term
July 4th 2014
Jill Tuffnell
Scope & coverage
• My credentials
• Overview of statistical platform & key
concepts: confidence interval; confidence
level and sample size
• Survey design and methodology
• Response rates
• Examples
• Questions
My background
• Degrees in economics & statistics; operational
research
• Work for National Coal Board, local
government & public policy consultancy
• Headed Research Group at Cambs County
Council; specialist in labour market, Census
analysis and involvement in many surveys –
ranging from new settlements to travellers to
fire-fighters to footpaths
(Brief) overview of statistics (i)
• Kit of tools enabling us to measure, interpret
and analyse data – quantitative & qualitative
• Sample surveys based on probability theory
• Sample size has a non-linear relationship with
the population being sampled
• The ‘best’ surveys require every member of a
population to have the same, random chance
of being surveyed; important to measure
response rates; also applicable to Censuses
(Brief) Overview of statistics (ii)
• Confidence interval: produce a result which is,
for example within + or – 5%, or + or – 3%
• Confidence level or margin of error: measures
how ‘sure’ we can be: i.e. that we are 95%
certain
• Sample size: generally the larger the sample,
the lower the confidence interval
• See sample size calculator
(Brief) overview of statistics (iii)
• Example: 5,000 in your total population
• Question on, e.g. involvement in volunteering
• Want a sample to generate a 95% confidence level within
a + or – 2.5% confidence interval
• Requires a random sample giving 1,176 responses
• Actual sample numbers must account for non-response
• But a population of 75,000 would only require a response
by 1,506; 1,000,000 population: 1,534
• These sample response numbers must apply to all subpopulations of interest – e.g. women, aged over 70
Survey design
• Depends on ‘population’ being sampled: e.g.
– Households
– Population
– Businesses
– Footpaths
• Sampling frame (total ‘population’):
– Electoral register (now restricted access)
– Address list
– Telephone book – BT list very restricted; random dialling?
– Business rates; Companies House; Inter Departmental
Business Register
– Record of public rights of way
Main types of survey
•
•
•
•
•
•
Random or quota
Face to face: interviewer
Postal – written questionnaire
Telephone
Online/email
Critical to measure who responds and who
does not; non-response is a major issue and
can seriously impact on results
• Must adjust results for non-response
Face to face
• Randomly-chosen from total population of interest best –
but ‘quota’ often used, e.g. for street surveys
• Collect information on key demographics such as sex,
age, ethnicity, proxy for socio-economic group, such as
job or tenure
• Can adjust results for response rates from different
sectors
• Household-based surveys generally most robust, but
expensive: must involve ‘call-back’ for non-response
• Problems with restricted access to electoral rolls &
coverage of electoral rolls
• Language issues; also rarely cover children
• If you can afford it – the ‘Rolls Royce’ of surveys
Postal
• Requires comprehensive and up-to-date address list (students
& private lets an issue!); no national Address List
• Problems with multiple-occupancy homes & institutions
• Issue of language important
• Non-response very high; how to attract? Often require second
and third postings, so time frame can be long
• Offer incentives, such as inclusion in prize draw; promise of
feed-back on results
• Always provide stamped, addressed envelopes
• Keep as short as possible
• Known response bias – owner-occupiers, professionals,
elderly{ high; private renters, unemployed, young{ low
Telephone
• Most adults have a phone
• But there is no comprehensive sampling
frame/population, even for landlines
• Landline inadequate – no coverage of young people
who mainly use mobiles
• Text surveys now developing
• Response bias – so has to be accounted for
• Random dialling?
• Best suited to marketing – but increasingly used for
short surveys, especially by Political Parties
On-line surveys
• Data protection may prevent access to emails
• Bias due to user profile – against elderly, poor &
children
• Have to adjust for differential response rates
• Survey overload an increasing problem
• More use of incentives (e.g. entry to prize draw)
• Free software (e.g. Survey Monkey) suggests surveys
are easy – but can be awful!
• Cheap to run, as data can be downloaded easily into
survey analysis programs
Example – National Trust ‘visitor experience survey’
• Moved from written questionnaire of visitors to on-line
survey, based on a sample of visitors drawn from the barcodes
of membership cards ‘zapped’ and then emailed where
possible
• Discovered ‘drop’ in satisfaction levels
• But the card survey was biased – mainly offered to ‘happy’
smiling visitors!
• Survey restricted to existing members – rather than the wider
audience of ‘one-off’ visitors and prospective members; not
very helpful if you want to find out how to attract more
members
• NT ‘VE’ targets for properties are set very high
Questionnaire design (i)
• Identifying what to include: initial use of ‘focus’
group(s) to bring out key issues valuable
• But beware recruitment of the ‘loony’ brigade!
• Order of options/questions important – most people
lose interest/concentration towards the end; can
rotate order
• Try to avoid giving an uneven number of choices –
e.g. excellent, good, average, poor, very poor – as
people tend to go for the middle option: ‘average’
• Avoid ‘leading’ questions: ‘You enjoy your visits to
Wimpole Hall, don’t you?’ Yes or No
Questionnaire design (ii)
• Many people will avoid the top & bottom choices in a
range; therefore unrealistic for the NT to set targets
based on ‘top’ responses alone (i.e. scores on ‘very
satisfied’ alone, not counting ‘satisfied’). 7 rather
than 5 options?
• Change order of questions over a survey as a whole
• Always provide respondents the opportunity to
complete an ‘any other comments/views’ section
• No point in carrying out a survey to support a predetermined approach – a waste of money!
• Keep it as short as possible & feed back results
Dealing with response bias
• Respondents do not fully mirror the population sampled –
response bias
• Even the Population Census (100% of homes) has a biased
response (low on young men living in shared, rented homes)
• If you know the true population shares you can adjust, e.g.:
– Elderly home owners are over-represented
– Young private renters in multiple-occupancy homes underrepresented
– Social renters under-represented
– Unemployed under-represented
– Small service businesses under-represented in business
surveys
Example of interesting surveys (i)
• Future housing/site needs of Travellers living in parts of the
East of England, 2007 for next 15 years
• Very expensive! Involvement of ARU and University of
Buckingham specialist academics
– Face to face only realistic option
– Identified key Traveller with respect of the community(ies)
– Recruitment and training of Travellers to carry it out
– Had to be person to person because of illiteracy
– Used hand-held recorders
– Issues of women only interviewing female travellers and
men interviewing males
– Questions restricted due to community concerns
Examples of interesting surveys (ii)
• Cambourne new settlement in Cambs: surveyed in 2006
• 100% of households; known private/social split
• Postal, with prize draw & free post-back; two waves required to enhance
social-rented sector responses (recorded questionnaires by code)
• Quantitative & qualitative issues covered
• Critical demographics on who moved in: sex, age, tenure, jobs,
occupations, place of work, previous home location & tenure; compared
with Census & administrative data to identify response bias
• Also views on the community and how it was developing; short-comings;
plus points, why people moved here
• Feeds into planning for other new communities, such as Northstowe
• All new housing estates in Cambs surveyed since 2007
• Results available on Cambs CC website
Cambourne survey results 2007
• People attracted by Comberton Village College
secondary catchment – brought in far more families
with children aged 9+ than expected
• Identified more ‘private-renters’ than expected
• High levels of commuting to Papworth and
Cambridge; also London commuters via St Neots
• Mainly professionals, with most adults working
• Residents liked the environment overall
• But were fed-up with living on a building site, with
few community facilities provided as promised
Implications for U3AC surveys
• Likely to be a biased response from members: but we
are the demographic group most likely to complete
surveys!
• Should we be surveying eligible non-members? How
do we identify and approach them? Offer incentives?
• Use Census 2011 data, compared with our
membership list, by age/sex/ qualifications /tenure.
This is generally of high quality!
• Use on-line surveys as most potential new members
will have email
Your questions/views
• Over to you!