Transcript NDC Household Survey 2004/5 - Sheffield Hallam University

NDC Household Survey 2006
An overview
Rachel Williams, Jessica Vince and Leon Page
Session objectives
Background information about this year’s household survey
Review data outputs available and give practical tips on reading
and interpreting data
The Extranet (www.ipsos-mori.com/ndc)
Sources of help at Ipsos MORI
2
Background to the
household survey
3
Overview
Survey aims to provide information on the key outcomes from the
NDC programme and to show change over time
Third survey in the series
 initial survey in 2002
 2004 and 2006 surveys are longitudinal – going back to as many of
the original 2002/4 respondents as possible plus a cross-sectional topup sample
Ipsos MORI/GfK NOP conducted 15,792 interviews between May and
October 2006 – c.400 in each NDC area
National benchmarks - from existing surveys and new Omnibus
survey – are NDC areas ‘catching up’?
4
Changes for 2006
Sample size reduced to 400 interviews
 Longitudinal and top-up addresses issued in the same proportion
as 2004
Questionnaire length reduced:
 25 minute core questionnaire
 Partnership specific questions optional
No research among movers, beneficiaries or business
5
Questionnaire
6
Core questionnaire – asked in all
NDC areas
Core questionnaire covers each “theme” in national evaluation
 Housing and the local environment
 Crime
 Worklessness
 Education
 Health
Plus profile information, basic income questions and general
perceptions of area/community
7
Core questionnaire – 2
Wherever possible questionnaire kept identical to 2002/04 to allow
changes in views to be compared
Designed with NET – following outcome review and review following
each wave
Full pilot in similar regeneration areas each wave
Some questions ‘follow-up’ answers given by respondents in 2004
e.g. Last time you were very worried about being burgled now you
are not very worried – why have your views changed?
Includes national benchmarks to compare against the country as a
whole
This year each partnership could add 5 minutes worth of their own
questions
8
Sampling
9
Why do we sample?
Not feasible to question all residents of a particular area – e.g.
17,000 residents in the East Brighton NDC area alone
Interview a smaller number on the basis that they share the
same characteristics as the population generally
Sampling is making an inference about a…
Population…
…from a…
 Works but relies on strict procedures
…Sample
10
Bigger is better…but size isn’t
everything
Bigger samples are bet t er - but not proport ionat ely
95% confidence intervals
20
15
10
5
0
0
Base:
samples
100
200
300
400
500
600
700
Sample
size
Sample
size
800
900
1000
1100
Source: MORI
11
Household survey – two samples
Longitudinal
sample
Return to respondents interviewed
in 2004
Top-up
sample
New addresses selected at random
to top up to 400 per area
12
Longitudinal sample
Longitudinal
Return to individuals interviewed in 2004 and
attempt to interview them
If named respondent has moved or died then
randomly select a substitute
10,770 successful interviews at original
addresses
13
Top–up sample
Top-up sampling involves a number of steps…
Random selection of addresses from within each
of the NDC partnerships
Random selection of one property/dwelling or
household unit at each sampled address
Random selection of one adult within each
selected household (5,022 successful interviews)
Means every person has an equal chance of being interviewed
14
Sample structure - example
EXAMPLE – Salford NDC
400 original
2004
addresses sent
out
Interviewed 283
people at these
addresses
238 original
respondents
Select 237 new
addresses
- 124 new
respondents
45 new
respondents
407 interviews in total – 238 longitudinal and 169
top-up
15
Fieldwork
16
Fieldwork
Fieldwork May - October 2006
Full briefings for all interviewers (28 sessions)
 dummy interviews before starting work
 where possible using the same interviewers as 2002/4
Advance letter and FAQ sheet sent to respondents
Interviews conducted using Computer Assisted Personal
Interviewing (CAPI)
Minimum of 6 calls, including two at weekend or evenings, plus
further re-issues.
Quality control - back-checking and accompaniment
Over 370 interviews conducted in other languages
Response rates improved on 2002 and 2004
17
Household survey
outputs
18
Household survey outputs
Range of outputs provided by Ipsos MORI team




Key indicators (most accessible)
Topline questionnaires
Computer tabulations (simple sub-group analysis)
SPSS dataset (most ‘technical’)
Will suggest how and when you might want to use these
Extranet
19
What are key indicators?
Show summaries of some key questions from the survey
Easy to read if showing to non-researchers in your NDC
Useful if you want a quick overview of the results and change
over time
Can be used to create a summary of findings for internal use
e.g. Board meetings etc.
Cover all the theme areas (Crime, Housing etc)
20
What are key indicators?
Theme heading
Columns giving partnership and
aggregate data for all three waves, and
national benchmark
Summary of all those
very/fairly satisfied with
accommodation. Does not
show % dissatisfied
21
What are toplines?
Show all of the questions from the survey in the order these
were asked on the questionnaire
All responses to each question are shown (e.g. very and fairly
satisfied/dissatisfied) not just summaries
Questions are grouped by theme to make them easier to find –
e.g. CR questions (CR1, CR2 etc) are from the Crime theme
Use the marked-up questionnaire if you want overall results but
do not want to analyse by sub-groups e.g. men vs. women
Fairly easy to read for those not experienced in research
22
What are toplines?
Theme heading and
question number
Full question text
List of all responses to the
question
23
What are toplines?
Filter – an indication of
who is asked the question
Base – a definition of who
was asked the question and
how many people this
includes
National source – most recent benchmarking
figures available (mostly for England)
24
Toplines - Interpretation
Number of pitfalls to be aware of when reading and interpreting
toplines:
 Check the bases underneath the question – is everyone being
asked or not?
 Some demographic questions are based on all household
members rather than all respondents. Gives more accurate
information on age, gender and work status
 Read the introductory page!
25
Toplines - functionality
Topline document is now in Excel rather than Word
Advantages
 Display only question text
 Display only benchmarked questions
 Quickly calculate change between years using formulas
=IF(H17="*",0,H17) etc
 Run charts from the data
Will be available in PDF version
Working on formulas to produce automated significance testing
(TBC)
26
Statistical reliability - 1
Not all differences between 2002, 2004 and 2006 data will be
significant
E.g. if satisfaction with area has increased by 2 percentage
points between 2004 and 2006 this will not mean that there has
been a significant increase in satisfaction
Whether changes are significant also depends on how many
people were asked a question
The fewer people asked a question, the greater the difference
will have to be for this to be significant
E.g. more difficult to find significant difference where 50
respondents have been asked compared with where 400 have
been asked
27
Statistical reliability - 2
Where all respondents have answered a question need
differences of:
 8 percentage points or more between 2002, 2004 and 2006
figures to be sure they represent actual change and are NOT
due to chance
 6-7 percentage points between individual NDCs and the
aggregate or national benchmark figures
Can use our “ready reckoner” – available on the Extranet
An example…
28
Statistical reliability calculator
Testing change in proportion feeling unsafe between 2004 and
2006 for Partnership X
29
What are computer tabulations?
All responses to the survey are broken down by around 40
different sub-groups
Sub-groups are respondents who are grouped together in the
data because they have something in common. This may be
demographic (e.g. their age) or attitudinal (e.g. they want to
move out of the area)
Examples of sub-groups are:
 Men (demographic)
 Those who earn less than £100 per month (demographic)
 Those who have heard of NDC (attitudinal)
30
What are computer tabulations?
Why would you want to use these?
You may want to see if all sections of the population are
benefiting equally from NDC
E.g. fear of mugging may be going down overall, but are any
groups being left behind?
Can compare 2006 results with 2004 results – young people
may be 2% less likely to be mugged than in 2004 compared
with a drop of 15% overall
Can also use for baseline/monitoring data – are Black residents
more likely than average to be earning £100 a month or less?
31
What are computer tabulations?
Question on tenure – who are most likely to be private renters?
16-34 year olds significantly more likely to rent
privately (text shown in bold)
32
What are computer tabulations?
BUT: Crucial not to compare sub-groups with small bases as these are unreliable
Only 13 people in overall base
Can’t compare refugee
status of men vs. women
33
Computer tables - significance
testing
Letters used to identify which results are significantly different
from each other
Results for male respondents are in
column “a”, females in “b”
“x” = total column
34
Computer tables - significance
testing
Each column is tested against the total to see whether the
differences in results are significantly different
 a letter under a result indicates it is
More in column “b” (females) and column “d”
(35-44 year olds) are lone parent families than
average (i.e. column “x”)
Fewer in column “a”
(men), column “e” (4564 year olds) and
column “f” (aged 65+)
are lone parent
families than average
35
A few things to watch…
Some tables run over two pages because the list of possible
answers is so long – i.e. 4 pages per question (e.g. WO1)
As well as the actual answers that can be given, also include
derived summaries at the end of the list (eg WO1 ILO
Unemployed)
There may be tables for the same question but with different
bases. E.g. at WO1 there are tables based on all respondents
and also on all working age respondents
Sometimes have ‘summary tables’ where a battery of
questions are asked
There are a few derived tables e.g. HO3 cross-tabulated by
HO4
36
SPSS data
A software program that allows you to conduct your own
analyses (Statistical Package for the Social Sciences)
Allows you to do a wide range of analyses that are not included
on the Computer Tables
E.g. You could find out the characteristics of ‘Trapped residents’
– those who want to move but do not think they will, by crosstabulating two survey questions against each other
Every NDC has their own dataset
File contains some additional derived variables eg workless
households
BUT – your NDC may not have bought SPSS software so you
may not be able to use this data (LA may have it though …)
Ask the Ipsos MORI team if you need any help with using SPSS
37
A note on weighting
Two sets of weights applied to the survey data
 Selection weighting to correct for unequal selection
probabilities (every person should have equal chance of
selection)
 Profile weighting to ensure that survey is representative of the
population as a whole (based on Census)
All data presented in the survey outputs is WEIGHTED
EXCEPT, the final column on the computer tables - this
presents the raw data
Correct weights must also be applied to the SPSS data
NB weighted samples are less accurate – need to bear in mind
when checking for significant differences
38
Extranet
39
Extranet - 1
You may have used this with the 2002 and 2004 data
Analysis tool taken off-line recently to add the new 2006
data
Can still access survey documentation
2006 data will be available in February next year
40
Extranet - 2
Extranet allows you some of the flexibility of an analysis
programme like SPSS but easier to use
Can analyse your partnerships’ results by demographic factors
such as age or gender
Can compare your results against those of other relevant
partnerships and against the aggregate data set
Can filter results e.g. the findings just for women aged 16-24
Can compare change in data over time
41
Extranet - 3
An on-line resource containing all NDC data
Run your own tables on issues of interest
Download toplines, key findings and computer tables
Data available for all partnerships
Possible to compare yourself with other “local” NDCs or those
with similar characteristics
Each partnership has their own log in
Benchmarking data and links to useful sites
Information about the household survey
 Background to each wave
Glossary of research terms and training slides
42
Some on-screen
examples
43
Using the data
44
Using the data
Background information/reality check along with other sources
of data
 eg if feelings of safety have increased how does this relate to
crime statistics/what the police are saying locally.
Partnership specific questions used to evaluate specific
projects
Post coding data to provide neighbourhood level information
Wider area benchmark surveys
45
Thank you
For further information or help please contact:
[email protected] (020 7347 3148)
[email protected] (020 7347 3152)
[email protected] (020 7347 3252)
46