Transcript Quantitative Social Survey Design

Structured Surveys in
Social Research
What are surveys?


One method of collecting, organising analysing data
Data collection - is structured and systematic




Data analysis - looking for systematic variation between
variables and cases
Variable by case matrix form – like experimental method


Info. on same variables from multiple cases (people, groups, orgs.)
Both quantitative and qualitative data
But more difficult to attribute differences to experimenter
intervention
Multiple techniques Questionnaires most common;

But also: structured and in-depth interviews, direct observation
techniques etc.
Survey v Participatory methods
Survey
Participatory
Strengths
Statistical inference
Stakeholder empowerment?
Qualitative &Quantitative Rapid ‘quick & dirty’ method
Generalisability
Potentially less resources req.
Flexible and highly adaptive
Weaknesses
Intrinsically manipulative
-Power to practitioners
-Ideological persuasion
Iterative rigidity
Causal connections?
Simplicity results in misuse?
Mostly qualitative methods
Reliability & generalisability
Short term fashion?
The Steps in a Survey Project







Establish the goals of the project - What you want to
learn
Determine your sample - Whom you will interview
Choose interviewing methodology - How you will
interview
Create your questionnaire - What you will ask
Pre-test the questionnaire, if practical - Test the
questions.
Conduct interviews and enter data - Ask the
questions.
Analyze the data - Produce the reports.
Descriptive and Explanatory
Research 1

Survey research has two fundamental questions:

What? – Descriptive


i.e. Social structure, market research, opinion poll
Why? – Explanatory
Theorising
 Grounded / ex post facto theory
 Efficient data collection and analysis

Descriptive and Explanatory
Research 2

Descriptive research – focusing the topic
Time frame?
 Geographical location?
 Comparison of sub-groups?


Units of Analysis
-Village / Household/
Individual
Explanatory research
Either causes of change
 And / or consequences of change? (i.e. PIM)

Farming
Systems and
Units of
Analysis
Theory Construction & Testing
Obs1
Obs2
Obs3
Obs4
Empirical
level
Theory Construction
Conceptual
abstract level
Theory
Theory Testing
Obs1
Obs2
Obs3
Obs4
Empirical
level
Survey Design 1

Classic design – experimental v control group
Availability of controls?
 Repeated measures for same group?


Panel design – same group over time
Longitudinal survey
 Problem of re-visiting all panel members?


Quasi-panel design – diff groups over time
Survey Design 2

Retrospective panel or experimental design


Based on recall – selective memory?
Cross-sectional design – most common
Matched experimental and non-experimental groups
 Comparison at one point in time


One shot ‘case’ study – most primitive
One group at one point of time
 Limited use for evaluating causal processes

Sampling


Two main types of sample
1. Non-probability samples


Some cases have greater or unknown selection chance
2. Probability samples to achieve representative
sample of population to permit generalisation
Each case has equal or known chance of selection
 Random selection from sampling frame
 Sampling error – standard error

Probability Samples 1

Four types – choice depends on


Resources - Desired accuracy, survey method and
availability of a good sampling frame.
1. Simple random sampling (SRS)
Determine sample size
 Random number table selection
 Requires good sample frame


2. Systematic sampling
Like SRS but simpler – same limitations
 Proportional sampling i.e. every fifth member of
frame – periodicity?

Probability Samples 2

3. Stratified sampling
Select stratifying variable(s) i.e. wealth
 Proportion in different groups same as population
 Requires a priori knowledge of stratifying variable


4. Multi-stage cluster sampling

Area based i.e. random selection of districts, streets
and finally households
Sample Size?

Size depends on two key factors:
Degree of accuracy required (max 5% SE?)
 Population variability re. key study variables





For small samples, a small increase in sample
size can result in a large increase in accuracy
Variability of most heterogeneous variables?
Requirements of statistical techniques?
Allow for non-response – be aware of bias!
Non Probability Samples



Where no reliable sampling frame is available
Or where populations are highly dispersed
Or where no requirement for generalisation
from sample to population
Exploratory analysis
 Hypotheses generation
 Testing questionnaires


Purposive, quota and availability sampling
From Concepts to Questions



Descending the ladder of abstraction!
Clarify concepts – functional v substantive
Develop good indicators
Measurable: No’s & dimensions (space and time)
 Reliability – Consistency in repeated use
 Validity – Ability to measure the concept as intended




Consistency of meaning to different people !
Pilot test or borrow from existing questionnaires
Clarify concepts and indicators with iteration
Explanatory variables



Dependent variables – i.e. farmers level of relevant
knowledge of information and skills and experience.
Independent variables – i.e. farmers learning & info.
gathering methods, socioeconomic status, ethnicity, etc.
– factors which might explain their knowledge of the
dependent variables
Confounding variables – Other factors related to
both dependent and independent factors which may
distort the results and have to be adjusted for.
Open or Closed Format Questions?
Open format questions
 Allows exploration of the range of possible themes arising
from an issue
 Can be used even if a comprehensive range of alternative
choices cannot be compiled





Closed or forced choice-format questions
Easy and quick to fill in
Minimise discrimination against the less literate or the less
articulate
Easy to code, record, and analyse results quantitatively
Easy to report results
Design and Response Rates




Short simple questionnaires usually attract higher
response rates than long complex boring ones.
Adding variety in the types of questions asked will help
Make and give incentives?
Ordering of questions






Go from general to particular.
Go from easy to difficult.
Go from factual to abstract.
Start with closed format questions.
Start with questions relevant to the main subject.
Do not start with personal questions.
Other considerations


Avoid leading and double barrelled questions
Keep language simple (short) and precise


Anticipate interpretational difficulties


By respondents and enumerators – piloting
Appropriate questions – cultural sensitivity


Frame of reference: who and over what time period
Direct and indirect questions
Be aware of prestige bias – i.e. wealth ranking
Cultural
Sensitivity?
Administering the questionnaire
Advantages of self administered questionnaires
 Cheap and easy to administer.
 Preserve confidentiality
 Can be completed at respondent's convenience
 Can be administered in a standard manner
Advantages of interview administered questionnaires
 Allow participation by illiterate, less well educated
people.
 Higher response rate??
 Allow clarification of ambiguity through dialogue
Examples
from
Sri Lanka
Surveys used to measure fish yields
from stocked village reservoirs


Second Phase: 5 reservoirs belonging to 4
villages stocked, 1 control village
1. Direct observation


Collective fishing and staggered harvesting
2. Longitudinal panel survey
7 day recall of household fish consumption
 Fortnightly, 41 wealth stratified households


3. Retrospective panel survey
Participatory impact monitoring (PIM)
 Stakeholder perceptions and yield recall

1. Longitudinal Survey -Mean monthly
per capita consumption by source
1.6
Mean monthly consumption (kg/ capita)
1.4
1.2
1.0
0.8
0.6
0.4
0.2
0.0
Nov 00
Dec 00
Jan 01
Feb 01
Mar 01
Apr 01
May 01
Jun 01
Jul 01
Aug 01
Sep 01
Oct 01
Nov 01
Major irrigation system
1.34
1.26
1.08
0.62
0.73
0.81
0.38
0.34
0.78
0.90
1.13
0.91
1.03
Other village tank
0.09
0.08
0.10
0.11
0.06
0.13
0.27
0.42
0.29
0.23
0.18
0.18
0.24
0.03
0.19
0.07
0.67
0.36
0.24
1.22
0.67
0.11
Stocked village tank
Month
Monthly Yield by Species & Tank
200
180
160
140
Yield (kg)
120
100
80
60
40
20
0
Apr
Jun
Jul
Aug
Sep
GBW
Jun
Jul
Aug
LHG
May
Jun
Aug
Jun
SER
Jul
Aug
Sep
Oct
Jun
Jul
LUN
KBW
Tank / Month
Tilapia Spp.
Channa striata
Trichogaster pectoralis
Cyprinus carpio
Aug
Anabas testudineus
Sep
Oct
Area at
50%
Phase
FSL
Ha
Tank
GURt
IMKt
1
23.1
9.05
48.7
MAD
KBWt
2. Household
/ survey
3. PIM
342
49.4
19.5
12.4
78.6
1.75
SERt
GBWt
LHGt
MAD
LUNt
1. Direct
12.95
IMK SERt
IMK
Yield estimate (kg ha-1 yr-1)
111.1
2
4.21
6.78
59
15.4
1.9
100.7
171
695
0.95
84.6
281.1
Data Management
Data Management

Spreadsheets: Lotus, Excel etc.
Regular ‘symmetrical’ data matrices
 Arithmetic calculations


Databases: Access, FoxPro, File-Maker, Oracle
Relational Design - 3D v 2D : Large, irregular and longitudinal surveys
 Data Accessibility and exploratory analysis



Non-numerical Databases i.e. Nudist, InVivo
Statistical packages i.e. Minitab, SPSS
Spreadsheet Approach
HH
Hse. No.
Head
Name1 Age1 Sex1 Name2
Age2 Sex2
Sunil
26
1
Sunil
34
M
2
Dusan Dusan 55
M
3
Indira Indira 24
F
4
Ranjit Ranjit 47
M
Fatima
Kumar 4
F
M
Database Relational Design
Primary Key
Primary Key
Hse No. HH
Head
Hse No. Name
1
Sunil
Age Sex
34 M
1
Fatima
26
F
2
Dusan
55
M
24
F
1
Sunil
2
Dusan
3
Indira
3
Indira
4
Ranjit
3
Kumar 4
M
4
Ranjit
M
‘One to many’ relationships
47
Database Benefits 1

Ease of Data entry
Organised ‘compact’ tables
 Automated coding systems
 Entry forms look like questionnaire formats
 Multiple Users - online (web pages) or offline
synchronisation


Data entry error checks
Through relational design
 Other built in validation steps

Database Benefits 2

Data Analysis
Demands a priori planning
 Exploratory Analysis before higher level statistical
analysis
 Data reduction - cross-tabulation (Pivot Tables)
 Integrated with MS office suite and statistical
packages - SPSS, Minitab…

Limitations


Exponential learning curve
Dependence on technical backup support



Fallback strategies
Structured design Limits flexibility / iteration
possibilities
Empiricism v anthropological view of human
behaviour as complex sets of meanings