Transcript Slide 1
Evaluating behaviour change programs
Liz Ampt Concepts of Change
What are we measuring?
- Whether people are doing things differently
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
How can we measure it?
- Measuring changes in levels of reducing/ reusing/ recycling - Observing/recording behaviours - Asking about change - Each is a form of survey or data collection exercise WasteMinz Roundup 2014was WasteMINZ Roundup 2014
Preliminary Planning Selection of Survey Method Survey Instrument Design Sample Design Pilot Survey Conduct of Survey Data Coding Data Editing Data Correction and Expansion Data Management and Analysis
The survey process
Richardson, Ampt, Meyburg 1995
T idying-Up
WasteMinz Roundup 2014was
Presentation
WasteMINZ Roundup 2014
Each is a form of survey
Method of measurement Aspects of survey design needed
Measuring changes in levels of reducing/ reusing/ recycling Observing/recording behaviours Asking about change - Sample selection - Pilot - Survey - Expansion/weighting - Analysis - Sample selection - Pilot - Survey - Expansion/weighting - Analysis - Sample selection - Survey design - Pilot - Survey - Expansion/weighting - Analysis WasteMinz Roundup 2014was WasteMINZ Roundup 2014
Key elements of survey design
1. Preliminary planning 2. Selection of method 3. Sample design 4. Survey instrument design 5. Pilot 6. Survey implementation • 7. Expansion/weighting
Analysis – over to you..
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
1. Preliminary planning
• • • • Define survey objectives – Very specific: what, by whom, over what period, where Review of existing information – Useful methodologies from elsewhere; use of stated preference?
Define terms – From your objectives and from respondent’s perspective Survey content – Dot points 7 WasteMinz Roundup 2014was WasteMINZ Roundup 2014
Exercise
• Think of 2 terms you would want to use in a survey related to your work – Write down clear definitions for both – Ask another person (not from your organisation) to do the same with your terms – Compare WasteMinz Roundup 2014was WasteMINZ Roundup 2014
2. Selection of a Survey Method for Measurement
• • • • • • Observation surveys Intercept surveys Self-administered surveys Telephone surveys Personal interview surveys Internet/online surveys 9 WasteMinz Roundup 2014was WasteMINZ Roundup 2014
Observation methods
• Chosen when – Possible to count accurately – Possible to count all or select a representative sample • Can be manual, automatic, video 10 WasteMinz Roundup 2014was WasteMINZ Roundup 2014
Intercept Surveys
• • • Intercepting people – At an activity centre (e.g. workplace, transfer station, shopping centre) Possible methods – distribution - mail-back/on-line – personal interview – collect phone no.
Always needs a total classification count 11 WasteMinz Roundup 2014was WasteMINZ Roundup 2014
Intercept Surveys
• • Advantages – Able to reach specific populations – Can combine with observational counts – Can use multiple survey methods Disadvantages – Generally low response rates (20-30% for self-completion) – Hurried conditions – – Must allow for non-random sampling No follow-up possible in most cases 12 WasteMinz Roundup 2014was WasteMINZ Roundup 2014
Self-administered Surveys
• • • Possible targets – households – activity centres/workplaces/transfer stations Method of Distribution – mail-out vs. hand delivered Method of Collection – mail-back vs. hand collection 13 WasteMinz Roundup 2014was WasteMINZ Roundup 2014
Self-administered Surveys
• • Advantages – Can get extensive geographical coverage – No interviewer effects – – Can obtain considered responses Hand-collection to good response rate Disadvantages – Layout and wording must be clear - hard to design – No probing possible – – Answers not independent Response rates lower than face to face 14 WasteMinz Roundup 2014was WasteMINZ Roundup 2014
Telephone Interviews
• Advantages – Wide geographic coverage – Intermediate costs – – Good supervision - CATI Multilingual capabilities – Computerised WasteMinz Roundup 2014was WasteMINZ Roundup 2014
Telephone Interviews
• Disadvantages – Sample usually weak • Low phone ownership for some groups • Answer-phones, mobile phones, screening devices • Hard to know how it represents the population – Credibility of interviewer (confusion with telemarketing) – Low response rate – No follow-up for refusals WasteMinz Roundup 2014was WasteMINZ Roundup 2014
Personal Interviews
• • Can be paper or computer Advantages – Generally higher response rates (60-90%) – Flexibility of information – Presence of interviewer – Maintain interest – Spontaneous answers WasteMinz Roundup 2014was WasteMINZ Roundup 2014
Personal Interview
• Disadvantages – High costs – Interviewer influence • personal characteristics • interrupt household/work routine • opinions of interviewers • interpretation of vague answers – Considered response difficult WasteMinz Roundup 2014was WasteMINZ Roundup 2014
On-line Surveys
• • Advantages – Low costs – Can use elaborate visual effects – Can use adaptive techniques (can give different scenarios for different responses) – Good for workplaces if sufficient follow-up Disadvantages – – Usually very biased sample Low response rate – Hard to get all people in household if needed WasteMinz Roundup 2014was WasteMINZ Roundup 2014
Exercise
• • Think of a behaviour you would like to measure Discuss with a partner – Best method of collection • Strengths • Weaknesses WasteMinz Roundup 2014was WasteMINZ Roundup 2014
3. Sample Design in the Survey Process
Preliminary Planning Selection of Survey Method Survey Instrument Design Sample Design Pilot Survey Conduct of Survey Data Coding Data Correction and Expansion Data Editing Data Management and Analysis T idying-Up Presentation of Results
WasteMinz Roundup 2014was WasteMINZ Roundup 2014 Sampling Methods
Preliminary Concepts
• • • • What is a sample?
– a collection of things which is some part of a larger population and which is selected so as to be representative of some or all of that population Target Population – who are we trying to survey?
Sampling Units – what are we going to sample?
Sampling Frame – where are we going to get a list of these things?
WasteMinz Roundup 2014was WasteMINZ Roundup 2014 Sampling Methods
Sampling Frame
• • • a base list to identify the sampling units should contain all the sampling units examples, – all households on a street (e.g. Council records) – telephone directories – mailing lists – maps – electoral rolls – blocklists WasteMinz Roundup 2014was WasteMINZ Roundup 2014 Sampling Methods
Sampling Frame Problems
• • • • • inaccuracy incompleteness duplication inadequacy out-of-date • Must check the reason for which the list was originally compiled to understand likely deficiencies.
WasteMinz Roundup 2014was WasteMINZ Roundup 2014 Sampling Methods
Sampling Error & Sampling Bias
• Sampling Error – due to the simple fact that we are taking a sample, and not the population. – can minimise error by taking larger sample.
• Sampling Bias – due to systematic omission of some elements from our final sample.
– cannot minimise error by taking larger sample.
WasteMinz Roundup 2014was WasteMINZ Roundup 2014 Sampling Methods
Random Sampling
• Each unit is selected
independently
and each unit in the population has an
equal probability
of being selected.
• Must use random sampling to avoid sampling bias.
WasteMinz Roundup 2014was WasteMINZ Roundup 2014 Sampling Methods
Random Sampling Methods
• Simple Random Sampling • Stratified Random Sampling • Variable Fraction Stratified Random Sampling • Multi-stage Sampling • Cluster Sampling • Systematic Sampling •
Note quotas not on list
WasteMinz Roundup 2014was WasteMINZ Roundup 2014 Sampling Methods
Sample Size
• How much data do we need?
• • Too much data >>> too expensive Not enough data >>> not able to draw conclusions • Somewhere in the middle is a sample size which enables us to draw sufficient conclusions at a reasonable cost
Stopher, P. (2012) Collecting, Managing and Assessing Data Using Sample Surveys , Cambridge.
WasteMinz Roundup 2014was WasteMINZ Roundup 2014 Sampling Methods
Exercise
• Hand out random sampling sheet – Explain – Questions?
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
4. Survey design in the survey process
Preliminary Planning Selection of Survey Method Survey Instrument Design Sample Design Pilot Survey Conduct of Survey Data Coding Data Correction and Expansion Data Editing Data Management and Analysis T idying-Up Presentation of Results
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
Instrument design for reliable measurement
• Question content • Question types • Physical design - also for observation/counting WasteMinz Roundup 2014was WasteMINZ Roundup 2014
Question Content
• • • Reliability – repeatable – easy to answer Accuracy – – no question bias measures what we want Relevance – must
appear
relevant to respondent WasteMinz Roundup 2014was WasteMINZ Roundup 2014
Question Types
• Factual – “What did you do?” • Classification (e.g. socio-demographic) – for comparing with secondary data • Opinion and attitude questions – “What do you think about ……?” • Stated Response Questions – “What would you do if ……?” WasteMinz Roundup 2014was WasteMINZ Roundup 2014
Physical Design of Forms/Apps
• Observational surveys – Ergonomic – Size/format – not too big or small – Weather-proof – Need log forms – Test under actual conditions WasteMinz Roundup 2014was WasteMINZ Roundup 2014
•
Physical Design of Forms
Self-administered forms – Layout vital – Minimal writing should be required – – No coding aids should appear Instructions very clear – Professional appearance – Include ID number 35 WasteMinz Roundup 2014was WasteMINZ Roundup 2014
5. Pilots in the survey process
Preliminary Planning Selection of Survey Method Survey Instrument Design Sample Design Pilot Survey Conduct of Survey Data Coding Data Correction and Expansion Data Editing Data Management and Analysis T idying-Up Presentation of Results
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
Pilot Surveys
• • Why not do a pilot survey?
– too expensive – not enough time Why do a pilot survey?
– – too expensive to omit it not enough time to omit it WasteMinz Roundup 2014was WasteMINZ Roundup 2014
Pilot Surveys
• • • pilot survey is a test of ALL aspects of design scope for experimental design saves expensive mistakes WasteMinz Roundup 2014was WasteMINZ Roundup 2014
Uses of the Pilot Survey
• • • • • • "Skirmishing" of wording Adequacy of questionnaire – definitions clear?
– too many "don't knows“?
– too long?
– open to closed questions Efficiency of interview/surveyer training Non-response rate Analysis Cost and duration WasteMinz Roundup 2014was WasteMINZ Roundup 2014
6. Survey implementation
Preliminary Planning Selection of Survey Method Survey Instrument Design Sample Design Pilot Survey Conduct of Survey Data Coding Data Correction and Expansion Data Editing Data Management and Analysis T idying-Up Presentation of Results
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
Conducting the Surveys
• • Need high response rate for validity Gross sample . . . . . . . . . . . . . . . . . . = 100 (houses who are eligible to put out bin) Sample loss (vacant, invalid phone no.) = 5 (vacant houses) Net sample . . . . . . . . . . . . . . . . . . . . . . Responses . . . . . . . . . . . . . . . . . . . . . . Response rate . . . . . . . . . . . . . . . . . . . . = 95 = 50 (put out green bin) = response/net sample = (50/95) = 53% Consider – Announcement letter/message >> higher response – Follow-up regime >> higher response rate WasteMinz Roundup 2014was WasteMINZ Roundup 2014
Exercise – dot points for a survey you need
• • • • What method?
How to get a sampling frame?
What questions? Need any for weighting?
Other issues/questions?
– – Richardson, Ampt, Meyburg (1995)
http://www.geog.ucsb.edu/~deutsch/geog111_211a/code_bo oks/Survey_Methods_For_Transport_Planning.pdf
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
7. Weighting (correction)/Expansion
Preliminary Planning Selection of Survey Method Survey Instrument Design Sample Design Pilot Survey Conduct of Survey Data Coding Data Correction and Expansion Data Editing Data Management and Analysis T idying-Up Presentation of Results
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
Weighting & Expansion of Data
• • Getting the sample data to represent the population from which it was drawn, as nearly as possible Why? – systematic errors – Non-response >> weighting certain type of respondents higher – Missing Data >> can make assumptions – or note – Inaccurate Reporting >> e.g. social desirability bias WasteMinz Roundup 2014was WasteMINZ Roundup 2014
Example of weighting
• Your response is 50/95 – what about the 45?
Say 30 males (67%) 15 females (33%) - Your secondary data (e.g. counts, other data) males 50% females (50%) – missing responses from females - Responding females are therefore ‘weighted’ with a slightly higher value (1.5) males (.75)
Stopher, P. (2012) Collecting, Managing and Assessing Data Using Sample Surveys , Cambridge.
WasteMinz Roundup 2014was WasteMINZ Roundup 2014
Summary
• • • To measure behaviour change it is essential to understand the data collection and survey process In particular need to understand: – Survey method – Sampling principles – Importance of a pilot – Implementation options – Need for weighting Vital for future funding as well as for sharing methodologies WasteMinz Roundup 2014was WasteMINZ Roundup 2014
WasteMinz Roundup 2014was WasteMINZ Roundup 2014