Transcript PowerPoint-presentatie

Response quality in the Structural
Business Survey questionnaire
Deirdre Giesen & Joep Burger
Q-Conference, June 2-5 2014, Vienna
Outline
•
•
•
•
•
•
•
Why this research project?
Some background on SBS
Previous results and current step
Data used
Quality indicators
Results
Issues for discussion
Purpose research project
1. Develop quality indicators for quality “raw data” of
business survey.
2. Explore how this quality is related to
– characteristics response process
– data collection design
Possible use of such indicators
– Allow quick response during data collection (also: direct
feed back to respondents during completion of
electronic questionnaire).
– To guide further research and adjustments in design
data collection.
Structural Business Survey
– Annual survey
– 2nd largest business survey at SN (about 70000 business
units in sample)
– Collects detailed specification of income and costs of
businesses
– Most industries and size classes included
– Used for European SBS and Dutch National accounts
– About 200 versions of the questionnaire: common core
part with size-class/industry specific specifications
– Redesign introduced in 2006
5
Our previous research (1/2)
– General quantitative indicators developed that can be
used for all types of SBS:
‐ Item response to 5 core variables
‐ Set of consistency rules
‐ Use of “other costs” item
– Related these to time (before / after redesign), size of
business, type of industry, timeliness response, mode,
response burden
6
Our previous research (2/2)
– Overall quality of indicators was rather high
– Not many systematic and large differences related to
background variables studied however:
‐ Smaller businesses perform worse than larger
businesses
‐ Some improvement on item response and
consistency after introduction of electronic
questionnaires
‐ After redesign in question block on personnel one
core variable clearly higher item response, other
clearly worse.
7
Current step of research
– Focus on one unique questionnaire, which allows
detailed analyses of the quality of cost specifications.
– Selection: general manufacturing questionnaire for size
class 5 ( 20-49 employees).
– For this questionnaire: pilot study with introduction of
electronic questionnaire in 2005, introduction of total
redesign in 2006.
– Data for 2003-2007 analysed, about 1400 respondents
each year.
8
Example of calculation in SBS qnr
9
Quality indicators
1) Quality of calculation (6 sets defined)
2) Overall item response: fraction of subitems used
3) Item response on set of 11 core variables
4) Fraction of total value specified for each core variable.
10
Quality of calculations
11
Overall item response (costs)
12
Item response core variables
13
Fraction of total value specified
14
Conclusions
– Demonstrates methodology to quickly assess quality of
this type of raw data.
– Seems useful to test if and how indicators can be used
for
- individual feed back to respondents.
- standard monitoring of data collection.
- Illustrates that also for “hard business data”
questionnaire design affects response process.
- Limited use of indicators for evaluation of design
characteristics without experimental data.
15
Questions, ideas?
– What aspects of quality in raw data in business surveys
do you monitor and/or feed back to respondents?
16