Transcript Measuring Research and Experimental Development

Innovation Surveys:
Advice from the Oslo Manual
South Asian Regional Workshop on Science, Technology and Innovation Statistics
Kathmandu, Nepal
6-9 December 2010
www.uis.unesco.org
Ch 8 OM - Innovation Survey
Procedures
 Guidelines - collection and analysis of innovation
data;
 Comparable results;
 Particular
circumstances may
methodology  comparability!
require
other
www.uis.unesco.org
Populations
 The target population:
• Business enterprise sector (goods-producing and
services industries);
• At a minimum, all statistical units with at least ten
employees.
www.uis.unesco.org
Populations
 The frame population:
• Units from which a survey sample or census is drawn
= frame population;
• Basis: last year of the observation period;
• Ideal frame = up-to-date official business register
established for statistical purposes  NSOs;
• If the register forms the basis for several surveys
(innovation, R&D, business), the information collected
can be restricted to issues specific to innovation.
www.uis.unesco.org
Survey methods
 Mandatory surveys increase response rates;
 Census or sample survey?
• Sample surveys should be representative of the
basic characteristics of the target population (industry,
size, region)  a stratified sample is necessary;
• Census, though costly, might be unavoidable in some
cases (legal requirement, small frame population,
inclusion of all units in the frame with a certain number
of employees).
www.uis.unesco.org
Survey methods
 Domains
(sub-populations) are subsets of the
sampling strata;
• Potential sub-populations: industry groupings, size
classes, regions, units that engage in R&D and
innovation-active;
• Guidelines:
» Same statistical units and classifications;
» Consistent methods;
» Documentation of deviations in data treatment or differences in
the quality of the results (from the domains).
www.uis.unesco.org
Survey methods
 Sampling techniques:
• Stratified sample surveys (reliable results): based on
the size and principal activity of the units;
• Sampling fractions should not be the same for all
strata: the sampling fraction of a stratum should be
higher for more heterogeneous strata and for smaller
strata.
 Cross-sections: standard approach - new random
sample drawn from a given population;
 Alternative/supplementary approach: panel data.
www.uis.unesco.org
Survey methods
 Suitable respondents:
• Various
methods: postal surveys,
surveys, personal interviews;
electronic
• Questions are very specialised and can be answered
by only a few people in the unit;
• It is highly recommended to make a special effort to
identify respondents by name before data collection
starts.
www.uis.unesco.org
Survey methods
 The questionnaire:
• Pre-test before fieldwork;
• Simple and short;
• Order of the questions;
• Questions on qualitative indicators - binary or ordinal
scale;
• International
innovation surveys:
design of the questionnaire;
translation
and
• Short-form
questionnaires for units with little/no
innovation activity previously reported.
www.uis.unesco.org
Survey methods
 Combination of Innovation and R&D surveys:
 Reduction in the overall response burden;
 Scope for analysing the relations between R&D and
innovation activities at the unit level;
 Efficient method of increasing the frequency of innovation
surveys;
 It is possible to obtain reliable results for R&D expenditures;
 Longer questionnaire;
 Units not familiar with the concepts of R&D and innovation
may confuse them;
 The frames for the two surveys are generally different.
www.uis.unesco.org
Survey methods
 Guidelines for conducting combined surveys:
• The questionnaire should have two distinct sections;
• Individual sections for R&D and innovation should be
smaller than in separate surveys;
• Comparisons of results from combined surveys with
those from separate innovation surveys should be done
with care;
• Surveying methods should be reported;
• Samples to carry out such surveys should be extracted
from a common business register.
www.uis.unesco.org
Estimation of results
 Weighting methods:
• The simplest one is weighting by the inverse of the
sampling fractions of the sampling units, corrected by
the unit non-response;
• Stratified sampling technique with different sampling
fractions  weights should be calculated individually;
• Commonly based on the number of enterprises in a
stratum;
• In international and other comparisons, be sure that
the same weighting method is used.
www.uis.unesco.org
Estimation of results
 Non-response:
• Unit non-response: a reporting unit does not reply at
all;
• Item
non-response: response rate to a specific
question / % of blank or missing answers among the
reporting units;
» Disregarding missing values and applying simple weighting
procedures based only on the responses received implicitly
assumes that non-respondents are distributed in the same
way as respondents  biased results;
» Possibility: imputation methods to estimate missing values
on the basis of additional information.
www.uis.unesco.org
Presentation of results
 Descriptive analysis:
• Description
of
the
statistical units in terms of
their innovative or noninnovative
activities
without
drawing
any
conclusions about the
underlying survey or target
population;
• No generalisation of the
 Inferential analysis:
• Drawing
of conclusions
about the target population;
• The results should give a
representative estimation
of the situation;
• Weighted results;
• Unit non-response rate is
very important.
results;
• Unit non-response rate is
of minor importance.
www.uis.unesco.org
Presentation of results
 Errors:
• Random errors due to the random process used to
select the units;
• Systematic errors containing all non-random errors
(bias);
» Results’ variance: it is recommended to calculate both
(average) values for innovation indicators and also their
coefficients of variation and/or confidence intervals;
» Results presentation: metadata (including information on data
collection procedure), sampling methods, procedures for
dealing with non-response and quality indicators.
www.uis.unesco.org
Frequency of data collection
 Innovation surveys: every two years;
 When not economically feasible  three or four
years;
 Surveys must always specify an observation
period for questions on innovation;
• The length of the observation period for innovation
surveys should not exceed three years nor be less
than one year.
www.uis.unesco.org
Annex A - 5. Methodological issues
for developing country contexts
 Information system specificities:
• Relative weakness of statistical systems
• Absence of linkages between surveys and data sets;
lack of official business registers  information
from other surveys cannot be used;
• Involvement of NSOs;
• When
lacking, basic variables about firms’
performance can be included in the innovation survey
- to enable further analysis.
www.uis.unesco.org
Annex A - 5. Methodological issues
for developing country contexts
 General methodological considerations:
• Questionnaire design:
» Separated sections - different respondents;
» Guidance / definitions;
» Language and the translation of technical terms;
• Survey application:
» In-person;
» Trained personnel.
www.uis.unesco.org
Annex A - 5. Methodological issues
for developing country contexts
 General methodological considerations:
• Frequency:
» Every three to four years (e.g., timed to CIS rounds);
» Try to update a minimum set of variables every year;
• The purpose of surveys needs to be clearly stated
and the questions clearly formulated;
• An adequate legislative base for the collection of
innovation statistics can help ensure the success of
such an exercise;
• The results should be published and distributed
widely.
www.uis.unesco.org
Thank you!
http://www.uis.unesco.org
[email protected]
(CIS: http://www.oecd.org/dataoecd/37/39/37489901.pdf)
www.uis.unesco.org