Transcript Measuring Research and Experimental Development

Innovation Surveys:
Advice from the Oslo Manual
National training workshop
Amman, Jordan
18-20 October 2010
www.uis.unesco.org
Ch 8 OM - Innovation Survey
Procedures
 Guidelines on central elements for the collection
and analysis of innovation data;
 Following these guidelines will generally lead to
comparable
countries;
results
over
time
and
across
 Particular
circumstances may require other
methodology  comparability should be in mind.
www.uis.unesco.org
Populations
 The target population:
• Innovation surveys should refer to innovation activities
in the business enterprise sector (goods-producing
and services industries);
• It should include, at a minimum, all statistical units with
at least ten employees.
www.uis.unesco.org
Populations
 The frame population:
• The units from which a survey sample or census is
drawn form the frame population;
• It is based on the last year of the observation
period for surveys;
• Ideal frame = up-to-date official business register
established for statistical purposes  NSOs;
• If the register forms the basis for several surveys
(innovation, the R&D, general business), the
information collected in the innovation survey can be
restricted to issues specific to innovation.
www.uis.unesco.org
Survey methods
 Mandatory surveys increase response rates;
 Census or sample survey?
• Sample surveys should be representative of the
basic characteristics of the target population (industry,
size, region)  a stratified sample is necessary;
• Census, though costly, might be unavoidable in some
cases (legal requirement, small frame population,
inclusion of all units in the frame with a certain number
of employees).
www.uis.unesco.org
Survey methods
 Domains
(sub-populations) are subsets of the
sampling strata;
• Potential sub-populations: industry groupings, size
classes, regions, units that engage in R&D and
innovation-active;
• Guidelines:
» Statistical units and classifications should be the same;
» Methods used to calculate results for subsets should be
consistent with those used for results from the main sample.
» Deviations in data treatment or differences in the quality of the
results from the domains should be documented.
www.uis.unesco.org
Survey methods
 Sampling techniques:
• Stratified sample surveys (reliable results): based on
the size and principal activity of the units;
• Sampling fractions should not be the same for all
strata: the sampling fraction of a stratum should be
higher for more heterogeneous strata and for smaller
strata.
 Cross-sections:
standard approach - a new
random sample is drawn from a given population for
each innovation survey;
 Alternative/supplementary approach: panel data.
www.uis.unesco.org
Survey methods
 Suitable respondents:
• A variety of methods can be used to conduct innovation
surveys, including postal surveys and personal
interviews;
the unit’s most suitable respondent is
particularly important in innovation surveys (questions
are very specialised and can be answered by only a
few people in the unit);
• Choosing
• It is highly recommended to make a special effort to
identify respondents by name before data collection
starts.
www.uis.unesco.org
Survey methods
 The questionnaire:
• Pre-test each questionnaire before fieldwork;
• Keep it as simple and short as possible;
• Pay attention to the order of the questions;
• Questions on a number of qualitative indicators can
use either a binary or an ordinal scale;
• In the case of international innovation surveys, special
attention should be given to the translation and design
of the questionnaire;
• Short-form
questionnaires for units with little/no
innovation activity previously reported.
www.uis.unesco.org
Survey methods
 Combination of Innovation and R&D surveys:
 Reduction in the overall response burden of the reporting
units;
 Scope for analysing the relations between R&D and
innovation activities at the unit level;
 Efficient method of increasing the frequency of innovation
surveys;
 Country experiences indicate that it is possible to obtain
reliable results for R&D expenditures;
 Longer questionnaire;
 Units not familiar with the concepts of R&D and innovation
may confuse them;
 The frames for the two surveys are generally different.
www.uis.unesco.org
Survey methods
 Guidelines for conducting combined surveys:
• The questionnaire should have two distinct sections;
• Individual sections for R&D and innovation should be
smaller than in separate surveys;
• Comparisons of results from combined surveys with
those from separate innovation surveys should be done
with care, and surveying methods should be reported;
• Samples to carry out such surveys should be extracted
from a common business register.
www.uis.unesco.org
Estimation of results
 Weighting methods:
• The simplest one is weighting by the inverse of the
sampling fractions of the sampling units, corrected by
the unit non-response;
• If
a stratified sampling technique with different
sampling fractions is used, weights should be
calculated individually for each;
• Commonly based on the number of enterprises in a
stratum.
• In international and other comparisons, be sure that
the same weighting method is used.
www.uis.unesco.org
Estimation of results
 Non-response:
• Unit non-response: a reporting unit does not reply at
all;
• Item non-response: refers to the response rate to a
specific question and is equal to the percentage of
blank or missing answers among the reporting units;
» Disregarding missing values and applying simple weighting
procedures based only on the responses received implicitly
assumes that non-respondents are distributed in the same
way as respondents  biased results;
» Possibility: imputation methods to estimate missing values
on the basis of additional information.
www.uis.unesco.org
Presentation of results
 Descriptive analysis:
• Description
of
the
statistical units in terms of
their innovative or noninnovative
activities
without
drawing
any
conclusions about the
underlying survey or target
population;
• No generalisation of the
 Inferential analysis:
• Drawing
of conclusions
about the target population;
• The results should give a
representative estimation
of the situation;
• Weighted results;
• Unit non-response rate is
very important.
results;
• Unit non-response rate is
of minor importance.
www.uis.unesco.org
Presentation of results
 Errors:
• Random errors due to the random process used to
select the units;
• Systematic errors containing all non-random errors
(bias);
» To get at least an idea of the variance for the results, it is
recommended to calculate both (average) values for
innovation indicators and also their coefficients of
variation and/or confidence intervals;
» Results
presentation should contain: metadata (including
information on data collection procedure), sampling methods,
procedures for dealing with non-response and quality
indicators.
www.uis.unesco.org
Frequency of data collection
 Innovation should be conducted every two years;
 Where this is not economically feasible, a
frequency of three or four years may be chosen;
 Surveys must specify an observation period for
questions on innovation;
• The length of the observation period for innovation
surveys should not exceed three years nor be less
than one year.
www.uis.unesco.org
Annex A - 5. Methodological issues
for developing country contexts
 Information system specificities:
• Relative weakness of statistical systems
• Absence of linkages between surveys and data sets;
lack of official business registers  information
from other surveys cannot be used;
• Involvement of NSOs;
• When
lacking, basic variables about firms’
performance can be included in the innovation survey
- to enable further analysis.
www.uis.unesco.org
Annex A - 5. Methodological issues
for developing country contexts
 General methodological considerations:
• Questionnaire design:
» Sections can be separated to allow different persons in the
firm to reply them;
» Guidance / definitions;
» Language and the translation of technical terms;
• Survey application:
» In-person;
» Trained personnel.
www.uis.unesco.org
Annex A - 5. Methodological issues
for developing country contexts
 General methodological considerations:
• Frequency:
» Every three to four years (e.g., timed to CIS rounds);
» Try to update a minimum set of variables every year;
• The purpose of surveys needs to be clearly stated
and the questions clearly formulated;
• An adequate legislative base for the collection of
innovation statistics can help ensure the success of
such an exercise;
• The results should be published and distributed
widely.
www.uis.unesco.org
Thank you!
http://www.uis.unesco.org
[email protected]
CIS: http://www.oecd.org/dataoecd/37/39/37489901.pdf
www.uis.unesco.org