Transcript Measuring Research and Experimental Development

Survey methodology and procedures:
General advice from Frascati Manual
National Workshop on Science, Technology and Innovation (STI) Statistics
Abu Dhabi, UAE
14 October 2012
www.uis.unesco.org
Sources of this presentation
 Chapter 7 of the Frascati Manual - R&D Survey
Methodology
 Section 8 of the Annex to the Frascati Manual
(Measuring R&D in developing countries) Strengthening R&D Statistical Systems
 Chapter 7 of UIS Technical Paper no. 5
(Measuring R&D in developing countries) Strengthening R&D Statistical Systems
www.uis.unesco.org
Institutionalization of STI statistics
 Political support
 Infrastructure and sustained staff
training/capacity building
 Involvement of NSOs: “Official statistics”
status for R&D surveys.
 Adequate legal framework
www.uis.unesco.org
User-producer networks
Recommendations:
• User-producer networks and other forms of stakeholder
consultation should be instituted.
• Establishing national S&T statistics groups.
• Involve multiple actors.
• Coordinating/networking among institutions/databases.
• Partnering with business associations.
• Conducting face-to-face visits by statisticians and project leaders.
• Exploit pre-existing personnel ties.
• Get NSO involved; to deal with privacy of information.
• Training of interviewers/primary data producers.
www.uis.unesco.org
General issues
 Statistics on R&D require regular, systematic and
harmonised special surveys
 Other sources provide information, but:
• concepts of R&D used often different from FM concepts
• concepts may change over time
• very difficult to obtain all data for the same period
• difficult to avoid double counting when tracking flows
from financial statements and other sources
 Estimates are a necessary supplement to surveys
• especially in higher education sector
www.uis.unesco.org
Scope of R&D surveys
 R&D surveys should identify and measure all
financial and personnel resources devoted to all
R&D activities in all R&D units
 R&D surveys are mainly addressed to R&Dperforming units
 Chapter 7 of the FM only addresses performerbased surveys
 Statistical methodologies and other procedures
have to be established to capture all R&D,
especially for units in the business enterprise
sector with little R&D
www.uis.unesco.org
Identifying target population and survey
respondents – General issues
 Exhaustive survey not possible in most countries
 Constraints include:
• number of respondents may have to be restricted to keep
costs down
• R&D survey may have to be taken in conjunction with
another survey
• surveys of some groups may require the participation of
other agencies with different data needs and hence
different questions for respondents
 One size does not fit all: every country has different
constraints – advice is therefore of general nature
www.uis.unesco.org
Establishing registers
 R&D in developing countries tends to be very much the
purview of public bodies
Recommendations:
 Establishing a database of public sector R&D projects
• include human and financial resources; align with national policies.
• design could reflect the R&D statistical reporting/definitions.
• source for evaluation of such projects.
 Establishing Science and Technology Management
Information System (STMIS)
• provide overview of research system.
• framework for establishing complete registers as sample frames
for R&D surveys.
www.uis.unesco.org
Science and Technology Management
Information System (STMIS) and other
secondary sources
 STMIS (e.g. database of scientists, research
grants, CV databases, etc): frequent source for the
production of R&D statistics.
Recommendations:
• need close integration between the statistical system
and the STMIS.
• need adjustments to produce comparable statistics,
taking into account issues of definitions and coverage.
• need a balanced approach using both STMIS and
surveys.
• need different approach to Private sector organizations
as they are frequently not covered by these systems.
www.uis.unesco.org
Establishing registers
 Other sources
• Associations (trade, academic).
• Learned societies.
• Registers or databases of scientists and engineers.
• Database of research grants.
• Databases of scientific publications.
• Patents and other IP documents.
• Business registers.
www.uis.unesco.org
Survey procedure for each sector
 Each sector has different management styles,
approaches and institutional culture.
 Consider the existing norms in relation to data
exchange.
 First R&D Survey: through interviews rather than
relying on telephonic, e-mail or postal survey.
 Higher cost and labour intensive.
 Who is the target of the survey?
• Need to consider the sector and the size and complexity
of the organizations.
www.uis.unesco.org
Government sector: Identifying target
population and survey respondents
 Units to include in surveys are:
• R&D institutes: Public research institutes (PRIs);
Department-based research institutions (DBRIs)
• R&D activities of general administrations of central or
state government.
• Public institutions dealing with STS: statistical,
meteorological, geological and other public services,
museums, hospitals.
• R&D activities at the municipality level.
 Recommendation: the best way to survey is to
send questionnaires to all units known or assumed
to perform R&D.
www.uis.unesco.org
Government sector
cont..
 Department-based research institutions (DBRIs)
• Director-General or Permanent Secretary
• Issues: non-availability of information in compiled form
 Public research institutes (PRIs)
• Chief Executive Officer or executive responsible for
research management
www.uis.unesco.org
Higher education sector: Identifying target
population and survey respondents
 Recommendation: The surveys and estimation
procedures should cover all universities and
corresponding institutions, especially those
awarding degrees at the doctorate level. Other
institutions in the sector known or assumed to
perform R&D should also be included.
• Identification generally easy.
• preferable to use smaller units, such as departments or
institutes of the university, as statistical units.
www.uis.unesco.org
Higher Education sector
cont..
 Higher Education institutes (HEIs) are the main seat of
R&D activity
 Different degree of autonomy
• Staff employed as civil servant – list of employees is available
• Academics directly employed by HEI – staff details are protected
 Maturity of HEIs and historic relationships with Government
 Researcher CV database
 Publications databases (Web of Science™ or Scopus™)
 If there is no central registry – approach through Vice
Chancellor or Dean of Faculties, Dean of Research or,
Head of Departments.
www.uis.unesco.org
Business enterprise sector: Identifying
target population and survey respondents
 The enterprise is recommended as the main
statistical unit in the business enterprise sector
• Some enterprises perform R&D on a regular basis from
year to year, and may have one or several R&D units
• Other enterprises perform R&D only occasionally
 It is recommended that all enterprises performing
R&D, either continuously or occasionally, should
be included in R&D surveys.
www.uis.unesco.org
Business enterprise sector: Survey
population – first possible approach
1. A census-based survey of large enterprises and a
sample of smaller ones in order to identify R&D
performers and request the information from them
•
R&D performed in the past in the enterprise is not
considered
•
this is the approach followed in innovation surveys
•
very small enterprises and enterprises in certain less
R&D-intensive industries often excluded for cost reasons
•
when the sample size is very small, estimates may be
less reliable, owing to raising factors
•
Method not strictly followed in any country
www.uis.unesco.org
Business enterprise sector: Survey
population – second possible approach
2. Try to survey all enterprises known or assumed to
perform R&D, based on a register of R&Dperforming enterprises
•
•
•
•
•
•
lists of enterprises receiving government grants and
contracts for R&D
lists of enterprises reporting R&D activities in previous
R&D surveys, in innovation surveys or other enterprise
surveys
directories of R&D laboratories
members of industrial research associations
employers of very highly qualified personnel
lists of enterprises claiming tax deductions for R&D.
www.uis.unesco.org
Business enterprise sector: Survey
population – joint approach
Recommendation
 To include in R&D surveys of the business
enterprise sector all firms known or supposed to
perform R&D.
 To identify R&D performers not known or
supposed to perform R&D by a census/sample of
all other firms:
• In the industries on the next slide.
• In principle, enterprises in all size classes should be
included, but if a cut-off point is necessary, it should be
at ten employees.
www.uis.unesco.org
Business enterprise sector:
Industries to be included
Industry
ISIC Rev. 3/NACE Rev. 1
Mining
14
Manufacturing
15-37
Utilities, construction
40,41,45
Wholesale
50
Transport, storage and communication
60-64
Financial intermediation
65-67
Computer and related activities
72
R&D services
73
Architectural, engineering and other
technical activities
742
Plus any other industry relevant for the country
www.uis.unesco.org
Business sector
cont…
 R&D performed in business sector remains low in many
developing and emerging economies.
 How to detect R&D activity in Business?
• trade associations, or chambers of commerce.
• businesses listed on the main stock exchange.
• large firms/MNC - discussion with the Chief Financial Officer or Chief
Technology Officer.
• missing a large firm might result in significant error.
• exclude holding companies, construction, retail, and utilities as subsectors likely to perform little or no R&D.
• list of business beneficiaries of research or innovation grants by NRC.
• cooperation with the departments responsible for R&D tax incentives.
www.uis.unesco.org
Business enterprise sector:
Structural issues
 Publicly-owned businesses play a major role in
R&D in some developing countries
Recommendations:
• should consider issuing data for ‘publicly-owned
businesses’ separately from the ‘fully private enterprise
sector’.
• private enterprises could also be disaggregated by
ownership, in particular the various degrees of foreign
ownership.
www.uis.unesco.org
Business enterprise sector: Structural
issues
cont..
 Business enterprise R&D is presumed to be generally
weak in developing countries when compared to industrial
countries.
Recommendations:
• take into account when conducting sample surveys,
perhaps by over-sampling, especially amongst larger
companies.
• big companies should not be missed out as it might
imply significant error.
• invest time in interviewing key firms to understand their
R&D function and obtain a clear picture of their activity.
www.uis.unesco.org
Private non-profit sector: Identifying
target population and survey
 Private non-profit (PNP) sector: make a significant
contribution to R&D in developing countries, but
the sector tends to be very volatile
 Same challenges as in business – difficulty in
identify PNPs engaged in R&D
 Not clear about, status; ownership.
 Engaged in wide range of activities.
 Perform in-house R&D as well as contract R&D.
www.uis.unesco.org
Private non-profit sector
cont..
 The sources for identifying possible survey
respondents are mainly the same as for the
government sector.
 Register information may be less comprehensive
and could be completed by information from
researchers or research administrations.
 This sector may be more relevant for surveys on
R&D funding.
www.uis.unesco.org
Who is the right respondent?
R&D Manager
Accountant or personnel
manager
 Better understanding of
R&D and FM norms
 May not refer exactly to
R&D as defined in FM
 But may not be able to
supply exact figures
 But able to supply exact
figures
 Cooperation of all three may be needed
 Useful to identify in advance the person responsible for
providing information and for co-ordinating information
from smaller sub-units
www.uis.unesco.org
Working with respondents
 Questionnaire: simple and short, logical and with
clear definitions and instructions
• Optional: simpler survey for smaller units
 Test questionnaires on a sample of respondents
www.uis.unesco.org
Survey procedure and estimation
Recommendations:
• Attention needs to be paid to questionnaire design.
• Frequency of survey.
• Prioritize area of work; accompanied by step-by-step approach.
• Use of survey questionnaires of other countries for inspiration:
need adaptations to local situation.
• Get expertise from the NSO, in conducting survey, in sampling..
• Different questionnaires might be designed for different sectors
based on stakeholder consultations. “One size does not fit all”.
• Procedures need to be developed for estimating missing data.
www.uis.unesco.org
Encouraging co-operation
 Secure co-operation of respondent
• Make them appreciate the potential uses of the
data
• Respect confidential data
• Minimise the response burden
• Share the results (option: customised
information)
• Provide technical assistance and contact details
www.uis.unesco.org
Estimations
R&D measurement could be done in three stages:
 Identification of all specialised R&D units and
measurement of their total activity.
 Estimates of the non-R&D portions of their activity
and subtraction of these estimates from the total.
 Estimates of the inputs used for R&D in other units
and addition of these estimates to the total.
www.uis.unesco.org
Operational criteria
Tools for “translating” theoretical FM concepts into
practical questionnaire:
 Explanatory notes
 Hypothetical examples
Covered in FM
 Guidance to individual respondents
 Documentation on treatment of different cases
To be covered by data collection agency 
keeping good documentation is essential
www.uis.unesco.org
Estimation procedures
 Imputation methods for item non-response
• Use previous answer
• Hot decking (use info from same survey)
• Cold decking (use info from previous survey)
 Imputation methods for unit non-response
• Use past R&D data (adjusted for sales or employment
growth)
• Impute as a function of the relation to personnel or sales
(test with non-response analysis)
www.uis.unesco.org
Thank you!
http://www.uis.unesco.org
[email protected]
www.uis.unesco.org