Surveying eValues: Experiences and Challenges

Panel at the Bled Conference Monday June 5, 2006 Ronald Batenburg Institute of Information and Computing Sciences Utrecht University The Netherlands Bled panel Surveying e-values June 5 2006 1

Agenda for this panel

14:00-14:20  Ronald Batenburg, Utrecht University  14:20-14:40  14:40-15:00  15:00-15:20   Georgios Karageorgos, European Commission, DG Enterprise & Industry (not present) Fernando Reis, European Commission, DG Eurostat Hannes Selhofer, Empirica Gesellschaft für Kommunikations- und Technologieforschung Vasja Vehovar, University of Ljubljana 15:20-15:30 Closing remarks an discussion June 5 2006 Bled panel Surveying e-values 2

Introduction

    Why this panel?

My recent survey experience The STILE project and beyond My three main problems to tackle in the next 5 years 3 June 5 2006 Bled panel Surveying e-values

Example: evaluating the Dutch ‘Go Digital’ program

    Goal of this policy program: improving the ICT/e-business maturity of Dutch SMEs to meet the ‘European’ level Sponsored/funded by the Dutch Ministry of Economic Affairs Executed by intermediary organizations, providing (free) IT advise, consultancy, workshops, etc Budget: € 40 Mio over 2001-2005, app. 20,000 Dutch SMEs participated June 5 2006 Bled panel Surveying e-values 4

Evaluation questions

   Was the program executed efficiently and effectively?

Did the program made a significant difference to the ICT-maturity of Dutch SMEs?

(how) to continue with Go Digital program for Dutch SMEs in the near future?

June 5 2006 Bled panel Surveying e-values 5

Research questions

  How to measure ICT/e-business maturity and effectiveness in SMEs?

How to conclude if the program was successful, i.e.: – How to make a ‘ fair’ comparison between large, small and non-participants of the Go Digital program?

– How to control for other determinants of ICT maturity and developments over time?

June 5 2006 Bled panel Surveying e-values 6

% yes

Result 1: Go Digital participants are very ICT mature

100 80 60 40 20 0 have an intranet have a corporate w ebsite support customers and e sales IT is externally connected for sales IT is externally connected for purchasing Dutch average Go Digital participants non Go Digital participants June 5 2006 Bled panel Surveying e-values 7

Result 2: Go Digital participants are successful ICT users as well

ICT saved time ICT improved customer satisfaction ICT improved communication ICT increased labor productivity ICT stimlated product innovation ICT increased turnover ICT increased product quality ICT reduced costs Fully disagree 1 1,5 Go Digital participants 2 2,5 3 3,5 non Go Digital participants 4 4,5 5 Fully agree June 5 2006 Bled panel Surveying e-values 8

Reflection (1)

   Standardized questions on ICT/e-business maturity were available by Dutch Statistics Standardized questions on the added value, effectiveness and success ICT/e-business were not available, or ‘under construction’ Hence: – Two new surveys among Dutch SMEs and ICT were conducted (n=900) – ‘Acceptable’ response rates (30%) among SMEs were realized only after major investments – The validity, reliability and potential future use of the new surveys are to be explored June 5 2006 Bled panel Surveying e-values 9

Reflection (2)

    The Ministry of Economic Affairs (and its policy partners) have little incentives to monitor survey methods and measurements Dutch Statistics has little incentives to interfere or advise in survey methods and measurements of other survey projects We as (academic) researchers do have an incentive to conduct new and customized survey methods and measurements Hence: – Users, sponsors, researchers and statisticians of surveys are loosely coupled, not optimally cooperating or integrating – New ICT/e-business/Go Digital evaluations will lead to new survey projects June 5 2006 Bled panel Surveying e-values 10

The STILE project 2001-2004

Goals: “to provide innovative methodologies and content for the statistical monitoring of the European labor market in the eEconomy ” “ a critical assessment of the European statistical system and to increase awareness in the international statistics and research community because of the current lack of measurement accuracy ” Research partners:  Cork Telework Centre (CTC) and Central Statistical Office (CSO), Ireland  Institute for Employment Studies (IES), United Kingdom   CAMIRE Estadística y Análisis, S.L. (CAMIRE), Luxembourg Institut für Arbeitsmarkt- und Berufsforschung (IAB), Germany    Istituto di Ricerche Economiche e Sociali (IRES), Italy Organisatie voor Strategisch Arbeidsmarktonderzoek (OSA), The Netherlands Institute of Sociology-Hungarian Academy of Sciences (ISB), Hungary Products:  Ramioul M., Huws U. & Bollen A. (eds.), Measuring the Information Society, STILE report, HIVA-K.U.Leuven, Leuven  www.stile.be

– Questionnaire database – Digital toolkit June 5 2006 Bled panel Surveying e-values 11

The 11 STILE enterprise surveys

•Longitudinal or panel •Telephone, web, F2F, postal •Sample size 1,000-23,000 •Stratified UK-1980-1989 Belgium-1999 Netherlands-1987-2001 Belgium-1992-1997 US-1994-2000 Germany-1993-1000 Sweden-1995-1997 Canada-1996-1999 UE-1998 US-1987-2002 Germany-1987-1997 0 10 20 30 40 50 60 70 80 90 June 5 2006 Bled panel Surveying e-values 12

The STILE survey and questionnaire database Very informative overviews, however …

– What would be the ‘best’ survey questions?

• Or: when to apply which questions?

– What would be the ‘best’ way to sample and approach enterprises for surveys?

• Or: when to apply which approaches?

– What would be the ‘best’ method to conduct survey enterprise questionnaires?

• Or: when to apply which methods?

June 5 2006 Bled panel Surveying e-values 13

June 5 2006

Again: the actors involved (derived from STILE project)

Statisticians 1 2 (academic) Researchers 3 Policy-makers Respondents Bled panel Surveying e-values 14

My Research Agenda

Three main problems to be tackled in the next five years: 1. To improve the survey standards, by researchers and research organizations 2. To improve the and survey standards, by statisticians 3. To improve the use of questionnaire and monitoring of awareness questionnaire of questionnaire and survey standards, by policy makers June 5 2006 Bled panel Surveying e-values 15

European Commission, DG Enterprise & Industry

e-Business W@tch

in the context of e-business related activities of DG Enterprise and Industry Georgios KARAGEORGOS 19 th Bled eConference, Surveying eValues Panel Bled, 5 June 2006

Overview of the presentation • Background info and policy context (slides 3-5) • What is the e-Business W@tch (slides 6-13) • Lessons learned and future plans • Support / Background info (slides 14-16) (slides 17-20)

Main lessons learned • Fulfilling existing, real need => Great holders (even university from Mexico) interest + support from stake to continue; • Project unique (internationally ?

) in combining statistics (going one step further than ‘traditional’ surveys), qualitative analysis (incl. case studies) and policy recommendations; • Strong points = Timeliness, quality and transparency!

• ‘Victims of success’: involvement BUT Growing expectations, demands and different interests => Can NOT satisfy all !

• Continuous trade-offs , e.g. increase sectoral focus reduced survey’s capacity; => • Increased burden for flexibility and coordination capacities.

Future Plans: a ‘new’ eBW (within the policy context described in slide 5 ) • Insights from on-going Cambridge conference project, interim evaluation (by IDEA) and (http://ec.europa.eu/enterprise/ict/conferences/cambridge.htm) ; • Emphasis on 3 themes : Impacts, impacts and impacts!

• Focus on specific sectors , companies’ size classes & regions; • Approach: Maintaining quantitative ( data ) and qualitative ( case studies ) information collection, combined with high quality analysis , leading to identification of policy challenges; • ‘Common’ data coming from available official statistics (mainly Eurostat), complemented by purpose-made surveys ; • More flexibility of implementation arrangements .

Thank you

for your attention!

For more information: http://europa.eu.int/comm/enterprise/ict/index.htm

http://www.ebusiness-watch.org

[email protected] & [email protected]

[email protected]