Transcript 14 Oct 2010, TELECENTRE-EUROPE Summit TELECENTRE-EUROPE Summit: Third Sector contribution to e-Inclusion in Europe: Impact Measurement and Analysis framework Budapest, 14th October 2010 Gabriel Rissola,

14 Oct 2010, TELECENTRE-EUROPE Summit
TELECENTRE-EUROPE Summit:
Third Sector contribution to e-Inclusion in Europe:
Impact Measurement and Analysis framework
Budapest, 14th October 2010
Gabriel Rissola, JRC-IPTS
Maria Garrido, Technology & Social Change Group | University of Washington
Joint Research Centre (JRC)
Institute for Prospective Technological Studies
The European Commission’s Research-Based Policy Support Organisation
Institute for Prospective Technological Studies
14 Oct 2010, TELECENTRE-EUROPE Summit
Part of Joint Research
Centre of the EC
IPTS: Research Institute
supporting EU policy-making
on socio-economic, scientific
and/or technological issues
MSP for eInclusion
14 Oct 2010, TELECENTRE-EUROPE Summit
Actor for
inclusion
and
innovation
Sustainable
business
model
Showing
and
measuring
impact
Last TE Summit in Istanbul:
IA policy relevance and constraints
14 Oct 2010, TELECENTRE-EUROPE Summit
Demonstrate Impact is useful for:
Get a recognition of telecentres role in EU policy making:
- Evidence is essential to get policies adopted (including funding programs)
- Evidence helps to set priorities, make the right choices
Smarter action by combining social innovation and economic
innovation, stimulated by technological innovation
…but few IA good practices were identified in the field
In Vienna Study on Inclusive Innovation for Growth and Cohesion,
1000 cases were analysed but only 52 had measured outputs and
outcomes (mostly from UK and IRL), for example:
– UK Onlines Centres trained 76,000 people; active job seekers increased from 66% to 100%
– EOL UK digital literacy training helped 76 people to get a job, saving £670,000 in state subsidies
– FIT Ireland trained 6500 young unemployed people in ICT; 3500 found a job, saving $13 million.
– Also examples of improved welfare benefits distribution, savings through ICT-enabled home care.
Last TE Summit in Istanbul:
Measuring Impact (outputs, outcomes)
14 Oct 2010, TELECENTRE-EUROPE Summit
Demand
DOMAINS BY MAIN
FOCUS OF INITIATIVES
eINCLUSION
Supply
Enabling
MEASURE
CATEGORY
TYPICAL OUTPUT
public access points availability
# of regular customers
basic digital literacy
# individuals completing courses
tax relief & incentives for affordability
% of eligible beneficiaries
using the incentives
eAccessibility
# of accessible public services
broadband
% Δ broadband coverage
TYPICAL
POTENTIAL
OUTCOME
beneficiaries becoming
regular Internet users
Supply
Accelerating
Demand
Contribution to
Regional growth
Community/ territorial initiatives
Holistic not possible to indicate 1
Various measures of
community social capital
learning through ICT
# individuals completing courses
Various learning outcomes
skill-building for employability
# individuals completing courses
trainees finding a job
ageing-well at home services
# of homes covered
fewer days spent in
institutional care/hospital
healthcare services
Holistic not possible to indicate 1
Various health outcomes
access to welfare entitlement
Holistic not possible to indicate 1
Beneficiaries increase in
disposable income
Cristiano Codagnone et al, EC study March 2009 (Vienna Study on Inclusive Innovation for Growth and Cohesion)
Last TE Summit in Istanbul:
Telecentre eInclusion critical role and challenge
14 Oct 2010, TELECENTRE-EUROPE Summit
• Increasing political awareness on the critical role of
Telecentres and NGOs in promoting e-Inclusion goals
through different social and digital programs
• Wide range of target groups addressed - from immigrant
communities, women, impoverished youth to prisoners and
isolated people in rural areas to people with physical disabilities
and the elderly – which goes even beyond Riga’s targets
• However, 3rd Sector organisations working in the eInclusion
field don’t balance their efforts in critical areas (like IA)
Last TE Summit in Istanbul:
Telecentre eInclusion critical role and challenge
14 Oct 2010, TELECENTRE-EUROPE Summit
• NGOs often struggle to systematically:
– collect information from their beneficiaries without
overburdening staff capacity
– articulate how and the extent to which their programs
impact the lives of the people they serve
– use this evidence as a tool to gain more visibility and
recognition to their work especially among policy makers
and donors
• Measuring Outputs (quantity of trainees, quantity of Internet
users) and End user satisfaction surveys are important,
but insufficient; need to go a step forward and find a way to
systematically measure Outcomes
Practitioner perspective
Telecottage impact measurement system
14 Oct 2010, TELECENTRE-EUROPE Summit
•
•
•
•
•
Adapted UK’s IMProving Accountability, Clarity and
Transparency (IMPACT) to Hungary
Prepare impact report of the real social effect of their work.
Primary data (outputs), with Secondary (outcomes).
Created intended effects and possible indicators, on
whether life is better or not.
Most important effects of telecottages:
– strengthened community spirit,
– improved human conditions,
– increase public trust
•
Challenge:
– to adapt intended effects and possible indicators to other local
contexts
– to measure them in an aggregated way at regional/national/EU level
Funder perspective
MS UP impact measurement system
14 Oct 2010, TELECENTRE-EUROPE Summit
•
•
•
Driven by need for systematic and credible evidence of
how UP helps unemployed & others excluded to get essential
skills to find a job.
Most important thing is that evaluation scales globally. Is it
improving lives? Are they moving along a trajectory?
UP goes beyond self-reported quarterly reports, doing
evaluations and case studies.
Funder perspective
MS UP impact measurement system
14 Oct 2010, TELECENTRE-EUROPE Summit
•
Lessons learnt:
–
–
–
–
track results on ongoing basis and make adjustments as needed
metric that decided to avoid is how many people they put into jobs;
commissioned research can be expensive;
strongest grantees & partners are data driven and report back
consistently, which builds a solid picture on impact over time
– Need for common indicators, plural of anecdote is not evidence.
•
Opportunity: telecentres are multipliers
14 Oct 2010, TELECENTRE-EUROPE Summit
Building an IA framework
Building an IA framework
Possible steps
14 Oct 2010, TELECENTRE-EUROPE Summit
1. Consensus building on the objectives of measuring
impact
– to provide funders or investors with data on impact;
– to provide a tool for organisations to manage their own choices
internally
– to better understand long-term processes of social change and impact.
2. Targeted methodological framework for IA based on the
state-of-the-art (there is no “perfect” social metric framework)
3. Targeted set of indicators
– Relevant enough as to measure different e-Inclusion dimensions
– Simple and easy to implement by organizations of different sizes
– Appropriate enough as to provide consolidated results related to the e-Inclusion
goals of the DAE
– Compatible with existing indicators but going deeper into the specificity of the
measured field.
Building an IA framework
Possible steps
14 Oct 2010, TELECENTRE-EUROPE Summit
4. Need to facilitate social organizations self-analysis and
self-interpretation of their own local impact.
Indicators should be:
– Available online (save costs and facilitates its distribution)
– Translated and contextualized in each country (localisation)
– Complemented by adequate guidelines for a proper data collection, self-analysis
of data and production of reports for different organisational scopes
Guidelines for implementation and interpretation would surely be necessary
5. Need a software for the implementation of an IA system
•
•
•
•
•
based on those indicators
To facilitates the process of user data collection and impact analysis for NGOs
To create with these data a repository that can be aggregated and analyzed globally
To protect anonymity of NGO users and of the organizations themselves
Easily implementable by the interested organizations in each local context
Centrally maintained for the provision of constantly-updated consolidated information
Building an IA framework
Indicators to be measured
14 Oct 2010, TELECENTRE-EUROPE Summit
Measuring indicators is a complex endeavor due to the
complexity of isolating the effects of one single variable
(e.g. effect of e-Skills training in people’s lives).
This process is further complicated by the diversity of NGO
programs, target groups, local factors, individual
factors, etc.
Different types of impact can be measured (i.e. Access,
frequency and purpose of ICT use, skills development,
size and diversity of social networks, user perception of
usefulness of ICT for improving their lives)
However, how to connect them with desired Outcomes
remains an issue (e.g. “Qty of DL trainees that become
regular Internet users”, “DL trainees that improve their
employment status”)
FEEDBACK FROM THE AUDIENCE
14 Oct 2010, TELECENTRE-EUROPE Summit
Constraints
- Voluntary organisations
- Data protection laws
- Local advocacy skills
Org needs/Selling points
- Carefully select and agree the objectives of IA
- Start simple, small
- Highlight useful bits of data
- Simple and quick implementation
- Pay attention to incentives
14 Oct 2010, TELECENTRE-EUROPE Summit
16
Thank you
[email protected]
http://is.jrc.ec.europa.eu/pages/EAP/eInclusion.html
http://is.jrc.ec.europa.eu/pages/EAP/eLearning.html
[email protected]
http://tascha.washington.edu/
Last TE Summit in Istanbul
One example (female migrant users)
14 Oct 2010, TELECENTRE-EUROPE Summit
Perceived usefulness of ICT
Appreciated features of TCs
Friendly, open atmosphere 75%
Clear information 58%
NGOs
Most
Appreciated Aspects
Access to Free-Low cost services 57%
Competence of NGOs Operators
46%
Mediation towards Public
Authorities and Privates
30%
END USER SATISFACTION IS IMPORTANT,
BUT INSUFFICIENT
The NGOs
Network of
contacts
29%
Satisfaction with Social services + Education and Employment services
Building an IA framework
Objectives of the Metrics
14 Oct 2010, TELECENTRE-EUROPE Summit
Metrics are useful to show what works and what deserves to be
grown (from The Open Book of Social Innovation):
• There are many metrics for judging whether innovations are working – at
various stages of development.
• Metrics can play a decisive role in determining whether innovations are
scaled up, or deserve to be.
1. NEED TO BUILD A CONSENSUS ON THE
OF MEASURING
IMPACT
• A recent surveyOBJECTIVES
found 150 different
metrics in use
in the non-profit sector.
However, relatively few of these are actually used to make decisions.
• This field has failed to make progress due to the confusion and conflicts
of interest between three different tasks performed by metrics:
– to provide funders or investors with data on impact;
– to provide a tool for organisations to manage their own choices internally
– to better understand long-term processes of social change and impact.
Building an IA framework
Social metrics system
14 Oct 2010, TELECENTRE-EUROPE Summit
There is no “perfect” social metric framework | Pros and
Cons
Example 1: Sustainable Livelihoods Framework
• Increasingly popular among development agencies
• It recognizes that people have a wide range of resources at their disposal and
their choice for deploying those resources are critical to assess impact
2. NEED
TO BUILD
A TARGETED
• Very closely align to Amartya
Sen’s
capabilities
approach
METHODOLOGICAL
I.A. CHOICES AND
LIMITATION: DIFFICULT
TO IMPLEMENT ANDFRAMEWORK
MEASURE HOWFOR
PEOPLE’S
MOTIVATION AFFECTS
THE OUTCOME
A PROJECT
BASED
ON THEOF
STATE-OF-THE-ART
Example 2: Cost-benefit analysis
• Used primarily by public authorities and government agencies
• Its goal is to assess a particular project taking into account costs and benefits
• Benefits and costs are quantified in money terms
LIMITATION: ASCRIBING MONETARY VALUES TO CERTAIN SOCIAL IMPACTS THAT
ARE VERY DIFFICULT TO MONETIZE
Building an IA framework
Selection of indicators
14 Oct 2010, TELECENTRE-EUROPE Summit
3. NEED A TARGETED SET OF INDICATORS
•
•
Relevant enough as to measure some dimensions of the
impact of e-Inclusion initiatives driven by NGOs in the
communities they serve;
Simple and easy to implement by organizations of different
sizes which are active in the field of e-Inclusion at
regional/national level in Europe (extendible to smaller NGOs
and other actors in a second stage)
•
Appropriate enough as to provide consolidated results
related to the e-Inclusion goals of the Digital Agenda of
Europe.
•
Compatible with existing indicators but going deeper into the
specificity of the measured field.
Building an IA framework
Some examples of indicators
14 Oct 2010, TELECENTRE-EUROPE Summit
Aggregated level
Organisational level
• User demographics
• NGO services and activities where
•
•
•
•
•
•
users participate
Digital skills level
Frequency of ICT use
Types of ICT use (email, chat, job
searching, homework, etc)
Employability situation
Social inclusion situation
Users perception of value of ICT to
improve their lives
•
•
•
•
E-Inclusion activities
Other social services
NGO mission
Reach of the work (local, regional,
national)
• Target groups
• Partnerships
Building an IA framework
Software for processing data
14 Oct 2010, TELECENTRE-EUROPE Summit
5. NEED A SOFTWARE FOR THE IMPLEMENTATION OF AN I.A. SYSTEM
BASED ON THOSE INDICATORS
•
•
•
•
•
To facilitates the process of user data collection and
program impact analysis for NGOs
To create with these data a repository that can be
aggregated and analyzed by country and at European level
In doing so, to protect anonymity of NGO users and of the
organizations themselves
Easily implementable by the interested organizations in each
local context
Centrally maintained for the provision of constantly-updated
consolidated information at European level
Building an IA framework
Stakeholders and Users
14 Oct 2010, TELECENTRE-EUROPE Summit
Stakeholders interested in the metrics to be produced
• Organisation’s staff (managers, fundraisers, etc)
• Local policy officers in charge of social services and
community development
• Donors/funders
• Policy makers operating at different levels, including the EC
Users who will collect the data
• eFacilitators in contact with the end users
• Volunteers
• Staff managing the telecentres
• Telecentre network coordinators
Building an IA framework
Other needs
14 Oct 2010, TELECENTRE-EUROPE Summit
•
•
•
•
Review of indicators currently used by EUROSTATS,
OECD, ITU, and academic articles
Needs assessment to determine the kind of information that
social organizations are currently gathering about the
beneficiaries of their programs and additional data they are
interested in collecting.
Piloting the Impact Measurement System in different
contexts
Wide implementation of the system across Europe
promoted by a multistakeholder partnership (MSP) or an
umbrella organization who represents several stakeholders
(like TE)