Transcript Slide 1

Measuring the impact of Egovernment
Deepak T. Bhatia
Sectoral and E-government Applications Practice Lead
Global ICT department (GICT)
The World Bank
Deepak T. Bhatia
It is no longer acceptable for technology to be utilized simply for technology’s sake.
Individual e-government projects need to demonstrate their contribution to overall
government objectives – with the focus
being on results and impact.
Deepak T. Bhatia
Why e-Government?
•Battling
bureaucracy
•Improving
Efficiency
•Better service
delivery
Strengthening
governance and
accountability
Deepak T. Bhatia
3
In the World Bank the Number of Projects with ICT is
growing
About 150 contain ICT components (out of 250 new WB projects each year).
57% of active
projects have
ICT
components
(this is 930
projects)
ICT
components’
combined
volume is
$6.2 billion
Deepak T. Bhatia
4
Analysis of e-Government Impact
• World Bank commissioned study to explore impact of egovernment on citizens, in collaboration with the Indian
Institute of Management, Ahmedabad, India and the
London School of Economics
• Effort to go beyond anecdotal assessments commonly
available
• Sample of 5 mature, wide scope/scale projects selected for
analysis from India
– Chosen to represent different stages of overall IT
readiness and evolution
Deepak T. Bhatia
5
Learning from Past Assessments
• Variety of approaches have been used for assessments -client satisfaction
surveys, expert opinion
• Often studies have been done by agencies that may be seen as being
interested in showing positive outcome
• Lack of credibility of results-different studies of the same project show very
different outcomes
• Lack of rigor in sampling-results can not be easily generalized
• Lack of rigor in controlling for external influence-need for counterfactuals
ignored.
• Lack of a standard methodology-making it difficult to compare projects
Deepak T. Bhatia
Dimensions of measurement - Stakeholders
•
•
Impact on Clients:
All measurements on the basis of a sample of clients for each major service
availed by the client. Measurements would be done for electronic delivery of
services as well as for the earlier mode of delivery of the same service. In cases
where alternate (non electronic modes) are currently being used by the same
set of users in similar contexts elsewhere, measurement would be recorded for
such usage
•
•
Impact on Agencies (Including Partners in Implementation):
All measurements on the basis of data collected from agency records, a sample
of employees or a group of managers. Measurements would be done for
electronic delivery of services as well as for the earlier mode of delivery of the
same service
•
•
Impact on Society:
Some of the questions to be framed as degree of improvements resulting from
the implementation of a specific project
Deepak T. Bhatia
7
Measurement Framework - Dimensions
Stakeholders
Key Dimension of Impact
Client
Economic (Direct & Indirect)
Governance (Corruption, Accountability, Transparency, Participation)
Quality of Service (Decency, Fairness, Convenience, etc.)
Over all satisfaction
Agency
(Including Partners in
Implementation)
Economic (Direct & Indirect)
Governance (Corruption, Accountability, Transparency, Participation)
Performance on Key Non-economic Objectives
Process Improvements
Work life of employees
Society
Other Departments
Government as a Whole
Civil Society
Desirability of investments in e-Government
Impact on vulnerable groups
Image of Government (Efficiency, Corruption, Accountability, Transparency,
Participation, Responsiveness)
Impact on development Goals
Deepak T. Bhatia
Proposed Framework
•
Focuses on retrospective assessment of e-delivery systems(B2C and B2B)
•
Balanced approach between case study and quantitative analysis
•
Recognizes that some part of the value to different stakeholders can not be
monetized
•
Understand how inputs lead to outputs and outcomes in different project contexts
•
A practical methodology that can be used for designing bench mark surveys, M&E
systems and prospective evaluation of projects in countries with various delivery
models and paucity of data
Deepak T. Bhatia
Methodology for Assessment
•
Select mature implementations , of e-delivery of services.
•
Collect data through structured survey from clients, employees, supervisors using
counterfactuals ( for old non computerized delivery and new e-delivery system)
•
Customize survey instrument to each project, adapt in local language
•
Data can be collected through Internet survey, face to face interviews and focus groups
•
Determine sample frame and size so that results can be extrapolated to the entire population
(often 300 clients may be sufficient).
•
Collect data on investments, operating costs, activity levels, revenues, employee strength from
agencies
Deepak T. Bhatia
Projects assessed
•
Bhoomi project. 2001. delivering two online services: issuance of records of
rights, tenancy, and crop inspection register (RTC) and filing of requests for
mutation (changes in land ownership) for affecting changes in land records.
•
Karnataka Valuation and E-Registration (KAVERI) project. 2003 delivering
three key services: registration of property sale and purchase deeds, issuance
of nonencumbrance certificates, and issuance of copies of previously
registered deeds.
•
Computer-aided Administration of Registration Department (CARD). 1998.
same three online services provided by KAVERI.
•
eSeva. 2002, These one-stop service centers now deliver 135 services from
central, state, and local governments and public utilities, and are used by 3.1
million citizens a month
•
Ahmedabad Municipal Corporation (AMC) Civic Centers project. 2002,
deliver three important services: annual payment of property taxes, issuance
of birth and death certificates, and issuance of shop licenses.
Deepak T. Bhatia
11
Caveat
•
largely relies on the survey of stakeholder perceptions of public
services before and after these projects.
•
Thus not an impact evaluation of investments in narrowly defined ICT,
which typically includes hardware, software, and communication
equipment.
•
Rather, a review of the outcomes of packaged efforts involved in the
transition from manual services to those using ICT as a key
component.
Deepak T. Bhatia
12
Impact: Computerized service delivery is preferable to manual
Preference for Computeriz ation (%)
100
80
60
40
20
0
Bhoomi
KAVERI
CARD
e-Seva
AMC
Preference for Computeriz ation (%)
Deepak T. Bhatia
13
User reports of changes in travel costs, by project (number of
trips * distance travelled)
Rupees per service
•
Bhoomi costs went up due to change in location of service centers
from villages to “taluka” headquarters
120
100
80
60
40
20
0
-20
-40
-60
-80
-100
Bhoomi
RTC
Bhoomi
Mutation
KAVERI
CARD
Deepak T. Bhatia
eSeva
AMC
14
User reports of reductions in waiting times, by project
Minutes
•
•
decreased in all five projects.
With manual systems, users often had to wait in long lines to receive services.
Automation has increasedthe efficiency of document processing.
100
90
80
70
60
50
40
30
20
10
0
Bhoomi
RTC
Bhoomi
Mutation
KAVERI
CARD
Deepak T. Bhatia
eSeva
AMC
15
User perception of improvement in service quality
•
Rating on elements such as service delivery location, accessibility,
convenience, cost, transparency, and service orientation
AMC
eSeva
CARD
KAVERI
Bhoomi-Mutation
Bhoomi-RTC
0.0
0.2
0.4
Deepak T. Bhatia
0.6
0.8
1.0
16
Users’ top four desired features of services, by
project
Project
Features
Bhoomi
Error-free transactions
Reduced delays in
transactions
Shorter waiting times
Fewer visits
KAVERI
Less corruption
Increased
transparency
Error-free transactions
Shorter
waiting
times
CARD
Increased convenience Shorter waiting times
Less corruption
Fair treatment
eSeva
Increased convenience Shorter waiting times
More convenient time
schedules
Fair treatment
AMC
Increased convenience Less corruption
Greater transparency
Good
complaint
handling
system
Deepak T. Bhatia
17
Proposed conceptual framework
Goals
Examples of performance
indicators
Key features of the enabling
environment
Outcomes
 Increased efficiency
 Increased transparency and
accountability
 Higher-quality public
services
 Better access to services







Outputs
 Reengineered processes
 New ICT systems
 Increased service coverage





Financial and time savings in
government activities
Public perceptions, such as
user satisfaction and score
cards
Financial and time savings
for citizens
Increased public service
timeliness and
responsiveness
Reduced errors
Financial saving per
transaction
Comparisons of old and new
business processes
Technical reviews of IT
infrastructure, applications,
and performance
Variety of available services
IT support capacity
Service training





Overall e-government
strategies
Political and popular support
for cross agency
coordination and public
sector reform
Telecommunications
infrastructure and cost
structures for increasing ICT
access
Supportive legal and
regulatory frameworks
Balances with competing
priorities (such as roads and
education)
Macroeconomic changes
Additional conclusions
•
Out of the three projects where significant corruption was reported in the manual
system, one project was able to eliminate corruption through computerization
•
Incremental operating costs for agencies per transaction are less than the
reduction in the direct cost of accessing the service reported by the clients.
•
Negative aspect is the observed variability of impacts across different delivery
centers in a project
•
Agencies: ability to cope with growth in transactions was enhanced
•
Citizens strongly supported the idea that more agencies need to be
computerized, but not necessarily through government investments
Deepak T. Bhatia
19
Conclusions
•
Undertake ex-post evaluation of successful ongoing projects.
•
Can be used for for ex-ante evaluation, with following assumptions
– extent and adoption rate of user demand need to be estimated
– Project designs and investment levels must be commensurate with
anticipated benefits to users.
– quality of the enabling environment should be assessed to gauge the
expected impacts of planned projects
– Detailed surveys at the initial stages of project planning, would help
establish a baseline.
Deepak T. Bhatia
20
Lessons Learned
•
IT cannot be a goal in itself - projects must have a direct link to
process reform
•
E-Government is a long term endeavor: not a silver bullet, not for
immediate gains
•
‘E’ stands for ‘Everything-but-IT’: institutional change, political
support, capacity development.. technology is the easy part
•
Ultimately can generate significant cost and efficiency gains
But requires…
Deepak T. Bhatia
21
Focusing on results
Deepak T. Bhatia
22
Credits
• Impact Study team
• Presentation research
• Subhash Bhatnagar , IIM,
Ahmedabad, India (Primary
Author)
• Asheeta Bhavnani, WB
• Mikhail Bunchuk, WB
• Jiro Tominaga, WB
• Reviewers
• Deepak Bhatia, WB
• Shirin Madon, London
School of Economics
• David L. Cieslikowski
• Christine Zhen-Wei Qiang
Deepak T. Bhatia
23
Annex
Deepak T. Bhatia
24
How have governments been measuring results of egovernment investments?
•
France – Mareva - A Method of Analysis and Value Enhancement
•
Germany -WiBe - Economic Efficiency Assessment Methodology
•
USA – Performance Reference Model Framework
•
Australia: Demand and Value Assessment Methodology
•
All these government methodologies broadly focus on the same
dimensions.
•
They offer two levels of impact assessment:
– first, in terms of how projects provide a business case justification for
expenditure and whether they meet the targets set for them,
– and second, in terms of how projects meet the goals of the agencies
concerned and, in turn, how that helps achieve wider government
strategies.
Deepak T. Bhatia
25
World Bank and Impact measurement experience
• What is the WBs experience with measuring impact?
• Independent evaluation group - IEG measures impact of Bank
programs and projects, and draws lessons from experience.
The Bank also has a large program of impact evaluation
although there may not be any that have centered around the
ICT sector.
•
Whether WB prescribes ONE methodology to all countries?
How much tailoring is needed?
• IEG generally uses objective based methodology to measure
impact. But if this question refers to tailoring strategies then the
key finding from IEG work is that strategies including ICT must
be tailored to country circumstances.
Deepak T. Bhatia
26