Transcript Monitoring health system performance - synthesis of some experiences from low-income countries Dina Balabanova, Tim Powell-Jackson, Richard Coker, Kara Hanson & Anne Mills London School.

Monitoring health system performance -
synthesis of some experiences from low-income
countries
Dina Balabanova, Tim Powell-Jackson, Richard Coker, Kara Hanson &
Anne Mills
London School of Hygiene and Tropical Medicine
Health System Metrics, Glion sur Montreux,
28-29 September 2006
Overview








Background
Complexity
Objectives and methods
Measurement
Health financing
Health care delivery
Emerging issues
Conclusions
Background



Commitment to invest in health systems is unprecedented, but will
not last unless it is possible to show results
Currently poor health information available but demand for
improved health system metrics (national / international)
Opportunities
– Health System Metrics and other initiative seeking to strengthen HIS
– Commitment to the health MDGs – need to measure progress
– Growing consensus of importance of measurement strategies & monitoring &
evaluation built into programme planning cycles

Threats
– Limited resources for health information and sustainability
– Capacity constraints (in the health and social sectors)
Objectives & Methods

Purpose of the study: a review of some low-income countries’
experiences with health system performance monitoring and use of
data

Case study countries:
–
–
–
–
–
Georgia
Rwanda
Uganda
West Bengal, India
Material from other countries

Selection criteria

Analytical approaches:
– uses the WHO health system performance framework
– synthesis around common themes and issues
– identifying unique lessons in each type of context
Complexity

How should health system performance be measured?
– Increasingly multiple contacts with the system, chronic diseases
– Outcomes determined by different care components, sectors
– Need for system-wide and inter-sectoral indicators

Tension between international (donor-driven) demands and
country-level agendas and needs

Use of normative approaches imply causality

To what extent monitoring influences policy?

Impact of measurement on health systems, e.g. Indicators that are
measured often improve

Monitoring information may be complex to interpret where a
range of interventions co-exist.
Measurement
What approaches are taken to measure health
system performance in the study countries?
What is measured ?
Data
Georgia
Rwanda
Uganda
West Bengal
Demographic
Census (2002)
Census (2002)
Census (2002)
Census (2001)
NHA (2004)
NHA (2003)
NHA (2001), public
expenditure reviews,
Tracking Study (2001)
NHA (2001)
Health outcomes incl.
births and deaths
RHS (2005) & MICS
(2006), Vital
registration & HMIS
DHS (2005), HMIS
(facility)
DHS (2004)
DHS (2005)
Co-coverage of
interventions
MICS (2006) & RHS
(2005)
DHS (2005)
DHS (2004)
DHS (2005)
HMIS
HR inventory
HR inventory
HMIS
HMIS & SAM (200506)
HMIS, SAM (2004) &
Service Provision
Assessment Survey
(2001)
HMIS, SAM (2004), Area
Team assessments
HMIS
n/a
n/a
Various / accreditation
n/a
Immunisation, TB,
HIV/AIDS
Immunisation, malaria,
HIV/AIDS, TB etc.
Immunisation, malaria,
HIV/AIDS, TB etc.
Malaria, RCH, TB,
Leprosy, Polio, HIV/AIDS
etc.
IDRS
Sentinel sites (HIV), early
warning system, IDRS
Sentinel sites (HIV)
HMIS
Health financing
Human resources
Service provision
Quality
Vertical programme
monitoring
Disease surveillance
Health financing
How has information been used?
Where are the gaps?
What challenges remain?
Use of health financing information

Identification of financing gaps and advocacy for increased allocation
of funds to health (Rwanda)

Health sector leadership and management of funds (Tanzania,
Rwanda)

Equity of health financing in the health system (South Africa, Rwanda)

Protection against the financial burden of ill health (Mexico)

Resource allocation with the health sector (Rwanda)
Gaps in health financing information

Private health expenditures – difficult to collect compared to
public and external health financing sources

Coverage of NHA relatively low in developing countries but
expanding

Health financing data at decentralised levels for local decisionmaking

Financial burden of ill health and impact on impoverishment at
the household level
National Health Accounts in Africa
Number of NHA Rounds 1994 – 2004
NHA Rounds
>2
2
1
0
0
5
10
15
Countries (N=46)
20
25
30
Remaining challenges

Institutionalisation of NHA into the routine activities of Government

Underlying problems in Public Expenditure Management systems and data
reliability

Timeliness of data (NHAs and household surveys)

Collection of private expenditure health financing data

Tension between disease expenditure and general health expenditure
financial tracking

Addressing the needs of in-country policy makers vis-à-vis that of external
agencies
Health care delivery
How has information been used?
Where are the gaps?
What challenges remain?
Use of information: country examples
West Bengal, India
Aim: to monitor the performance of public sector programmes. Improve
accountability and planning at national level

Standard service use indicators & regular meetings in PHC facilities
Uganda
Aim: to link health system performance monitoring to SWAPs and national
policy process. Allows policy adjustment.

Data used in the annual health sector review process and to inform the
development of annual plans

District league tables to rank performance of districts & motivate
districts to improve indicators.

Tracking surveys – at the start of SWAP, 2001- to assess Govt systems
(financial procedures, drug distribution, HR deployment)
Major gaps in measurement

Private sector – service use, service availability
(infrastructure, human resources, services
offered)

Vital events

Efficiency of health system

Quality of health care

Effective coverage
Remaining challenges

Low capacity and motivation to use data:
– Locally
– For decision-making or for policy initiatives

Lack of ownership by health providers, who are not involved in
designing of monitoring procedure and indicators

Capacity for analysis concentrated at central level

Feedback to lower levels is limited, poor internal feedback

HMIS is often mistrusted

Selection of indicators often creates distortions

Information systems do not reflect move from project to system
performance
– India: ‘critical milestones’ & vertical project indicators
Emerging issues
Data quality and reliability

Existing information systems, but data inaccessible or
inappropriate to needs and policy process

Developing parallel monitoring frameworks rather than
adapting & use of existing data: concerns for complexity and
data reliability

HIS not always reflecting reform developments

Limited external data audit and reliance on single data
sources (Rwanda, Uganda)

Technology involved in data collection, analysis and use often
rely on bespoke software.
Parallel systems

Donor agenda regarding data collection, unsustainable

Data collection, analysis and use for policy is fragmented
– Uganda/Nepal: lack of unified data linked to SWAPs
– Private sector is often not covered (India/Uganda)

Multiple reporting requirements (Rwanda/India).

Lack of inter-sectoral information systems and unified quality
standards. (Uganda/ Rwanda)

Vertical donors-supported programmes often function well in
the short-term but may distort wider systems (e.g. Georgia &
Angola)
Information flows & level of use

One-way traffic for information
– Disaggregated data not available at sub-national level
– Information intended to be used locally, is used at national level, or for
different purpose reflecting governance & aid coordination

Information that is not aggregated nationally, less useful
internationally

Governance and stewardship at local level needs to be able to draw
effectively on aggregate & disaggregate data
– Disaggregated data feeds effectively into local planning when linked to
decentralised decision-making (TEHIP)

Peer comparisons at district level – productive vs unhelpful
Factors facilitating measurement & use of data

Health system monitoring embedded within reform process
– SWAPs/ PRSP in Uganda, Rwanda; district autonomy (TEHIP)

Unintended consequences (Afghanistan)
– Selective use of data internationally (user fees/HIV, in Uganda)

In post-conflict settings, the aid influx promotes monitoring health
systems & early warning systems. Possible inefficiencies.

The importance of governance
– Channels for policy exist (annual reviews, SWAPs meetings) & comparable
timelines.
– Communities and non-health system stakeholders involved

Large-scale data collection exercises are resource-intensive and not
synchronised with the policy process (some In-DEPTH/ LSMS).

Technology, appropriate to context
Conclusions
Effective health systems monitoring requires:

Capacity: to collect or use existing data, analyse, inform policy

Ownership

Coherence between domestic and external demands

Coherence between external agencies

Coherence between system-wide monitoring and vertical
programmes performance measurement

Coherence between assessing the performance of different system
elements

Domestic governance

Impact measurement to ensure sustainability/reform (scaling up)

Foster partnership between stakeholders
Acknowledgements
Georgia
India
Rwanda
Tanzania
Uganda
George Gotsadze
Barun Kanjilal
Vianney Nizeyimana
Graham Reid
Valeria Oliveira-Cruz
Freddy Ssengoba