Transcript Impact Evaluation of Performance

Paying Health Care Providers for
Performance: Evidence from Rwanda
Paul Gertler
UC Berkeley
January 2009
Collaboration

Research Team





Paulin Basinga, National University of Rwanda
Paul Gertler, UC Berkeley
Jennifer Sturdy, World Bank
Christel Vermeersch, World Bank
Policy Counterpart Team


Agnes Binagwaho, Rwanda Rwanda MOH
Agnes Soucat, World Bank
Overview







Background/Motivation
Rwanda
Program Description
Evaluation Design and Methodology
Baseline Descriptive Statistics
Impact of PBF
Next Steps
Context: Developing World

Africa





Very poor health status
Weak health care systems
Brain drain – doctors & nurses leaving
Massive AID could be wasted
World Wide (WDR 2004)



Low Quality of Care
Training/technology have had small effect on Quality
Provider absenteeism high & effort low
Pay For Performance

Pay Medical Providers a bonus based on
performance measurement



Improve quality of care and outcomes
Improve job satisfaction & retention
Organization Challenges



Individuals versus team incentives
Measuring performance
Cheating/Misreporting
Rwanda: Central African Country




9 million people
Genocide in early 1990s
GNP per capita: 250 US$
Weak Health Care Infrastructure

36 Hospitals, 369 health centers
Doctors: 1/50,000 inhabitants
Nurses: 1/3,900 inhabitants;

17% of nurses in rural areas



Poor health status, but getting better
MDG 4: Infant and child mortality
Infant mortality
Under 5 mortality
350
196
300
250
152
151
200
99
150
100
85
107
86
60
50
0
DHS 1992
DHS 2000
DHS 2005
DHS 2007
Performance-based Financing (PBF)


Local Initiative
Objectives



Financial incentives to providers to see more
patients and provide higher quality of care



Increase quantity & quality of health services provided
Increase health worker motivation
Increased resources
Financial incentives
Operates through contracts between


Government
Health facilities providing services
Quarterly Payment to Facility i in period t
Pj = payment per unit of each PBF service j
Uijt = number of patients using service j in
facility i in period t
Qit = facility i’s quality in period t
PBF Facility Quality Score
1  Sikt  0
Where Skit = facility i’s Quality index of Service k

Indicator types:


Structural: Availability of medical equipment/drugs
needed to deliver adequate medical care

Process: Clinical content of care (CPGs)
PNC Quality Indicators
Monitoring Facility Reporting

District Comite de Pilotage



Random utilization audit (once quarterly)


One focal point per administrative district
Random quality audits (once quarterly)


Approves quarterly payment
Based on facility reports & independent audits
District supervisors based in District Hospital
Interview random sample of patients


Identify phantom patients
MSH study – less than 3-5% phantom patients
Evaluation Questions: Did PBF…

Increase the quantity of contracted health
services delivered?

Improve the quality of contracted health
services provided?

Improve child health status?
Identification Strategy



During decentralization, phased rollout at district level
Identified districts without complete PBF in 2005
Group districts into “similar pairs” based
on population density & livelihoods






Decentralization reallocated districts
Some new districts had PBF in an area of the new district
Gov’t rolled PBF to remaining clinics (treatments)
Districts matched to these partials controls
Others: randomly assign one to treatment and other to control
8 pairs
Isolating the incentive effect

PBF



Performance incentives
Additional resources
Compensate control facilities with equal
resources



Average of what treatments receive
Not linked to performance
Money allocated by the health center management
20
Sample

165 health facilities


2145 households in catchment areas


all rural health centers located in districts
Random sample of 14 per clinic
Panel data: 2006 and 2008
Survey Content

Health Facility Data





Financials and Human resources
Lab test, equipment and medicine availability
Provider interview for competency (vignette)
8-10 patient exit Interviews for prenatal process quality
Household survey



Socio-economics
Utilization
Health outcomes
Health Facility Results

Did we isolate incentives effect?





Log expenditure between Tr and Phase II
Did randomization balance treatment/control
groups?
Did utilization increase?
Did structural quality improve?
Did process quality improve?


Prenatal Care (PBF pays for this)
Child Curative Care (PBF does not pay for this)
Log Expenditures
Year
2006
2008


Treatment
15.812
(1.042)
16.906
(0.71)
Control
15.612
(1.007)
16.989
(1.08)
Difference
0.200
0.241
-0.083
(0.14)
P-Value
0.418
0.568
Randomization balanced baseline
Follow-up balanced, so difference in follow-up
outcomes due to incentives not resources
Baseline Balance


Utilization (PBF)
Structural Quality



Availability of staff, equipment & drugs
needed to deliver care (PBF)
Little room to improve
Process Quality


Competency (Vignettes)
Process Quality (Patient exit survey)
Baseline Expenditures & Staffing
Prenatal Competency & Quality

Standardized vignette presented to provider




Unprompted responses for competency
Measure of ability/knowledge
Based on Rwandan Clinical Practice Guidelines
Process quality



Patient exit interview for process quality
Clinical content of care
Provider effort
Quality Conceptual Framework
What They Do:
(Quality)
Production
Possibility
Frontier
What They Know (Ability/Technology)
Returns to Training/Technology low
(data from 12 countries)
PPF
What They Do
Actual
Performance
Ability/Technology (More Training & Equip/Drugs
)
Goal: Use Pay for Performance to
Close Productivity Gap
PPF
What They Do
Productivity Gap
Conditional on Ability
Actual
Performance
Ability/Technology
Prenatal Provider Competency & Quality
Impact of PBF: Statistical methods


Have balance at baseline on all key outcomes
Use difference in differences analysis





Not a pure randomized experiment
Clustered at district year level
Facility Fixed Effects
Year dummy
Controls: age, parity, education, household
size, health insurance, land, value of assets
Impact on # of Prenatal Care Visits
3.05
3
2.95
2.9
2.85
2.8
2.75
2.7
2.65
2.6
2006
2008 No PBF
2008 PBF
Impact on 4+ Prenatal Visits and
Facility Delivery
0.7
0.6
0.5
0.4
2006
2008 No PBF
2008 PBF
0.3
0.2
0.1
0
4+ prenatal vists
Facility Delivery
Baseline Prenatal Provider Competency & Quality
.5
.4
.3
.2
.1
.2
.4
.6
.8
Provider Prenatal Competency (Knowledge)
1
P4P Impact on Prenatal Care Quality
0.4
0.35
Change in Quality
0.3
0.25
0.2
0.351
0.15
0.1
0.05
0.127
0.145
Low
Medium
Provider Competency (Ability)
0
High
Impact Probability Sick in Last 4 Weeks
0.6
0.5
0.4
2006
2008 N0 PBF
2008 PBF
0.3
0.2
0.1
0
0-12 months
24-47 months
Impact on Child Height


0-11 months = +0.28***
24-47 months = +0.86***
Results Summary



Balanced at baseline
Expenditures same, so isolate incentives
Impact on utilization


Impact on prenatal quality



Delivery & Child prevention, but not prenatal
Bigger for better doctors
Reduced child morbidity
Taller children
Policy