A collaboration between the Rwanda Ministry of Health, CNLS, SPH, INSP Mexico, UC Berkeley and the World Bank

Paulin Basinga

Rwanda School of Public Health

Christel Vermeersch

World Bank

Our team…

 Research Team  Paulin Basinga, National University of Rwanda    Paul Gertler, UC Berkeley Jennifer Sturdy, World Bank and UC Berkeley Christel Vermeersch, World Bank  Policy Counterpart Team     Agnes Binagwaho, Rwanda MOH and CNLS Louis Rusa, Rwanda MOH Claude Sekabaraga, Rwanda MOH Agnes Soucat, World Bank

P4P for Health in Rwanda

 Objectives    Focus on maternal and child health (MDGs 4 & 5) Increase quantity and quality of health services provided Increase health worker motivation  What?

   Financial incentives to providers t For more quantity And more quality  How?

 Contracts between g overnment & health facilities  When?

 Piloted in 2001-2005, full scale from 2006 5

Table 1: Output Indicators (U’s) and Unit Payments for PBF Formula OUTPUT INDICATORS Visit Indicators: Number of …

5 6 7 1 2 3 4 curative care visits first prenatal care visits women who completed 4 prenatal care visits first time family planning visits (new contraceptive users) contraceptive resupply visits deliveries in the facility child (0 - 59 months) preventive care visits

Content of care indicators: Number of …

8 9 10 11 women who received tetanus vaccine during prenatal care women who received malaria prophylaxis during prenatal care at risk pregnancies referred to hospital for delivery emergency transfers to hospital for obstetric care 12 13 14 children who completed vaccinations (child preventive care) malnourished children referred for treatment other emergency referrals

Amount paid per unit (US$)

0.18 0.09 0.37 1.83 0.18 4.59 0.18 0.46 0.46 1.83 4.59 0.92 1.83 1.83

8

Evaluation Questions Did P4P improve… … the quality and quantity of maternal and child health services?

… the health of the population?

Conceptual framework for quality

What They Do (Quality) Production Possibility Frontier Productivity Gap Conditional on Ability Actual Performance What they know (Ability/Technology)

Identifying the impact of P4P

 When we see a change in outcome, how do we know it is caused by P4P?  And not by something else  Evaluate the impact = identifying a comparison group  Equivalent to the treated group in all aspects  Except that they receive the treatment  Gold standard: randomized evaluation 11

Evaluation Design

 Phased roll-out at district level  Identified districts without P4P in 2005  Group districts into “similar pairs”  based on population density, location & livelihoods  Randomly assign one to treatment and other to control  Phase I: 12 districts, started 2006  Phase II: 7 districts, started 2008  Unit of observation is health facility 12

Rollout of P4P 2006 – 2008

13

A few challenges



The decentralization “surprise”

 A few new districts had some facilities with P4P– must be treatment  Exposure time to the “treatment”  Is it more money? Or more incentives?

 Other interventions & time trends  Coordination with multiple donors 14

Data

 Independent data  Facilities level     Utilization (quantity) Structural quality Knowledge (vignettes) Process quality (patient exit surveys)  Household level    Utilization Process quality Health outcomes 15

Sample

 Out of 30 districts  12 Phase I (treatment)  7 Phase II (comparison) 

165 health facilities

 All rural health centers located in 19 districts  2156 households in catchment areas  Power calculations based on expected treatment effect on prenatal care visits, institutional delivery  Panel data: 2006 and 2008 16

Econometric model



Basic difference-in-differences model specified as a two-way fixed effect cross-sectional time-series regression models.

Y ijt

 

j

  2008   

PBF jt

 

k



k X kit

 

ijt

where : •

Y ijt

is the outcome of interest for individual i living in facility j’s catchment • area in year t; • • • • •

γ 2008



PBF j,2008



X ijt j itk

= 1 if facility j was paid by PBF in 2008 and 0 if otherwise; are facility fixed effects; =1 if the year is 2008 and 0 if 2006; are time varying individual characteristics; is a zero mean error term.

17

Evaluation design challenges

 Organizational  Managing expectations  The John Henry effect in practice  Building capacity  Time & effort  Technical  Small sample size  Reconciling provider and client data 18

Baseline, health facilities

Observations

Treatment

75

Control

70

Difference p value* Expenditures and Budget Shares

Log Total Expenditures (2006) Log Total Expenditures (2008) Personnel Budget Share Medical Supplies Budget Share Non-medical Budget Share

Staffing

Medical Doctors Nurses Other Clinical Staff Non-clinical Staff

Structural Quality (Baseline 2006)

Curative Care Delivery Prenatal Care Immunization Laboratory 15.81 16.91 0.46 0.22 0.32 0.05 6.31 4.13 5.25 0.80 0.78 0.96 0.94 0.49 (1.04) (0.71) (0.23) (0.19) (0.25) (0.23) (6.90) (3.09) (3.56) (0.07) (0.11) (0.15) (0.17) (0.32) 15.61 16.99 0.49 0.20 0.30 0.05 5.48 4.47 5.33 0.81 0.79 0.97 0.94 0.43 (1.01) (1.08) (0.26) (0.19) (0.22) (0.27) (3.30) (4.05) (5.09) (0.07) (0.10) (0.11) (0.15) (0.32) 0.200 -0.083 -0.031 0.013 0.018 0.003 0.826 -0.335 -0.076 -0.01 0.00 -0.01 0.00 0.06 All of the data, except Log Expenditures 2008, are measured at baseline prior to the intervention. Data are n (%) or mean (SD). Sample size varies slightly according to characteristic measured *P-values are for cluster-adjusted t-test (continuous variables). 0.418 0.568 0.555 0.705 0.726 0.940 0.409 0.554 0.901 0.575 0.840 0.285 0.897 0.402 19

Baseline, utilization of maternal health services

Treatment Control Difference p-value* Maternal care utilization

Any prenatal care (=1) 4 or more prenatal care visits Delivery in health facility (=1) 0.95

0.18

2.76

0.35

(0.21) (0.38) (0.84) (0.48) 0.96

0.11

2.62

0.36

(0.20) (0.31) (0.80) (0.48) -0.01

0.07

0.14

-0.01

Data are n (%) or mean (SD). Sample size varies slightly according to characteristic measured *P-values are for cluster-adjusted t-test (continuous variables) or chi-squared (dichotomous variables) tests of independence. 0.773

0.036

0.180

0.801

20

Baseline, women 15-49 with birth in last 24 months

Treatment

620

Control

670

Difference p-value*

Observations

Maternal characteristics

Age Primary education or more (=1) Currently married/union (=1) Partner present (=1) Number of pregnancies (Parity) Number of living children Health insurance (=1) 30.89

0.42

0.94

0.98

4.32

3.39

0.55

(7.05) (0.03) (0.23) (0.14) (2.46) (1.93) (0.50) 31.22

0.35

0.91

0.97

4.33

3.51

0.52

(6.85) (0.04) (0.29) (0.17) (2.43) (2.62) (0.50) -0.33

-0.01

0.04

0.01

-0.01

-0.12

0.04

0.594

0.889

0.214

0.325

0.969

0.490

0.591

Household characteristics

Number of household members Ownership of land (=1) Poverty inderx (Value of animals, durable

Poverty index quartile 1 Poverty index quartile 2 Poverty index quartile 3 Poverty index quartile 4

Distance : household - facility (in km) 5.15

0.93

0.22

0.25

0.27

0.25

3.24

(1.70) (0.25) (0.42) (0.43) (0.45) (0.43) (2.00) 5.40

0.87

0.22

0.25

0.28

0.24

3.40

(1.94) (0.33) (0.42) (0.43) (0.45) (0.43) (2.01) -0.25

0.06

0.00

0.00

-0.01

0.01

1.58

0.145

0.126

0.994

0.885

0.885

0.772

0.667

Data are n (%) or mean (SD). Sample size varies slightly according to characteristic measured *P-values are for cluster-adjusted t-test (continuous variables) or chi-squared (dichotomous variables) tests of independence. 21

22

Impact on structure quality

Impact of PBF N Treatment 2008 (=1) SD Time Trend 2008 (=1) SD

Availability of vaccines Prenatal care service 155 155 0.703** (0.355) 0.062 (0.426) -0.514*** -0.659*** (0.122) (0.100) Delivery service 155 0.239 (0.286) -0.484*** (0.083) 23

Kernel Non parametric regression practice-competency at baseline .3

.4

Competence .5

.6

Control facilities Treatment facilities .7

.8

Kernel Non parametric regression practice-competency at follow up .3

.4

Competence .5

.6

Control facilities Treatment facilities .7

Impact on quality of prenatal care

Impact of PBF Time Trend N Treatment 2008 (=1) SD 2008 (=1) SD Total Prenatal Quality Score (standardized score) Tetanus vaccine during prenatal visit (=1)

3683 2810 0.157*** (0.048) 0.054** (0.023) 0.090*** 0.032

(0.032) (0.028)

Productivity ratio

3757 0.072*** (0.017) 0.036*** (0.012)

Treatment * 2008 * Competency > 75% (=1)

3739 0.088*** (0.019) NA NA 26

Impact on quality of prenatal care

0.20

0.15

0.10

0.05

0.00

-0.05

-0.10

-0.15

-0.10

-0.13

Baseline (2006)

Control facilities

0.15

0

15 % Standard deviation increase due to PBF

Follow up (2008)

Treatment (PBF facilities)

27

Impact on use of prenatal care

Impact of PBF Time Trend N

Number of prenatal visits received 2223

Treatment 2008 (=1) SD

-0.028 (0.066)

2008 (=1)

0.237***

SD

(0.056) Likelihood of first prenatal visit in first trimester 2223 Made 4 or more prenatal care visits 2223 0.020 (0.036) 0.009 (0.039) 0.163*** 0.121*** (0.029) (0.023) 28

Impact on use of maternal services

Impact of PBF Time Trend N

Institutional delivery (=1) 2108 Delivery attended by Qualified Provider (=1) 2274 Use of any modern contraceptive method (=1) 3121

Treatment 2008 (=1) SD

0.077** (0.034) 0.083** (0.035) -0.017 (0.023)

2008 (=1)

0.134*** 0.195*** 0.245***

SD

(0.023) (0.028) (0.033) 29

Impact on institutional delivery

60.0

55.6

7.3 % increase due to PBF

50.0

49.7

40.0

30.0

36.3

34.9

Baseline (2006) Control facilities Follow up (2008) Treatment (PBF facilities)

30

What our results tell us

 You get what you pay for !  Need to get prices “right”  Returns to effort important   Bigger effects in things more in provider’s control Patient or community health workers for prenatal care/Immunization  Provide incentives directly to pregnant women? (conditional cash transfer program).

 Financial incentive to community health workers  Low quality of care : additional training coupled with P4P  Evaluation feedback useful 31

35