Transcript FPP Evaluation results - Part 1

Supporting community action on AIDS in developing countries
Evaluating Impact
The Alliance experience
with the Frontiers
Prevention Programme
Presentation to
IMPACT EVALUATION
WORKSHOP
24th May 2011
Supporting community action on AIDS in developing countries
Contents
•
•
•
•
•
•
Introduction to the Alliance
The Frontiers Prevention Programme
The FPP evaluation
Challenges to the Evaluation
Evaluation Results
Conclusions
Supporting community action on AIDS in developing countries
Alliance Mission: To reduce the spread of HIV
and meet the challenges of AIDS
Alliance Vision: A world in which people do not
die of AIDS
Supporting community action on AIDS in developing countries
A Global Alliance
•
Currently working in
over 33 countries
•
Supporting communitybased organisations
•
Building national
capacity
•
Spreading good
practice and lessons
learned
•
Emphasising the
importance of working
with marginalised and
key populations
INSTITUTO
NACIONAL
DE SALUD
PÚBLICA (INSP)
Supporting community action on AIDS in developing countries
Evaluating Impact:
A case study on the
Frontiers Prevention Program
Supporting community action on AIDS in developing countries
Overview of Frontiers Prevention Project
• Aim: To make a significant contribution to
reducing HIV infections in relatively lowprevalence countries
• BMGF-funded: US$25m for three years
• Comprehensive packages of interventions
targeting ‘key populations’ (KPs) in Ecuador,
India, Cambodia and Morocco
• KPs include: sex workers and clients; PLHIV;
MSM; IDUs; and STI service users
Supporting community action on AIDS in developing countries
FPP interventions
28 interventions in 8 clusters:
1.
2.
3.
4.
5.
6.
7.
8.
Strengthening (NGO & others) capacity to work with KPs
Implementing peer outreach activities
KP collective mobilisation for advocacy
Implementing KP mutual support, cultural and solidarity
building
Implementing KP risk reduction skills building activities
Developing & disseminating IEC for KPs
Strengthening clinical capacity & quality of services
Implementing anti-stigma and HIV prevention with the
general public
Supporting community action on AIDS in developing countries
FPP conceptual framework
GOAL
Decrease
In HIV
incidence
amongst
KPs
Decrease
Decrease
in
in HIV
HIV
Incidence
Incidence
in site
in site
Supporting community action on AIDS in developing countries
FPP conceptual framework
PURPOSE
Decrease
in KP risky
behaviour
Decrease
in KP STI
Prevalence
GOAL
Decrease
in HIV
Incidence
amongst
KPs
Decrease
in HIV
Incidence
in site
Supporting community action on AIDS in developing countries
FPP conceptual framework
INTERMEDIATE
OUTCOMES
Decrease
in KP risky
behaviour
Enabling
Environment
Service and
Commodity
provision for
KPs
PURPOSE
Empowerment
for prevention
for KPs
Decrease
in KP STI
Prevalence
GOAL
Decrease
in HIV
Incidence
amongst
KPs
Decrease
in HIV
Incidence
in site
Presented by H. Gayle at BNA IAC (similar to Avahan and APSACS)
Supporting community action on AIDS in developing countries
Decrease
in KP risky
behaviour
Enabling
Environment
Service and
Commodity
provision for
KPs
Empowerment
for prevention
for KPs
Decrease
in KP STI
Prevalence
Decrease
in HIV
Incidence
amongst
KPs
Decrease
in HIV
Incidence
in site
Will the implementation of the FPP program have
an impact in terms of in changes in behaviour
and STI/HIV prevalence in sites?
Supporting community action on AIDS in developing countries
FPP evaluation model
Qualitative study (India)
• India (Horizons, IHS Hyderabad)
• Cross section longitudinal design with two observations:
baseline (2004) and follow-up (2006).
• IDIs, FGDs and SPSS to compare main sub-categories
of SWs (brothel-based, street-based, and home-based)
and MSM.
• Baseline: 4 intervention and 4 comparison sites while
for follow-up: 3 + 2.
Supporting community action on AIDS in developing countries
FPP evaluation model
Quantitative study (India & Ecuador)
• Survey instrument developed by a multidisciplinary
team of researchers & validated with KPs
• Socioeconomic, demographic, behavioral, knowledge
and HSV2 & syphilis biomarkers
• Baseline (Q3-Q4 2003):
• India: 2,182 FSW & 2,929 MSM in 24 evaluation sites
• Ecuador: 2,026 FSW & 2,093 MSM in 6 evaluation sites.
• Follow-up (Q3-Q4 2007): Revised to include data on
exposure to interventions
• India: FSW: 2,374 & MSM: 2,014
• Ecuador: FSW: 1,760 & MSM: 1,676
Supporting community action on AIDS in developing countries
Design: intervention & comparison sites
INDIA (AP)
ECUADOR
SW MSM
PLHIV IDU
Supporting community action on AIDS in developing countries
Avahan in AP: disrupting the counterfactual
Adilabad
Nizamabad
Srikakulam
Karimnagar
Medak
Warangal
Hyderabad
Visakhapatnam
Khammam
East
Godavari
Rangareddy
Nalgonda
Mahabubnagar
Guntur
Prakasam
Kurnool
Ananthapur
West
Godavari
Cuddapah
Nellore
Krishna
APSACS + HLFPPT
8
APSACS + ALLIANCE
8
ALLIANCE
6
HLFPPT
1
• No district uncovered
Chittoor
• No overlapping of mandals
• Information shared across partners
Supporting community action on AIDS in developing countries
Results 1: India FSW
99%
100%
80%
74%
98%
70%
60%
50
40
40%
23
17
20%
7
6
23
17
5
23
22
10
0%
NFPP
Baseline
Condom last client
FPP
Baseline
Condom regular
NFPP
Follow-up
Syphilis
FPP
Follow-up
HSV-2
Supporting community action on AIDS in developing countries
Results 2: India MSM
96%
100%
91%
80%
58%
60%
45%
40%
27
26
22
20%
41
37
12
15
NFPP
Baseline
FPP
Baseline
29
8
14
9
13
0%
Condom last
Condom female
NFPP
Follow-up
Syphilis
FPP
Follow-up
HSV-2
Supporting community action on AIDS in developing countries
Key challenges for the evaluation (1)
Ethical concerns
• Control sites: ‘withholding’ interventions from those
who need them
Response: Limited funding only allowed
saturation of 20 sites.
• Biomarkers: biomarker samples and HIV
Response: Respondents provided with health
benefit. HIV testing was not done. (A mistake?)
Supporting community action on AIDS in developing countries
Key challenges for the evaluation (2)
Consent and confidentiality with KPs
• Over-sensitisation towards KP-related issues led to
researchers being overly guarded
• ‘Over-dependence’ on KPs made the process stilted
and artificial.
• In some instances, KPs used their ‘KP status’ to
complicate an otherwise simpler process which
could have been more effective.
Supporting community action on AIDS in developing countries
Key challenges for the evaluation (3)
Striving for inclusiveness
• Challenge of accommodating differing opinions and
approaches within the community
• Time-consuming
• Questions around the reliability of number of KPs
generated through the PSA
Supporting community action on AIDS in developing countries
Key challenges for the evaluation (4)
Management issues
• Multiple stakeholders with different expectations,
opinions and agendas
• High staff turnover
• Coordination and leadership
• Geographical lack of proximity of research partners
• Implementers as researchers created bias and lack
of objectivity
Supporting community action on AIDS in developing countries
Key challenges for the evaluation (5)
Maintaining control sites
• Avahan (2003) initiated comparable interventions
with similar set of organisations (IHAA included)
• Control sites were lost as a result.
• Lack of adequate coordination
• The research design was not flexible or sufficiently
nimble to accommodate the changes.
Supporting community action on AIDS in developing countries
Adapting the evaluation: dose response
FPP Interventions
FPP Interventions
FPP Interventions
SITE A
SITE B
SITE C
Supporting community action on AIDS in developing countries
Dosage evaluation for community interventions
Increased site level:
a) ‘social capital’
b) +ve behaviour
change
c) +ve biomarker
‘Continuum of interventions’
‘Comparison
sites’
in Ecuador
IAI sites in
India
FPP full sites
In India
& Ecuador
Measurements of program intensity and population coverage at site
Supporting community action on AIDS in developing countries
“These positive results, even considering the limitations of the
evaluation design, suggest that a strong community component may
significantly potentiate prevention impact. It is all the more
convincing because over time Avahan became increasingly focused
on community mobilization. While this supports the arguments in
favour of community participation, it also calls for more robust
evaluation in the future to characterize and quantify the benefits and
costs of different approaches for community engagement and
mobilization to accompany the provision of prevention services”
Gutierrez JP., McPherson et al, (2010), Community-based prevention leads to an increase in condom
use and a reduction in sexually transmitted infections (STIs) among men who have sex with men
(MSM) and female sex workers (FSW): the Frontiers Prevention Project (FPP) evaluation results,
BMC Public Health (Page 10)
Supporting community action on AIDS in developing countries
Conclusions & discussion points
• Maintaining a counterfactual is not easy
– Contamination of sites
– Understand the response-scape will evolve—be flexible
• For complex programs defining a generic intervention
‘logic model’ that is evaluable is hard
– Multiple causal factors, multiple attributions
• Significant challenges in assessing multi-component
interventions with modest effect.
–
–
–
–
–
Ability to distinguish the marginal effect of the ‘intervention’
Cost of these types of evaluations
Big investment in monitoring and surveys
Maintaining objectivity
Coordination and leadership
Supporting community action on AIDS in developing countries
Conclusions & discussion points (continued)
• Be very careful when undertaking ‘impact’
evaluations
• Look at ‘dosage evaluation’ approaches
• Work in synergy with other NGOs and donors to
evaluate ‘key development question’ evaluations
(don’t try to ‘prove’ attribution)
• Measure output/process data that we KNOW has
proven impact
• Focus on measuring efficiency and quality of
interventions
Supporting community action on AIDS in developing countries
Thank you!
Supporting community action on AIDS in developing countries
Acknowledgements
•
•
•
•
•
•
•
Fiona Samuels (ODI)
JP Gutierrez (INSP)
Stef Bertozi (INSP/Gates)
Lalit Donanda (ASCI)
AP Office staff (Alliance)
Key Population groups
Jeff O’Malley and Jerker Edstrom (ex Alliance)