29th-2 nd

/03/2009 Conference AfrEA - NONIE - 3ie Cairo, Egypt

Experiences with Lot Quality Assurance Sampling

Cairo – April 2009 Olivia Nakayiza & Luc Vanhoorickx – Save the Children in Uganda

Programme Overview

Save the Children in Uganda (SCiUG) is a unification of Save the Children Norway (Managing Member), Save the Children Denmark, Save the Children USA, Save the Children UK, Save the Children Sweden and Save the Children Italy • • • • In 2008: SCiUG programmes in over 26 districts in 4 regions of Uganda SCiUG works with 59 partners including local NGOs, CBOs and local government.

Total 313 staff as per December 2008 Reached 213,051 children (116,137 girls & 96,915 boys)

Strategic Choices

In the year of unification, Save the Children in Uganda (SCiUG) developed a strategic framework that builds both on the strengths of participating members as well as on emerging analyses of the child rights situation that calls for more integrated programming

( “Straw-man” )

Strategic Framework

Children in Uganda realize their rights to be safe, educated and healthy

Protection Education HIV/AIDS CRC Livelihoods Health / Nutrition

SCiUG has also continued to scale up its emergency response capacity to deal with emergency contexts, focusing primarily on emergency education and child protection

Community Based Family Planning Program Background

Increased voluntary use of key family planning services and RH behaviors among women and men of RA

   A 4 year USAID/FLEXIBLE Fund Community Based FP/RH- Luwero, Nakaseke and Nakasongola Districts in Uganda Women of Reproductive age ( 15-49) 68,826, Married WRA 33,036, Men 68,423. Total # of beneficiaries 170,285 Partnerships: District, MoH, etc

FP/RH Strategies and Intermediate results

strategic objective

: “Increased and sustainable use of key family planning services and RH behaviors among women and men of reproductive age”.

INTERMEDIATE RESULTS

IR-1

: Increased knowledge, interest and use of FP/RH services

IR-2

: Improved quality of FP service delivery by providers at the facility and community level

IR-3:

Increased access of communities to FP services and information I

R-4

: Improved social and policy environment for FP/RH services and behaviors   Partnership Defined Quality (PDQ) process…… Behavior Change Communication, …   community-based services to increase access to FP methods ,...

Build capacity of district staff, …

Results Frameworks

Where is results monitoring?

Time

Goal Strategic Objective IR 1 IR 2 IR 3 Strategies Impacts Longer Term IR 4

RESULTS MONITORING:

Outcomes Medium

Check change in

Term

knowledge, behavior, status

Inputs & Outputs Shorter Term Activities

Monitoring & Evaluation => LQAS

Higher level monitoring

  Measure outcome indicators : compare with baseline & final targets Effects of Behavior Change campaign: … behaviors From the Monitoring we knew that we were generally carrying out the activities as planned, but wanted to Monitor&Evaluate whether it all had the desired effect/result/outcome/impact

=> information needed = population/beneficiary based => sampling needed !

LQAS = Lot Quality Assurance Sampling  “determination of the quality of a lot by sampling”   small sample size, typically 19 per area comparison between areas (lots) possible Lot = sub county from a district

What is LQAS?

•A sampling method that: Can be used locally, at the level of a “supervision area,” to identify priority areas or indicators that are not reaching average coverage or an established benchmark •Can provide an accurate measure of coverage or health system quality at a more aggregate level (e.g. program area)

LQAS: dichotomic - 19

Lot Quality Assurance Sampling

 in general small sample of 19 provides an acceptable level of error for making management decisions; at least 92% of the time, it identifies whether a coverage benchmark has been reached or whether a supervision is substantially below the average coverage of a program area.  Samples larger than 19 have practically the same statistical precision as 19. They do not result in better information, and they cost more For example, what LQAS can: just by sampling 19 women in a targeted population, at least 92% of the time LQAS will determine correctly whether yes or no these women have adopted the family planning method.

  LQAS cannot be used for coverage estimates in lot / district!

Only if it’s above/below target  Decision Rule

• • • •

Some indicators that were measured

Contraceptive Use (CU) or Contraceptive Prevalence Rate (also known as Met Need) Percentage of women of reproductive age (WRA) 15-49 who are married or in union using (or whose partner is using) a modern method of family planning Percentage of women of reproductive age (WRA) currently married or in union who are fecund (not pregnant or unsure if they are pregnant and not sterilized) who desire to have no more children or postpone childbearing, and who are not currently using a method of family planning (Unmet Need for Family Planning) Percent of Demand (Met need + Unmet need) satisfied Knowledge about family planning methods:

FP METHODS THAT WERE MEASURED

• Depo-Provera • Pills • Condoms • Norplant and Tubal Ligation

LQAS in Family Planning project

As the mid-term evaluation of the Family Planning project

Effects/outcomes of project interventions:

… behaviors % of women of reproductive age who report having access to FP services % of women of reproductive age who were counseled about the birth spacing • • Sampling: Systematic Random Sampling from … 5

lots

were chosen for different districts 2-step random sampling procedure. Communities were selected using systematic random sampling. HH selected using classical random sampling Data collection tools: Adapted from FlexFund survey guidelines Data analysis: Standard LQAS tables in Excel worksheets for analysis & graphs

Process

      5 “supervision areas” Random samples of 19 from monitoring records of women of reproductive age (WRA) Interchanged extension workers for data collection Minimal data-entry where PDA were used Instant results Fast feedback

LQAS Summary Tabulation

Indicators

% of women 15-49 years who are not pregnant or are unsure, who are using a modern family planning method (Contraceptive Prevalence Rate or Met Need) % Women of reprod. age know about at least three methods of family planning (Knowledge about Family Planning Methods) % of women of reproductive age that receive counseling about birth spacing % of women of reproductive age who report having access to FP services % of women of reproductive age who report discussing FP with a health worker or family planning worker/promoter % of WRA (15-49) currently married or in union who are fecund (not pregnant or unsure if they are pregnant and not sterilized) who desire to have no more children or postpone childbearing, and who are not currently using a method of family planning (Unmet Need for Family Planning) Total Demand for Family Planning Percent of Demand Satisfied % of sexually active respondents who report discussing FP issues with their spouse or (cohabitating) sexual partner in the past 12 months % of respondents that lives within 5 km of a family planning service delivery point (SDP), [among women who know where to obtain a method] (Proximity to Family Planning Service Delivery Point)

Lot 1

OK OK

C

OK

T & C

N/A OK

C

Priorities where Target ( T ) and/or Coverage ( C ) is not achieved Lot 2 Lot 3 Lot 4

C

OK OK OK OK N/A

T C

OK OK

C

OK

T

N/A

T & C

OK OK OK OK OK OK N/A OK OK

Lot 5

OK OK OK OK OK N/A OK OK

Target LQAS Coverage 95% CI

50% 70% 60% 70% 70% 30% 60% 30% 68.9% 100.0% 68.8% 96.0% 67.4% 34.2% 93.1% 72.6% 54.3% 73.0% 10.8% 0.0% 8.7% 3.3% 9.5% 13.6% 4.9% 10.2% 8.7%

FP LQAS OUTCOMES

LQAS Family Planning '07

100.0% LQAS Coverage 90.0% 80.0% 70.0% 60.0% 50.0% 40.0% Target 30.0% 20.0% 10.0% 0.0% Contraceptive Prevalence Rate Knowledge about Family Planning methods Counseling about birsth spacing Access to Family Planning services Discussing Family Planning with health worker Unmet Need for Family Planning Total Demand for Family Planning Percent of Demand Satisfied Discussing Family Planning with spouse of sexual partner Proximity to Family Planning Service

Average Coverage

LQAS Health outcomes

100.0% 90.0% 80.0% 70.0% 60.0% 50.0% 40.0% 30.0% 20.0% Bas eline MTE FY07 Final Target 10.0% 0.0% Daily water use per capita above 20l Appropriate hand washing behavior Bring their children to appropriate provider with fever % of women reporting current use of modern contraceptives Giving complementary foods at 4 to 6 months Give more fluids during diarrhea episode Know 2 or more causes for growth faltering

After the quantitative survey…

• •  LQAS gives the numbers (it’s

quantitative

) Identifies the problem  How big the problem is  Where the problem is  BUT NOT why we have the problem… Hence the need for

qualitative

research /discussions to o Explain quantitative results o Identify the reasons why o Provide recommendations on program design and strategies

•

Lessons Learned (1)

Internal interviewers were used but swapped to reduce bias Quantitative LQAS survey replicated yearly • Although some statistical analysis is not possible with LQAS, indicators can be analyzed further to reveal additional information e.g. analyze which ‘part’ of knowledge / behavior adopted => focusing messages

• •

Lessons Learned (2)

Results discussed w/ program staff during participatory feedback meetings: discussions & decision making w/officers + review meeting w/extension staff Guided by principal areas for improvement identified in LQAS surveys: provide recommendations to guide the team in second half of the project and inform planning

PDA Experience/Lessons learnt (3)

• PDA to be easy to carry in the pocket of the riding gear • PDA contained automatic skip patterns • Of completeness of the data, all required questions were responded to • a reminder in most required questions in the PDA which could beep • Once an electronic questionnaire was filled in, data is instantly stored and possibilities were built-in to review the data • data is clear in terms of spellings and uniformity • Data entry time is obviously reduced to zero

LQAS & the MTE

Thank You !

Experiences with Lot Quality Assurance Sampling

Cairo – April 2009