Transcript An Assessment of Lot Quality Assurance Sampling (LQAS) to

Assessment of Malaria Outcome Indicators Using
Lot Quality Assurance Sampling (LQAS):
Estimation of Bed Net Coverage in Mozambique
Caitlin Biedron
ASPH M&E Fellow
CDC-Rwanda
January 15, 2009
Lot Quality Assurance Sampling (LQAS)
• Classification method used in industrial sampling to
identify batches of products (or lots) with an
unacceptable number of defective items.
• Typically implemented as part of a stratified random
sampling design: small samples are selected from all
lots in a given area.
• In each lot, the sample determines whether coverage
by a health intervention exceeds a specific target by
using a statistically determined decision rule.
LQAS Terminology
• A lot typically consists of a supervision area, such as a health district.
• Lots classified as being acceptable/unacceptable vis-a-vis the target.
• In this analysis lots are enumeration areas (what-if analysis).
• The decision rule is the minimal number of individuals in the sample
that should have the intervention.
• Selected to determine whether a population coverage target for an
intervention has been reached, such as 80% vaccination coverage.
• If this coverage target is deemed to have been reached, the lot is
classified as acceptable.
LQAS Methodology
The decision rule used when applying the LQAS method can be
represented with the composite hypothesis test:
HO : p  pc
HA : p  pc
The null hypothesis will be rejected if it is determined that the
coverage proportion (p) of the lot is below an adequate level of
coverage (pc).
The decision rule is defined by pc , the threshold value of coverage
below which program managers deem services to be unacceptable.
LQAS Methodology
The α error is also commonly referred to as type I error, termed the
provider risk in the LQAS context. A type I error is made if one rejects the
null hypothesis when it is true. It is labeled as provider risk because the
provider is at risk when the health area is identified as inadequately covered
by an intervention when it has in fact met the target coverage level.
The classification of substandard lots as acceptable is called consumer risk,
commonly referred to as the β error or type II error. A type II error is made
when one fails to reject the null hypothesis when it is false.
The magnitude of the classification error varies both with the sample size
and the maximum number of units without the intervention permitted in the
sample (d).
Mozambique MIS
• Mozambique MIS was conducted by the NMCP in
partnership with the Malaria Consortium and PMI.
• Included a sample drawn from a subset of EAs from a
population proportionate sample from the 1997 census.
• A total of 346 EAs with 5990 households were selected
from the 1510 EAs of the primary sample.
• After data cleaning, a total of 5745 household records
were available for analysis.
• The survey was conducted June - July 2007.
LQAS Parameters
• We based our decision rule on a coverage target (pc) equal to 70%
and defined our α error to be 10%.
• Year 1 Mozambique PMI target for bed net possession called for
70% of households to own at least one ITN.
• In urban clusters, the typical sample size was 20 households. In
rural clusters, the typical sample size was 15 households.
• Based on these parameters, we calculated the decision rule for both
urban and rural enumeration areas.
• The decision rule for urban clusters was 12 households; for rural
clusters it was 8 households.
LQAS Results: Any Bed-net (Urban EAs)
TABLE 1. LQAS results for household possession of any bed-net in Manica, Mozambique, 2007
Decision Rules set for α<0.10
Cluster ID
No
Yes
Total
Decision Rule
LQAS
175
11
8
19
11
failure
176
6
14
20
12
success
177
12
8
20
12
failure
178
8
12
20
12
success
179
8
11
19
11
success
180
10
10
20
12
failure
181
2
15
17
10
success
182
6
14
20
12
success
183
15
5
20
12
failure
184
8
12
20
12
success
185
7
14
21
12
success
186
11
9
20
12
failure
187
7
13
20
12
success
LQAS Results: Any Bed-net
TABLE 3. Coverage proportions, confidence intervals and LQAS result summaries for household possession of any bed-net in Mozambique
Province
Coverage Proportion
95% Confidence Interval
EAs w/adequate coverage
Niassa
0.406
(0.323, 0.489)
12 (34)
Cabo Delgado
0.376
(0.305, 0.448)
8 (34)
Nampula
0.300
(0.247, 0.353)
5 (36)
Zambezia
0.310
(0.243, 0.377)
6 (36)
Tete
0.318
(0.241, 0.394)
9 (34)
Manica
0.427
(0.363, 0.492)
12 (28)
Sofala
0.512
(0.445, 0.579)
18 (34)
Inhambane
0.315
(0.253, 0.377)
10 (34)
Gaza
0.368
(0.299, 0.437)
6 (24)
Maputo Province
0.335
(0.281, 0.389)
3 (32)
Maputo Cidade
0.481
(0.416, 0.546)
5 (20)
LQAS Results: Any ITN
TABLE 4. Coverage proportions, confidence intervals and LQAS result summaries for household possession of any ITN in Mozambique: 2007
Province
Coverage Proportion
95% Confidence Interval
EAs w/adequate coverage
Niassa
0.178
(0.130, 0.226)
0 (34)
Cabo Delgado
0.195
(0.134, 0.256)
3 (34)
Nampula
0.084
(0.053, 0.116)
0 (36)
Zambezia
0.143
(0.084, 0.201)
2 (36)
Tete
0.130
(0.081, 0.180)
1 (34)
Manica
0.368
(0.301, 0.435)
5 (28)
Sofala
0.228
(0.171, 0.284)
2 (34)
Inhambane
0.102
(0.072, 0.133)
0 (34)
Gaza
0.119
(0.072, 0.165)
0 (24)
Maputo Province
0.067
(0.042, 0.092)
0 (32)
Maputo Cidade
0.099
(0.068, 0.131)
0 (20)
Mozambique Bed-net Coverage
LC-LQAS Background
• May not be feasible to conduct LQAS in the traditional fashion;
instead only a subset of the areas in the catchment region
represented would be selected.
• Pooled data would no longer be a stratified random sample, but
instead a cluster sample, if the areas had been chosen at random.
• The recently developed large-country lot quality assurance sampling
(LC-LQAS) method is used to integrate LQAS with cluster sampling.
• We next investigated the possibility of taking only a sub-sample of
the EAs included in each province to calculate the provincial and
national coverage estimates.
• This second aggregation was done in order to compare the
estimates obtained using a smaller sample (220 EAs) to those
estimates resulting from the full sample (346 EAs).
LC-LQAS Results: Any Bed-net
TABLE 5. LC-LQAS sub-sample coverage proportions, confidence intervals and LQAS result summaries for possession of any bed-net
Province
Coverage Proportion
95% Confidence Interval
EAs w/adequate coverage
Niassa
0.395
(0.272, 0.518)
5 (20)
Cabo Delgado
0.395
(0.289, 0.500)
4 (20)
Nampula
0.316
(0.243, 0.390)
1 (20)
Zambezia
0.318
(0.210, 0.426)
3 (20)
Tete
0.355
(0.224, 0.486)
6 (20)
Manica
0.447
(0.373, 0.520)
9 (20)
Sofala
0.479
(0.390, 0.568)
10 (20)
Inhambane
0.350
(0.241, 0.459)
7 (20)
Gaza
0.374
(0.295, 0.452)
6 (20)
Maputo Province
0.330
(0.265, 0.394)
2 (20)
Maputo Cidade
0.481
(0.416, 0.546)
5 (20)
LC-LQAS Results: Any ITN
TABLE 6. LC-LQAS sub-sample coverage proportions, confidence intervals and LQAS result summaries for household possession of any ITN
in Mozambique: 2007
Province
Coverage Proportion
95% Confidence Interval
EAs w/adequate coverage
Niassa
0.128
(0.077, 0.180)
0 (20)
Cabo Delgado
0.222
(0.123, 0.321)
1 (20)
Nampula
0.067
(0.026, 0.107)
0 (20)
Zambezia
0.158
(0.051, 0.266)
1 (20)
Tete
0.149
(0.041, 0.257)
0 (20)
Manica
0.402
(0.324, 0.481)
4 (20)
Sofala
0.161
(0.108, 0.214)
2 (20)
Inhambane
0.103
(0.059, 0.146)
0 (20)
Gaza
0.135
(0.085, 0.186)
0 (20)
Maputo Province
0.067
(0.034, 0.100)
0 (20)
Maputo Cidade
0.099
(0.068, 0.131)
0 (20)
Comparison of Coverage Estimates
TABLE 7. Provincial and national coverage estimates for aggregate LC-LQAS, LQAS, and MIS cluster-samples: Mozambique 2007
Province
Any Bed-Net in HH
LC-LQAS sub
sample
full LQAS
sample
Any ITN in HH
MIS Cluster
estimate
LC-LQAS sub
sample
full LQAS
sample
MIS Cluster
estimate
Niassa
0.395
0.406
0.422
0.128
0.178
0.177
Cabo Delgado
0.395
0.376
0.378
0.222
0.195
0.196
Nampula
0.316
0.300
0.329
0.067
0.084
0.087
Zambezia
0.318
0.310
0.365
0.158
0.140
0.178
Tete
0.355
0.318
0.317
0.149
0.130
0.119
Manica
0.447
0.427
0.448
0.402
0.368
0.369
Sofala
0.479
0.512
0.504
0.161
0.228
0.217
Inhambane
0.350
0.315
0.323
0.103
0.102
0.112
Gaza
0.374
0.368
0.373
0.135
0.119
0.133
Maputo Prov.
0.330
0.335
0.297
0.068
0.067
0.057
Maputo Cidade
0.481
0.481
0.486
0.099
0.099
0.102
National Coverage
0.365
0.355
0.375
0.143
0.145
0.158
Strengths of LQAS
• Allows for pass/fail classification based on various decision rules;
• Dichotomous nature of the results can provide local supervisors with
a decisive judgment about action to be taken;
• Sample sizes are typically smaller than those required to perform
other estimation analyses;
• Inexpensive relative to traditional surveys;
• Surveys can capture variability across local areas, which
encourages local level monitoring and accountability;
• Surveys can be conducted more often, providing information more
frequently than HH surveys;
• Sample can be treated as a stratified sample for further analysis;
• With adequate sampling and appropriate weighting, can provide
estimates at aggregate levels (provincial, national);
• To accommodate large countries, sampling areas may be chosen
randomly based on cluster sampling methodology (LC-LQAS).
Limitations of LQAS
• A high level of technical competence is required to achieve more
than the basic pass/fail classification;
• Understanding of sampling frame requires knowledge of household
selection, weighting and binomial probability;
• Perception that methodology is ‘simple’ belies technical complexity;
• Decision rules may seem arbitrary if they are not based on explicit
program targets or certain thresholds related to transmission;
• Sample sizes tend to be small, so estimates may have large
confidence intervals when sample combined;
• Pass/fail outcomes do not give information on scale of change at the
local level, and it is not clear how dichotomous outcomes will be
used to track progress in programs;
• Survey EAs must be selected in accordance with health districts or
administrative units in order for results to have programmatic
relevance.
Potential Applications in the Field
• The demonstrated methods could play a tangible role in the field
where there is a growing interest in obtaining coverage estimates for
small geographic units.
• LQAS could be used to assess EA data collected by standard MIS
surveys to provide more frequent tracking of coverage indicators for
districts country-wide or within key target areas.
• The standard MIS methodology requires large sample sizes to
obtain reliable district level point estimates, making the survey both
cumbersome and costly to conduct.
• LC-LQAS could provide local and national level estimates for
coverage when it replaces the standard MIS sampling.
• LC-LQAS methodology would be applicable in those countries in
which survey sampling units are selected in accordance with health
districts or administrative units.
Potential Applications in the Field
• LQAS offers an alternative to malaria control program
managers who are interested in tracking coverage at a
local level to improve their service delivery strategies
and to adjust priorities.
• As more countries aim to control malaria and scale-up
coverage to do so, it may become necessary to offer
alternatives to the national level surveys that provide a
single set of indicators for a country and do not present
differences across local levels.
• National malaria programs are increasingly interested in
getting data from smaller geographic areas; LQAS may
be a suitable method for such programs to use.
Acknowledgements
Amy Ratcliffe, CDC Malaria Branch
Albert Kilian, Malaria Consortium
Marcello Pagano, Harvard School of Public Health
Marcia Castro, Harvard School of Public Health
Bethany Hedt, Harvard School of Public Health
Joe Valadez, Liverpool School of Tropical Medicine
Juliette Morgan, CDC-Mozambique