Transcript 32-Evaluation_of_surveillance_systems_2011

Evaluation of surveillance
systems
17th EPIET Introductory Course
Lazareto, Menorca, Spain
September – October, 2011
Günter Pfaff
Günter Pfaff 2009/10 / Viviane Bremer 2008 / Preben Aavitsland / FETP Canada
The surveillance loop
Health Care System
Public Health Authority
Reporting
Data
Event
Analysis &
Interpretation
Decision
Information
Intervention
(Feedback)
Importance of evaluation

Obligation
 Does the system deliver?
 Credibility of public health service

In reality
 Often neglected
 Basis for improvements

Learning process
 EPIET training objective
 ”Do not create one until you have evaluated one”
Does the surveillance system…
Detect trends? Epidemics?
 Provide estimates of morbidity and mortality?
 Identify risk factors?
 Stimulate epidemiologic research?
 Assess effects of control measures
 Lead to improved clinical practice?
 Lead to new/improved control measures?
 Lead to better advocacy and increased
funding?

Criteria to look at







Simplicity
Flexibility
Acceptability
Data quality
Sensitivity and Predictive
value positive (PvP)
 Capture-recapture
Representativeness
Timeliness
CDC guidelines
Simplicity
As simple as possible while meeting the objectives
 Structure





Information needed
Number and type of sources
Training needs
Number of information users
Functionality




Data transmission
System maintenance
Data analysis
Information dissemination
Components of system
• Population under surveillance
• Period of data collection
• Type of information collected
• Data source
• Data transfer
• Data management and storage
• Data analysis: how often, by whom, how
• Dissemination: how often, to whom, how
Confidentiality,
security
Flowchart (HIV in Norway)
Reference
laboratory
Blood sample for HIV test
Primary
laboratory
HIV reporting form,
part 1
Lab report and HIV reporting form
HIV infection
Patient
AIDS
Death, emigration
Primary care
physician
Hospital physician
HIV reporting form, part 2
(Prompting if necessary)
AIDS reporting form
Semiannual check
Oral information
Semiannual check
National Institute
of Public Health
Flexibility

Ability of the system to accommodate
changes
 New event to follow-up
 New data about an event
 New sources of information
Acceptability

Willingness to participate in the system




Participation (%) of sources
Refusal (%)
Completeness of report forms
Timeliness of reporting
Acceptability

Factors influencing the willingness to
participate






Public health importance
Recognition of individual contribution
Responsiveness to comments/suggestions
Time burden
Legal requirements
Legal restrictions
Data quality
Completeness
• Proportion of
blank / unknown
responses
Validity
• True data?
• Simple counting
• Comparison
 Records inspection
 Patient interviews
 ...
Completeness of information
AIDS cases
Information
HIV cases without AIDS
Total
Records with
Total
Records with
records
item filled in
records
item filled in
No.
No. (%)
No.
No. (%)
Name
703
703 (100)
na
Birth date
703
703 (100)
na
Birth month and year
703
703 (100)
1491
1489 (100)
Sex
703
703 (100)
1491
1491 (100)
Municipality of residence at HIV-diagnosis
703
703 (100)
1491
1479 (99)
Country of birth
703
703 (100)
1491
1489 (100)
Reason for stay in Norway
109
100 (92)
592
551 (93)
Length of stay in Norway at HIV-diagnosis
109
62 (57)
592
352 (59)
703
334 (48)
1491
998 (67)
196
171 (87)
665
606 (91)
Person
If not Norway
Place
Infection acquired in Norway or abroad
Cases acquired abroad
Country where infection was acquired
Sensitivity
= reported true cases
total true cases
= proportion
of true cases
detected
Report
Pos. specimen
Clinical specimen
Seek medical attention
Symptoms
Infected
Exposed
Sensitivity
Disease
+
--
+
True +
False +
Total
notified
-
False -
True -
Total not
notified
Notified
Total sick Total not sick
Sensitivity = True + / Total sick
Specificity = True - / Total Not sick
PVP = True+ / Total notified
Sensitivity versus specificity
The tiered system: confirmed, probable, possible
Consequences of low PvP

Frequent "false-positive" reports
 Inappropriate follow-up of non-cases
 Incorrect identification of epidemics


Wastage of resources
Inappropriate public concern (credibility)
Measuring sensitivity
• Find total true cases from other data
sources
 medical records
 disease registers
 special studies
• Capture-recapture study
Capture-recapture
• Used for counting total number of
individuals in population
using two or more incomplete lists
• Originally used in wildlife counting
(birds, polar bears, wild salmon…)
Uses in epidemiology
• Estimate prevalence or incidence
from incomplete sources
• Evaluate completeness
of a surveillance system
Principles
• Two/more sources of cases with disease
 Lists, registries, observations, samples
• Estimate total number in the source
population (captured and uncaptured)
from the numbers of captured
in each capture
Assumptions
1. The population is closed
 No change during the investigation
2. Individuals captured on both occasions
can be matched
 No loss of tags
3. For each sample, each individual has the
same chance of being included
 Same catchability
4. Capture in the second sample is
independent of capture in the first
 The two samples are independent, pYZ = pY pZ
Daddy, how
many fish are in
the aquarium?
Seaworld Oberhausen, August 2010
Your options as a scientist
• Don‘t answer
=> Expect repeat question
• Answer something
=> „How do you know?“
• Consult an expert
• Estimate yourself
Meet the expert - „Pulpo Paul“
•
•
•
Has nine brains and three hearts
Managed to predict all German
games during the 2010 Football
World Cup right
Predicted accurately the finale
Netherlands-Spain
Binomial
distributions
only
http://www.elpais.com/articulo/gente/tv/Muere/pulpo/Paul/elpepugen/20101026elpepuage_4/Tes
N=?
Two-source model
Y1
Source Y
Z1
Source Z
b
a
c
x=?
N= a + b + c + x
Two-source analysis
Source Y
Yes
No
Yes
a
b
No
c
x
Z1 = a + b
Source Z
Y1 = a + c
N=a+b+c+x
N = Y1 Z1 / a
Sensitivity of Y
Sensitivity of Z
Ysn = Y/N = (a+c)/N
Zsn = Z/N = (a+b)/N
How many persons are in the EPIET 2011 Introductory Course?
Isla del Lazareto, Dinner on Monday, 10 October 2011 – Case definiton: „Countable heads“
Hand does not
meet our case
definition
This is our first view
1
4
4
5
2
3
3
4
4
3
How many persons are in the EPIET 2011 Introductory Course?
Isla del Lazareto, Dinner on Monday, 10 October 2011 – Case definiton: „Countable heads“, n=33
This is our second view
4
3
2
6
3
How many persons are in the EPIET 2011 Introductory Course?
Isla del Lazareto, After Dinner Tutorial on Monday, 11 October 2011 – Case definition: “Countable heads“, N=18
How many participants at the course?
• Capture: Source ”View #1”
• Recapture: Source ”View #2”
• Estimations 
• Assumptions hold? 
Number of participants
Source View #2 – After Dinner Tutorial
Source
View #1
Dinner
Yes
No
13
20
Yes
No
5
View #1 = 33
x
View # 2 = 18
N = 13 + 20 + 5 + x
N = 33 * 18 / 13 = 47
Sensitivity of View # 1
Sensitivity of View # 2
Sn1 = 33/47 = 70.2%
Sn2 = 18/47 = 38.3%
+2
3
This is our second view
(revisited)
4
2
6
3
How many persons are in the EPIET 2011 Introductory Course?
Isla del Lazareto, After Dinner Tutorial on Monday, 11 October 2011 – Case definition: “Countable heads“, N=20
Number of participants
Source View #2, revised – After Dinner Tutorial
Source
View #1
Dinner
Yes
No
13
20
Yes
No
7
View #1 = 33
x
View # 2 = 20
N = 13 + 20 + 7 + x
Sensitivity of View # 1
Sensitivity of View # 2
N = 33 * 20 / 13 = 51
Sn1 = 33/51 = 64.7%
Sn2 = 20/51 = 39.2%
So, just how many are there?
9
9
9
25
18
30
2
5 off screen
Isla del Lazareto, Katharina‘s Lecture, Monday, 11 October 2010 – Case definition: “Persons in room“, N=53
The problem with the X:
Finding a comprehensive view
Assumptions may not hold
1. The population is closed
- Usually possible
2. Individuals captured on both occasions can be
matched
- OK if good recording systems
3. For each sample, each individual has the
same chance of being included
- Rarely true
4. Capture in the second sample is independent
of capture in the first
- Rarely true
Sources are independent
(most important condition)
Being in one source does not influence the
probability of being in the other source
Source Y
Yes
No
Yes
a
b
No
c
d
Z1
Source Z
Y1
ad
OR 
bc
N
OR > 1 (positive dependence):  underestimates N
OR < 1 (negative dependence):  overestimates N
Dependent sources
• Estimation of number of IVDU in Bangkok in
1991 (Maestro 1994)
• Two sources used:
 Methadone programme (April – May 1991)
 Police arrests (June – September 1991)
• Methadone  Need for drugs   Probability
of being arrested 
= negative dependence, overestimation of N
Usefulness of capture-recapture
• If conditions are met
 Great potential to estimate population size by
using incomplete sources
 Cheaper than exhaustive registers or full counting
• Two sources
 Impossible to quantify extent of dependence
• Multiple sources
 Can adjust for dependence and variable
catchability
Examples of capture-recapture
• STDs in The NL
 Reintjes et al. Epidemiol Infect 1999
• Foodborne outbreaks in France
 Gallay et al. Am J Epidemiol 2000
• Pertussis in England
 Crowcroft et al. Arch Dis Child 2002
• Invasive meningococcal disease
 Schrauder et al. Epidemiol Infect 2006
Representativeness

A representative system accurately describes
 Occurrence of a health event over time
 Distribution in the population by place and time

Difficult to determine





Compare reported events with actual events
Characteristics of the population
Natural history of condition, medical practices
Multiple data sources
Related to data quality, bias of data collection,
completeness of reporting
Timeliness
Disease
onset
Disease
diagnosed
Clinician, labs
Reporting
of event
Analysis and
interpretation
Public Health Authorities
Action
taken
Buehler’s balance of attributes
Sensitivity
Representativeness
Predictive value positive
Timeliness
Acceptability
Flexibility
Simplicity
Cost
Improving surveillance systems
• Recommendations of evaluation
 Continue
 Revise
 Stop

If revising





Increase participation rate of sources
Simplify notification
Increase the frequency of feedback
Broaden the net . . .
Activate data collection
Corollary
Carnunthum, Austria
Surveillance is like archeology of the immediate past –
It requires your responsible imagination of an invisible reality.
Thank you!
Literature
• CDC. Updated guidelines for evaluating public health
surveillance systems. MMWR 2001; 50 (RR-13): 1-35
• WHO. Protocol for the evaluation of epidemiological
surveillance systems. WHO/EMC/DIS/97.2.
• Romaguera RA, German RR, Klaucke DN.
Evaluating public health surveillance. In: Teutsch SM,
Churchill RE, eds. Principles and practice of public
health surveillance, 2nd ed. New York: Oxford
University Press, 2000.
Reading on capture-recapture
• Wittes JT, Colton T and Sidel VW. Capture-recapture
models for assessing the completeness of case
ascertainment using multiple information sources. J
Chronic Dis 1974;27:25-36.
• Hook EB, Regal RR. Capture-recapture methods in
epidemiology. Methods and limitations. Epidemiol
Rev 1995; 17: 243-264
• International Working Group for Disease Monitoring
and Forecasting. Capture-recapture and multiplerecord systems estimation I: History and theoretical
development. Am J Epidemiol 1995;142:1047-58
• International Working Group for Disease Monitoring
and Forecasting. Capture-recapture and multiplerecord systems estimation II: Applications in human
diseases. Am J Epidemiol 1995;142:1059-68