Transcript Evaluation and assessment in civil registration and vital statistics systems Discussion, United Nations Statistics Division United Nations Expert Group Meeting on International Standards for.

Evaluation and assessment in civil
registration and vital statistics
systems
Discussion, United Nations Statistics Division
United Nations Expert Group Meeting on International Standards for Civil Registration and
Vital Statistics Systems, 27-30 June 2011, New York
Australia – Evaluating the quality of vital statistics: paper for
discussion
• Overarching assessment/evaluation framework - WHO
and University of Queensland Health Information Systems
Knowledge Hub
• Several components covering inputs, processes and outputs
• An assessment tool for independent evaluation studies
• Legal framework/organizational
structure/coverage/accuracy/quality check/data
dissemination
• The tool itself covers a sub-component that asks whether
quality check is being conducted regularly (E.1)
United Nations Expert Group Meeting on International Standards for Civil Registration and
Vital Statistics Systems, 27-30 June 2011, New York
Australia – Evaluating the quality of vital statistics: paper for
discussion
• Comments on Chapter IV of the Principles and
Recommendations for a Vital Statistics System,
Rev. 2
• developing an assessment framework
• Technical and dense, may provide a set of guiding
questions (see above framework)
• Consider the following additional criteria when
evaluating vital statistics
• Relevance
• Ensuring international comparability
• Ensuring confidentiality of individuals
United Nations Expert Group Meeting on International Standards for Civil Registration and
Vital Statistics Systems, 27-30 June 2011, New York
The vital statistics system in China
• Three main sources of population statistics
• Population and housing censuses (every 10 years)
• Household sample surveys:
• 1% population survey (inter-censal)
• 1‰ population change survey (every year)
• Administrative records:
• Population by permanent residents (Ministry of Public
Security)
• Registered temporary residents (Ministry of Public Security)
• Marriages and divorces (Ministry of Civil Affairs)
• Health statistics (Ministry of Health)
United Nations Expert Group Meeting on International Standards for Civil Registration and
Vital Statistics Systems, 27-30 June 2011, New York
The vital statistics system in China
• Data checking:
• Resident population and migrants’ information from
the household registration administered by the
Ministry of Public Security; birth information collected
by family planning departments, the health
department as well as the community and village
committees – used to check and compare information
collected in the 2010 censuses.
United Nations Expert Group Meeting on International Standards for Civil Registration and
Vital Statistics Systems, 27-30 June 2011, New York
Reflections/some thoughts on the
revision of the P&R, rev 2.
The chapter should answer the following questions:
• Why is quality assurance and assessment important?
• Against what standards are we evaluating?
• What is quality assurance and assessment?
• How should it be done?
– What methods?
– How often?
– Who should be doing it?
United Nations Expert Group Meeting on International Standards for Civil Registration and
Vital Statistics Systems, 27-30 June 2011, New York
Measuring against the following standards
Principles and Recommendations for a Vital Statistics
System, Rev. 2 (para. 547)
• Completeness
• Correctness/accuracy
• Availability
• Timeliness
Additional ones to consider?
• Relevance
• International/national comparability
• data confidentiality
United Nations Expert Group Meeting on International Standards for Civil Registration and
Vital Statistics Systems, 27-30 June 2011, New York
Quality assurance – what is it/how is it done?
• Occurs at each operation stage within the civil
registration and vital statistics systems, for example
– Registration stage: making sure that all events are
registered without duplication; information verified; all
required information provided
– Data collection
– Data transfer: Monitoring the submitted statistics report:
timeliness, completeness (from all registration areas), and
monitoring any irregularities
– Data compilation, editing stage
– Data dissemination
• Making sure that all the checks are in place and are
integrated part of the two systems (Australia paper)
United Nations Expert Group Meeting on International Standards for Civil Registration and
Vital Statistics Systems, 27-30 June 2011, New York
Quality assurance – how often? Who does it?
• How often?
– Constant
• Who does it?
– Civil registrars at all levels
– Statisticians compiling, processing and
disseminating vital statistics
United Nations Expert Group Meeting on International Standards for Civil Registration and
Vital Statistics Systems, 27-30 June 2011, New York
2.0 Local Registration
LIVE BIRTH
A
1
Example: United States
Hospital
Completes Birth
Record
B
2
N
System
Automatically
Performs
NCHS Validations
Correct Errors and
Resubmit for
Validation
N
Passes all
NCHS edits?
Submit Record to
VR Jurisdiction
Via Web
Hold for
X Days
Record Received
From Local
Registrar?
Y
Match Records,
Manually Stamp,
Log, Key Local File
Number
C
1
Register Record
Y
Print and Send
Paper Record to
Local Registrar
Local Registrar
Manually Logs,
Reviews and Files
Record
A
2
Mail or Deliver
Record to
VR Jurisdiction
VR Database
END
B
1
C
2
Generate Routine
Submissions to
NCHS
Timeliness Metrics:
A.
Number of days from date of birth to when paper record is received from local registrar (A1 to A2).
B.
Number of days from when record is received from local registrar to when record is registered (B1 to B2).
C.
Number of days from when record is registered to when it is submitted to NCHS (C1 to C2).
END
United Nations Expert Group Meeting on International Standards for Civil Registration and
Vital Statistics Systems, 27-30 June 2011, New York
Process flow: deaths and stillbirths – South Africa
Collection
Sorting
Pasting
Data Capture
(ID variables)
STORES
Export to
Editing
Data Capture
& QA
Coding
Pre-coding
Data
Editing
Derivation
0f UCD
Data
Analysis
Publication &
Dissemination
United Nations Expert Group Meeting on International Standards for Civil Registration and
Vital Statistics Systems, 27-30 June 2011, New York
Quality assessment studies – what are they?
•
Answers specific questions and with specific objectives
– What is the coverage of birth registration?
– Who are the un-registered? What are their characteristics?
– Is there difference in the quality of cause of death data from
two provinces? What causes the difference and how to improve?
– Is my new cause of death coder competent?
– Is the information collected on birth records for District A
accurate?
•
•
Some studies carried out regularly, some ad-hoc basis
Best if carried out by an independent group
United Nations Expert Group Meeting on International Standards for Civil Registration and
Vital Statistics Systems, 27-30 June 2011, New York
Quality assessment studies – how?
•
Direct methods
–
–
–
–
–
–
–
matching birth and death records: identify un-registered records; an
assessment of both registers
Matching civil registration records with other administrative records (e.g.
social security, school records): such check may be automated and ran
constantly if a comprehensive population register is in place
Match birth/death records with records from census and surveys: identify unregistered recent births/deaths; also check accuracy of other information
Compare cause of death information with autopsy reports, hospital records.
Assess coder’s qualification: ask 2 coders to work on the same set of statistics
report and compare
Select a sample of birth records and send a questionnaire to parents asking
similar information as on the birth reporting form
Include a question in census asking whether recent births and deaths are
registered
United Nations Expert Group Meeting on International Standards for Civil Registration and
Vital Statistics Systems, 27-30 June 2011, New York
Quality assessment studies – how? (cont.)
Indirect methods:
• Monitoring births trends over a number of years;
• Monthly figure monitoring
• Intervals between date of occurrence and registration –
timeliness
• Work with census data:
– Population balance equation: P2 – P1 = B – D + net
in-Migration
– Comparing births/deaths in the last 12 months
– Comparing rates for similar periods
United Nations Expert Group Meeting on International Standards for Civil Registration and
Vital Statistics Systems, 27-30 June 2011, New York
Quality assessment studies – how? (cont.)
• Overall evaluation studies: touching on all aspects of the
two systems but not in-depth
– WHO/UQ HIS-Hub framework
– Eurostat
– UNSD assessment studies for countries through
regional workshops and questionnaires
• Often carried out on ad-hoc basis
• Could be self-assessment
United Nations Expert Group Meeting on International Standards for Civil Registration and
Vital Statistics Systems, 27-30 June 2011, New York
Points for discussion
• What kind of information we would like this
chapter to cover?
• Are the evaluation methods (direct/indirect)
up-to-date?
• Any new methods emerged in the past 10
years?
United Nations Expert Group Meeting on International Standards for Civil Registration and
Vital Statistics Systems, 27-30 June 2011, New York