Transcript United Nations Economic Commission for Europe Statistical Division Evaluation Angela Me, Chief Social and Demographic Statistics Section.

United Nations Economic Commission for Europe
Statistical Division
Evaluation
Angela Me, Chief Social and
Demographic Statistics Section
Why do evaluate the census?




As all other statistics, data collected through
the census include errors
To provide users with measures of the quality
of the census data to help them to interpret the
results
To identify as far as possible the type and
source of errors to help assisting future
censuses
To provide information on the magnitude of
errors that can be used to adjust the results
and construct the best estimate of census
aggregates
- UNECE Statistical Division
Baku, 30 October-3 November 2006
Slide 2
Census Evaluation

A comprehensive evaluation should
include an assessment of census
operations in each of its phases (to
improve the next census)

Evaluation should address census
process and census results
- UNECE Statistical Division
Baku, 30 October-3 November 2006
Slide 3
Common Sources of Errors
1) Coverage
a. Omissions
b. Duplications
2) Content
a. Non-responses
b. Influenced responses
 Interview effect
 Respondent effect
- UNECE Statistical Division
Baku, 30 October-3 November 2006
Slide 4
Omissions
Very mobile populations or people who spend a
very limited amount of time in their home may not
be counted during the census operations
Some persons may refuse to participate in the
census
Incomplete mapping and delineation of
enumeration areas
Difficult-to-reach populations
- UNECE Statistical Division
Baku, 30 October-3 November 2006
Slide 5
Omissions reported by ECE
countries in the 2000 census round
•Males 15-24, young persons (20-30 years)
•Illegal immigrants
•Homeless
•One-person households,
•People who were temporarily absent
•Very young children
•Some ethnic minorities
•Students
•Multi-occupied addresses
- UNECE Statistical Division
Baku, 30 October-3 November 2006
Slide 6
Duplications
People who commute between two different
households
People on long term staying in institutions
Members of defence forces on long-term positioning
away from their family
Persons who dies before census reference date
- UNECE Statistical Division
Baku, 30 October-3 November 2006
Slide 7
Duplications reported by ECE
countries in the 2000 census round
People who maintain more than one residence
Migrants
Persons in hospitals
Students
Some ethnic groups
- UNECE Statistical Division
Baku, 30 October-3 November 2006
Slide 8
Non-Response
If a specific group of people do not answer to a
specific question, the aggregated result is biased
Non-Response may be classified into three types:
a)
b)
c)
Those unable to respond
Absentees
Refusals
- UNECE Statistical Division
Baku, 30 October-3 November 2006
Slide 9
Language Difficulties
Some potential respondents may be illiterate or
cannot understand the language used in the
questionnaires
Unless such problems are dealt with, by
translating questionnaires and/or engaging
multi-language interviewers , bias could arise
because those unable to answer may be
'special' in other ways
A different linguistic group might, for instance,
have completely different lifestyle and customs
- UNECE Statistical Division
Baku, 30 October-3 November 2006
Slide 10
Absentees
If a person is absent because she or he has
moved away for a short period of time (less than
12 months), specific information may not be
collected
- UNECE Statistical Division
Baku, 30 October-3 November 2006
Slide 11
Refusals
Some persons may refuse to respond to some
“sensitive questions”
- UNECE Statistical Division
Baku, 30 October-3 November 2006
Slide 12
Influenced Response
Respondents will sometimes tend to over- or underestimate due to perceived advantage
Example 1: farmers might inflate their land holdings, by always
rounding figures upwards, because they believe that the survey
results will be used to allocate state aid
Example 2: the farmers might deflate, by rounding down,
in the hope of minimize taxation
- UNECE Statistical Division
Baku, 30 October-3 November 2006
Slide 13
Leading Questions
Sometimes response bias is caused through leading
questions such as, 'Do you agree that meat eating is
barbaric?'
Most people like to please and/or will take the easy
option of agreeing in the hope of avoiding further
questions!
- UNECE Statistical Division
Baku, 30 October-3 November 2006
Slide 14
US Survey Example
The following questions and results were obtained in an American survey
% 'Yes'
Have you ever heard the word AFROHELIA?
(no such word!)
Have you ever heard of the famous writer, John Woodson?
(no such writer!)
8
16
Have you ever heard of the Midwestern Life Magazine?
(no such magazine!)
25
Do you recall that, as a good citizen you voted last December in the
special election for your state representative?
(no election!)
Have you ever heard of the Taft-Pepper Bill concerning veteran's housing
(no such bill!)
33
53
Sometimes this type of bias is called prestige error
- UNECE Statistical Division
Baku, 30 October-3 November 2006
Slide 15
Interviewer effect
On occasions the very appearance of the
enumerator can cause bias
For instance, in certain cases, men may give quite
different answers when asked questions by
another man than they would if a female
interviewer were used
Similarly, if the setting of the interviewer is
inappropriate we may obtain a biased response
- UNECE Statistical Division
Baku, 30 October-3 November 2006
Slide 16
Example: Australian Survey
Average number of sex partners reported
•
By women who were watched as they filled in their
survey answers: 2.6;
•
By women who knew they were completely
anonymous: 3.4;
•
By women who thought they were attached to a lie
detector: 4.4
Sydney Morning Herald, August 31, 2003
- UNECE Statistical Division
Baku, 30 October-3 November 2006
Slide 17
Memory Lapse
Respondents may not be able to recall events
in the past
- in part
- in total
For example, mothers may not accurately recall
the number of their children who died aged less
than one month old, over the past 5 years
- UNECE Statistical Division
Baku, 30 October-3 November 2006
Slide 18
Root Mean Square Error
In general, for all statistics total error, sampling and bias combined,
is measured by the root mean square error, (RMSE)
This is defined as
RMSE = (Sampling error) 2  (Bias) 2
- UNECE Statistical Division
Baku, 30 October-3 November 2006
Slide 19
Schematic Representation
This is best thought of as the hypotenuse of a right-angled triangle
RMSE
Bias
Sampling error
- UNECE Statistical Division
Baku, 30 October-3 November 2006
Slide 20
From small samples to the census
Notice that, although a census (100% sample) has no sampling
error, the bias may be so large that the RMSE (equal to the bias
error in this case) is actually greater than for a sample survey
of a moderate size
small sample
big sample
- UNECE Statistical Division
Baku, 30 October-3 November 2006
census
Slide 21
How to evaluate census
coverage and content?
• Internal consistency checks
• Comparison of results with other data
•
•
•
•
sources
Post-enumeration survey: independent
operation
Re-interview surveys
Demographic analysis
Benchmarking
- UNECE Statistical Division
Baku, 30 October-3 November 2006
Slide 22
Comparison of results with
other data sources
Other data sources:
previous census
other surveys
administrative records
Comparison:
comparison of overall estimates
record checking (limited to special population
groups?)
- UNECE Statistical Division
Baku, 30 October-3 November 2006
Slide 23
Demographic analysis
• Derivation of an expected population
estimate taking account of vital registers of
births, deaths and net migrants between
censuses, as compared with the latest
census
• Population projections based on the results of
the previous census plus data on fertility,
mortality and migration from different sources
and comparing the projected estimates with
the new census results
- UNECE Statistical Division
Baku, 30 October-3 November 2006
Slide 24
Demographic analysis
• Comparison of two census age distribution based on
intercensal cohort survival rates
• Estimates of coverage correction factors using
regression methods to make the age results from the
two censuses mutually consistent
Limit the evaluation studies at national level
- UNECE Statistical Division
Baku, 30 October-3 November 2006
Slide 25
Benchmarking
• To track the contribution of non-responses,
editing and imputation to the final data quality
• Compare
• Non-response rates between censuses
• Responses before and after processing
It assesses quality
It can be produced based on as sample
of records
- UNECE Statistical Division
Baku, 30 October-3 November 2006
Slide 26
Benchmarking
• To track the contribution of non-responses,
editing and imputation to the final data quality
• Compare
• Non-response rates between censuses
• Responses before and after processing
It assesses quality
It can be produced based on as sample
of records
- UNECE Statistical Division
Baku, 30 October-3 November 2006
Slide 27
Methods used for census evaluation
in the 2000 census round in the ECE
region
Number of evaluation Number of
methods
countries
Only one method
Two methods
9
13
Three methods
Four methods
6
6
Five methods
No evaluation
3
7
Total
44
- UNECE Statistical Division
Baku, 30 October-3 November 2006
Slide 28
Methods used for census evaluation
in the 2000 census round in the ECE
region
Evaluation methods
Number of
countries
Quality PES
12
Coverage PES
20
Demographic Analysis
23
Field re-interviews
14
Comparison with other source
23
Other method
2
No evaluation
7
- UNECE Statistical Division
Baku, 30 October-3 November 2006
Slide 29
Methods used for census evaluation
in the 2000 census round in the ECE
region
None of the countries in
Central Asia reported
evaluation activities
- UNECE Statistical Division
Baku, 30 October-3 November 2006
Slide 30
Indicators to measure
coverage
• Percentage of omissions (in ECE ranging between
0% to 3.95%)
• Percentage of duplications (in ECE ranging between
0% to 0.96%)
• Gross coverage errors = percentage of omissions +
duplications + people erroneously counted
• Net coverage errors = percentage of omissions –
duplications (in ECE ranging between -3.1% to o.7%)
- UNECE Statistical Division
Baku, 30 October-3 November 2006
Slide 31
Communicating quality
• NSO should take a proactive role in
communicating the reliability of the census
data
• Users should be fully aware of the limitation
and strengths of the final census data
- UNECE Statistical Division
Baku, 30 October-3 November 2006
Slide 32