Transcript Validation in Statistics Canada Health Surveys

Validation in Statistics Canada
Health Surveys
Presentation to RRFSS
Workshop
June 20, 2007
Vincent Dale
Outline





Statistics Canada quality assurance
framework
Ensuring data accuracy
Past validation projects
Future projects
Future directions
Quality Assurance Framework
Relevance
Degree to which information meets the real needs of data users
Timeliness
Delay between the reference period and the date upon which the
information becomes available
Accessibility
Ease with which information can be obtained from the Agency
Accuracy
Degree to which the information correctly describes the phenomena it was
designed to measure
Interpretability
Availability of supplementary information and metadata necessary to
interpret and utilize information appropriately
Coherence
Degree to which information can be successfully brought together with
other statistical information within a broad analytic framework and over
time
Statistics Canada
Quality Assurance Framework


Trade-offs between aspects of quality
These are actively managed through a
variety of processes, including:
–
–
–
–
User and stakeholder feedback mechanisms
Program review
Data analysis and dissemination
Standards and documentation (concepts,
variables, classifications)
Ensuring Data Accuracy

Questionnaire development
–
Wherever possible, validated questionnaire modules are
used



Questionnaire testing
–
STC policy requires testing of all new questionnaires


Sometimes modified for use in population-based survey
Sometimes not as valid as advertised
Cognitive interviews and focus groups
Coherence versus accuracy
–
Sometimes better to keep measure stable even if imperfect
Ensuring Data Accuracy

Sampling error
–

error attributed to studying a fraction of a population rather
than carrying out a census
Non-sampling error
–
–
–
–
–
–
coverage errors
response errors
non-response errors
processing errors
estimation errors
analysis errors
Ensuring Data Accuracy

Explosion of health survey data
–

More data, more often for smaller levels of
geography
Increasing attention paid to validity
–
–
–
Health measures
Administrative data
Complimentary surveys
What is validity?


Face validity
Internal validity
–

construct validity
External validity
–
–
Criterion
Sensitivity, specificity, predictive value
Past CCHS Validation Projects
Health Care Utilisation
–
–
Data linkage of CCHS responses with BC
administrative health records
Supplemented with analysis of:



Respondent interpretation and formulation of responses
Interviewer behaviour and training
Patterns in response changes, edits and timing of
response entry
Contacts with Health Professionals

Results of linkage:
–
Compared to provincial health records:




Most CCHS respondents (58%) reported fewer primary
care physician contacts
On average, CCHS respondents reported 1.7 fewer
primary care physician contacts
Older CCHS respondents and respondents with better
self-perceived health tended to report fewer contacts
Younger respondents and respondents with poorer selfperceived health tended to report more contacts
Contacts with Health Professionals

Recommendations from study:
–
–
–
Revise wording of specific questions to minimize
misinterpretation
Facilitate consistent interviewer probing
techniques
Improved edits and CAPI/CATI application
navigation for interviewers to facilitate changes to
previously-answered questions
Evaluation of coverage of linked CCHS
and hospital inpatient records

Probabilistic linkage used to identify CCHS
1.1 respondents (excluding Québec)
hospitalized over a 14-month period
–

Health person-oriented information database
(HPOI) is a virtual census of hospital admissions
and used as the standard
Survey weights applied to the 8230 CCHS
records which were found in the HPOI
database
Evaluation of coverage of linked CCHS
and hospital inpatient records
Number hospitalized in acute-care hospitals, Sept. 1, 2000 – Nov. 3, 2001,
aged 12+, Canada excluding Québec
CCHS
Unweighted
(n)
TOTAL
8,230
HPOI
Weighted
(N)
Count
(N)
1,334,909
1,612,269
Coverage
rate
82.8%
Evaluation of coverage of linked CCHS
and hospital inpatient records

Under-reporting rates similar between women and
men
–
–

Lower among Manitoba residents (69.2%)
Higher among individuals aged 12-74 (86.1%) than those
aged 75+ (70.3%)
Under-reporting is an essential prerequisite to further
analyses based on the CCHS – HPOI linked data
–
Use of the linked file could lead to bias due depending on
province/territory of residence and age
CCHS Measured Height & Weight

In 2005, height / weight were measured for a
sub sample of CCHS Cycle 3.1 participants
(n=4567)
–
–
–
Weight: mean difference between measured and
self-reported weight of 2.1 kg (2.5 kg for women)
Height: mean difference between measured and
self-reported height of -0.7 cm (-1.0 cm for men)
BMI: mean difference between measured and
self-reported BMI was 1.1
CCHS Measured Height & Weight
Underweight
(<18.5)
Total
%
('000)
Self-reported BMI category
Both sexes
Underweight (<18.5)
Normal weight (18.5 to 24.9)
Overweight (25.0 to 29.9)
Obese Class I (30.0 to 34.9)
Obese Class II, III (=35)
Total
271
131
0
0
0
402
67
33
0
0
0
100
Normal weight
(18.5 to 24.9)
Total
%
('000)
308
10,163
388
0
0
10,859
3
94
4
0
0
100
Measured BMI category
Overweight
Obese Class I
(25.0 to 29.9)
(30.0 to 34.9)
Total
%
Total
%
('000)
('000)
1
2,651
5,851
244
0
8,746
0
30
67
3
0
100
1
120
1,894
2,247
22
4,288
0
3
44
52
1
100
Obese Class II, III
(=35)
Total
%
('000)
0
4
134
603
822
1,562
0
0
9
39
53
100
CCHS Mode Effect Study

Potential differences associated with two methods of
collection used in CCHS
–
–

CAPI: computer assisted personal interview
CATI: computer assisted telephone interview
Used a split-panel design with a unique sample
frame
–
–
–
secondary sampling units randomly assigned to CAPI or
CATI.
Fully integrated as part of CCHS cycle 2.1
11 sites selected to provide a good representation of each
region in Canada
CCHS Mode Effect Study

Important differences observed for obesity rates
–

Physical activity index – inactive persons
–


CAPI = 17.9%; CATI = 13.2%
CAPI = 42.3%; CATI = 34.4%
Statistically significant differences for contact with
medical doctors and unmet health care needs
No significant differences observed in the vast
majority of health indicators
CCHS Mode Effect Study



Overall results show that cycles 1.1 and 2.1
are largely comparable despite an increase
in CATI collection for Cycle 2.1 (2003)
Results led to a decision to measure exact
height and weight for a sub-sample of
respondents in cycle 3.1 (2005)
Led to improved standardization of
interviewer procedures across the two
collection modes
Future Validation Projects
Scale Reliability - Factor Analysis

Construct validity / scale reliability:
–
–
Cronbach’s Alpha calculated for scales used in
CCHS questionnaire
Results could be published in user guide


What are standards?
Some researchers feel that scores should be above 0.8
CCHS Depression Module

Currently, CIDI Short form for Major Depression
(CIDI-SF) is used in CCHS
–

Some problems with its use in CCHS
–
–
–

Also used in NPHS and several regional and provincial
surveys
Has not been validated against International Classification of
Disease (ICD)
Evaluates 12-month prevalence, not necessarily current
treatment need
Does not evaluate some items related to clinical significance
Patient Health Questionnaire (PHQ) identified as
potential CIDI-SF replacement
CCHS Depression Module

Primary goals of potential validation study:
–
–
Determine the validity of the CIDI-SF and PHQ in
relation to a gold standard diagnostic interview
(SCAN – Schedules for Clinical Assessment in
Neuropsychiatry)
Identify optimal scoring procedures for the PHQ in
Canadian population-based studies
CCHS Depression Module

Samples of n=200 subjects to be drawn in two sites
(English and French)
–

Supplemented with n=100 subjects selected from psychiatric
outpatient settings to increase the number of positive cases
of major depression
Each participant to be administered:
–
–
–
–
–
1) Standard demographic module
2) PHQ-9
3) Module to distinguish between clinical depression and
bereavement
4) SIDI-SF
5) Set of modules to assess consequences of construct in
terms of quality of life
CCHS Depression Module





Sensitivity and specificity of the CIDI-SF and PHQ to
be measured using the SCAN as a gold standard
Ordinal CIDI-SF ratings to be correlated with PHQ
ordinal ratings using Spearman correlation coefficient
Test of construct validity of PHQ to be performed
using exploratory factor analysis
Internal consistency of scales and subscales to be
assessed using Cronbach’s alpha
Test-retest reliability of PHQ and CIDI-SF and interrater reliability of the SCAN will be evaluated for 50
respondents
CCHS Depression Module


The estimated cost for the project exceeded
$200,000
Due to our inability to secure external funding
and the lack of available budget and
personnel internally, there are no concrete
plans to proceed with study
Directions Forward


Focus on accuracy, interpretability and coherence
Trade-offs between aspects of data quality



Improved timeliness, accessibility and relevance
How good is “good enough”?
Partnerships
–
Are there areas where CCHS, RRFSS and others can
collaborate ?
Contact Information
Vincent Dale
Survey Manager, Canadian Community Health Survey
613-951-4265
Sylvain Tremblay
Content Manager, Canadian Community Health Survey
613-951-2528