Transcript Validity and reliability of scales

Lecture 6: Reliability and
validity of scales (cont)
1. In relation to scales, define the following terms:
- Content validity
- Criterion validity (concurrent and predictive)
- Construct validity (discriminant and convergent)
- Responsiveness
2. Use of proxy respondents
Validity
• Depends on purpose:
– screening: discrimination
– outcome of treatment: responsiveness,
sensitivity to change
– prognosis: predictive validity
Content and face validity
• Judgment of “experts” and/or members of
target population
• Does measure adequately sample domain
being measured?
• Does it appear to measure what it is
intended to measure? (eyeball test)
Criterion validity
• Criterion (“gold” standard)
• Concurrent criterion validity
– e.g., screening test vs diagnostic test
• Predictive criterion validity
– e.g., cancer staging test vs 5-year survival
Construct validity
• Is the theoretical construct underlying the
measure valid?
• Development and testing of hypotheses
• Requires multiple data sources and
investigations:
– Convergent validity: measure is correlated with
other measures of similar constructs
– discriminant validity: measure is not correlated
with measures of different constructs
Responsiveness of measures
• Ability to detect clinically important change
over time or differences between treatments
• Requirement of evaluative measures
• Two approaches:
– external responsiveness (validation against
change in external criterion)
– internal responsiveness: compute effect size of
ratio of change score to measure of variability
(different formulae)
Validity of Delirium Index (DI)
• Convergent validity
– DI will be correlated with measures of current
health/function:
• Current ADL disability (Barthel Index)
• Current severity of illness (clinical judgment and
physiological severity score from APACHE)
Validity of Delirium Index (DI)
• Discriminant validity:
– DI will be weakly correlated with measures of
previous health/function:
• premorbid level of ADL disability
• severity of dementia
• comorbidity (prior)
Validity of DI (cont)
• DI score will predict probability of one-year
survival
• DI score will be sensitive to:
– changes in medication exposures
– changes in environmental factors
Spearman correlation coefficients between Delirium
Index and 3 baseline measures of current status
0.5
Delirium+Dementia (n=165)
0.4
Delirium (n=57)
0.3
0.2
0.1
0
Barthel Index
Clinical
severity
Physiological
severity
Spearman correlation coefficients between Delirium
Index and 3 baseline measures of prior status
0.6
Delirium+dementia (n=165)
0.5
Delirium (n=57)
0.4
0.3
0.2
0.1
0
IADL
IQCODE
Comorbidity
Delirium severity and survival
• Proportional hazards regression of delirium
severity in delirium cohort
• Mean of 1st 2 DI scores
• Results
– significant interaction: DI predicted survival in
patients with delirium alone, not in those with
dementia
Effects of medications and
environmental factors
• Repeated in-hospital measures of DI,
medications (medical record), and
environmental factors (using checklist)
• Repeated measurements analysis performed
to investigate associations of changes in risk
factors with change in DI score.
Environmental risk factors
for delirium
ICU vs medicine
Physical restraints
No reading glasses
Family member
present
No clock/watch
Medical restraint
Cumulative room
changes (one)
0
1
2
3
mean DI increase
Source: McCusker et al, JAGS 2001, 49:1327-34
4
5
Medications and DI
• Medication changes (increase in #
medications, use of anticholinergic
medications) were associated with increase
in DI score.
Responsiveness of DI
• Internal responsiveness: ability to capture
real change over time
– effect size (change in mean DI/SD at baseline)
– standardized response mean (change in mean
DI/SD of change)
• External responsiveness: are changes in the
DI correlated with other measures of
change?
Internal responsiveness of the Delirium
Index (DI) at 8 weeks
Delirium+dementia (n=94)
0.8
Delirium (n=33)
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Effect size
Standardized response mean
Validity of DI: Conclusions
• In patients with delirium +/- dementia:
– Correlated with measures of current function
– Responsive to change over time
– Responsive to changes in environmental factors
and medications
• In patients with delirium alone:
– Predicts survival
Example: Inflammatory Bowel
Disease Questionnaire (IBDQ)
• Disease-specific HRQoL measure
• 30 items covering 4 domains
–
–
–
–
bowel symptoms
systemic symptoms
emotional function
social function
• How to validate?
Validation of Inflammatory
Bowel Disease Questionnaire
• Questionnaires administered to 42 patients
with inflammatory bowel disease on 2
occasions, 1 month apart:
– IBDQ
– Global ratings of function
– Global ratings of change by physician and a
relative
– Disease Activity Index
– Emotional function scale from generic HRQoL
instrument
Validation of IBDQ: predictions
and results
• Correlation of 0.5 or more between patients’s global rating
of change and IBDQ subscale on bowel symptoms (result:
0.42)
• Correlation of 0.3 or more between change in Disease
Activity Index and IBDQ subscale on bowel symptoms
(result: 0.33)
• Correlation of 0.3 or more between change in Disease
Activity Index and IBDQ subscale on systemic symptoms
(result: 0.04)
• Correlation of 0.5 or more between change in generic
measure of emotional function and IBDQ subscale on
emotional function (result: 0.76)