Transcript No Slide Title

Quantitative Methods: Reliability,
Validity, and Use of SEM to Assess
Psychometric Equivalence
Ron D. Hays, Ph.D.
UCLA Department of Medicine
Los Angeles, CA
November 18, 2005
1
Measurement Range for
Health Outcome Measures
Nominal Ordinal Interval Ratio
2
Indicators of Acceptability



Response rate
Administration time
Missing data (item, scale)
3
Variability


All scale levels are represented
Distribution approximates
bell-shaped “normal”
4
Measurement Error
Observed = true + systematic + random
score
error
error
(bias)
5
Flavors of Reliability



Test-retest (administrators)
Intra-rater (raters)
Internal consistency (items)
6
Intraclass Correlation and Reliability
Model
Reliability
One-Way
MS BMS  MSW MS
MS BMS
MS BMS  MSW MS
MS BMS  ( K  1) MSW MS
Two-Way
Fixed
MS BMS  MS EMS
MS BMS
MS BMS  MS EMS
MS EMS  ( K  1) MS EMS
Two-Way
Random
N ( MS BMS  MS EMS )
NMS BMS  MS JMS  MS EMS
Intraclass Correlation
MS BMS  MS EMS
MS BMS  ( K  1) MS EMS  K ( MS JMS  MS EMS ) / N
7
Cronbach’s Alpha
Source
df
SS
MS
Respondents
(BMS)
4
11.6
2.9
Items
(JMS)
1
0.1
0.1
Resp. x Items
(EMS)
4
4.4
1.1
Total
9
16.1
Alpha = 2.9 - 1.1 = 1.8 = 0.62
2.9
2.9
8
Reliability Minimum Standards

0.70 or above (for group comparisons)

0.90 or higher (for individual assessment)
» SEM = SD (1- reliability)1/2
9
Construct Validity
Does measure relate to other
measures in ways consistent with
hypothesis?


Responsiveness to change
10
Responsiveness to Change and
Minimally Important Difference
 HRQOL measures should be responsive to
interventions that changes HRQOL
Evaluating responsiveness requires
assessment of HRQOL

» pre-post intervention of known efficacy
» at two times in tandem with anchor
» HRQOL change among people who changed on anchor
11
Self-Report Anchor
Overall
has there been any change in your
asthma since the beginning of the study?
Much improved; Moderately improved;
Minimally improved
No change
Much worse; Moderately worse;
Minimally worse
12
Clinical Anchor
“changed”
group = seizure free (100%
reduction in seizure frequency)
“unchanged” group = < 50% change in
seizure frequency
13
Responsiveness Indices
(1) Effect size (ES) = D/SD
(2) Standardized Response Mean (SRM) = D/SD†
(3) Guyatt responsiveness statistic (RS) = D/SD‡
D = raw score change in “changed” group;
SD = baseline SD;
†SD = SD of D;
‡ SD = SD of D among “unchanged”
14
Effect Size Benchmarks

Small: 0.20->0.49

Moderate: 0.50->0.79

Large: 0.80 or above
15
Hypothetical Multitrait/Multi-Item
Correlation Matrix
Item #1
Item #2
Item #3
Item #4
Item #5
Item #6
Item #7
Item #8
Item #9
Trait #1
Trait #2
0.80*
0.80*
0.80*
0.20
0.20
0.20
0.20
0.20
0.20
0.20
0.20
0.20
0.80*
0.80*
0.80*
0.20
0.20
0.20
Trait #3
0.20
0.20
0.20
0.20
0.20
0.20
0.80*
0.80*
0.80*
*Item-scale correlation, corrected for overlap.
16
Confirmatory Factor Analysis
Compares observed covariances with
covariances generated by hypothesized
model
 Statistical and practical tests of fit
 Factor loadings
 Correlations between factors
 Regression coefficients

17
Fit Indices
• Normed fit index:

2
null
-
2
model
null
2

2
2
null
df null
• Non-normed fit index:
model
-
df model
 null
2
df null
• Comparative fit index:
1-

- 1
2
model
- df
model
null - dfnull
2
18
19
20
Acknowledgments
Supported
in whole by UCLA Center for
Health Improvement in Minority
Elders/Resource Centers for Minority
Aging Research, National Institute on
Aging (AG-02-004)
21