Transcript FastTrack Study - University of California, Los Angeles

Evaluating Multi-item Scales
Ron D. Hays, Ph.D.
UCLA Division of General Internal Medicine/Health Services Research
[email protected]
http://twitter.com/RonDHays
http://gim.med.ucla.edu/FacultyPages/Hays/
HS225A 11/02/10, 3-4:50 pm, 61-262 CHS
Example Responses to 2-Item Scale
ID
Poor
Fair
Good
Very
Good
Excellent
01
2
02
03
1
1
04
05
1
1
2
1
1
Cronbach’s Alpha
01 55
02 45
03 42
04 35
05 22
Source
df
SS
MS
Respondents (BMS)
Items (JMS)
Resp. x Items (EMS)
4
1
4
11.6
0.1
4.4
2.9
0.1
1.1
9
16.1
Total
Alpha =
2.9 - 1.1 = 1.8 = 0.62
2.9
2.9
Computations
• Respondents SS
(102+92+62+82+42)/2 – 372/10 = 11.6
• Item SS
(182+192)/5 – 372/10 = 0.1
• Total SS
(52+ 52+42+52+42+22+32+52+22+22) – 372/10 = 16.1
• Res. x Item SS= Tot. SS – (Res. SS+Item SS)
Alpha for Different Numbers of Items and Average Correlation
Average Inter-item Correlation ( r )
Number
of Items (k)
2
4
6
8
.0
.2
.4
.6
.8
1.0
.000
.000
.000
.000
.333
.500
.600
.666
.572
.727
.800
.842
.750
.857
.900
.924
.889
.941
.960
.970
1.000
1.000
1.000
1.000
Alphast =
k* r
1 + (k -1) * r
Spearman-Brown Prophecy Formula
alpha
y
=
(
N • alpha
x
1 + (N - 1) * alpha
x
)
N = how much longer scale y is than scale x
Example Spearman-Brown Calculation
MHI-18
18/32 (0.98)
(1+(18/32 –1)*0.98
= 0.55125/0.57125 = 0.96
Reliability Minimum
Standards
• 0.70 or above (for group comparisons)
• 0.90 or higher (for individual
assessment)
 SEM = SD (1- reliability)1/2
Intraclass Correlation and Reliability
Model
Reliability
Intraclass Correlation
Oneway
MSBMS  MSWMS
MSBMS
MSBMS  MSWMS
MSBMS  (k  1) MSWMS
Twoway
fixed
MSBMS  MSEMS
MSBMS
MSBMS  MSEMS
MSBMS  (k  1) MSEMS
Twoway
random
N ( MSBMS  MSEMS )
NMSBMS  MS JMS  MSEMS
MSBMS
MSBMS  MSEMS
 (k  1) MSEMS  k ( MS JMS  MSEMS ) / N
BMS = Between Ratee Mean Square
WMS = Within Mean Square
JMS = Item or Rater Mean Square
EMS = Ratee x Item (Rater) Mean Square
9
Equivalence of Survey Data
• Missing data rates were significantly higher for
African Americans on all CAHPS items
• Internal consistency reliability did not differ
• Plan-level reliability estimates were significantly
lower for African Americans than whites
M. Fongwa et al. (2006). Comparison of data quality for
reports and ratings of ambulatory care by African
American and White Medicare managed care enrollees.
Journal of Aging and Health, 18, 707-721.
10
Item-scale correlation matrix
Item #1
Item #2
Item #3
Item #4
Item #5
Item #6
Item #7
Item #8
Item #9
Depress
Anxiety
0.80*
0.80*
0.80*
0.20
0.20
0.20
0.20
0.20
0.20
0.20
0.20
0.20
0.80*
0.80*
0.80*
0.20
0.20
0.20
Anger
0.20
0.20
0.20
0.20
0.20
0.20
0.80*
0.80*
0.80*
*Item-scale correlation, corrected for overlap.
11
Item-scale correlation matrix
Item #1
Item #2
Item #3
Item #4
Item #5
Item #6
Item #7
Item #8
Item #9
Depress
Anxiety
0.50*
0.50*
0.50*
0.50
0.50
0.50
0.50
0.50
0.50
0.50
0.50
0.50
0.50*
0.50*
0.50*
0.50
0.50
0.50
Anger
0.50
0.50
0.50
0.50
0.50
0.50
0.50*
0.50*
0.50*
*Item-scale correlation, corrected for overlap.
12
Confirmatory Factor Analysis
• Observed covariances comapred to
covariances generated by
hypothesized model
• Statistical and practical tests of fit
• Factor loadings
• Correlations between factors
Fit Indices
• Normed fit index:

2
-
null
2
model
null
2

2
2
null
-
df null
• Non-normed fit index:
model
df model
 null
2
df null
• Comparative fit index:
1-

- 1
2
model
- df
null - dfnull
2
model
19
20
Differential Item Functioning
(2-Parameter Model)
1
AA
Probability of "Yes" Response
0.9
0.8
0.7
0.6
White
White
0.5
0.4
Slope DIF
Location DIF
0.3
0.2
AA
0.1
0
-4
-3.5
-3
-2.5
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
2.5
3
3.5
4
Trait level
21
Location = uniform; Slope = non-uniform
Thank you.