Psychometric analysis of ADNI data

Transcript Psychometric analysis of ADNI data

Psychometric analyses of ADNI data

Paul K. Crane, MD MPH Department of Medicine University of Washington

Disclaimer

• Funding for this conference was made possible, in part by Grant R13 AG030995 from the National Institute on Aging. • The views expressed do not necessarily reflect official policies of the Department of Health and Human Services; nor does mention by trade names, commercial practices, or organizations imply endorsement by the U.S. Government.

Outline

• ADNI neuropsychological battery • Latent variable approaches – SEM and IRT • ADAS-Cog in ADNI • Memory in ADNI • Executive functioning in ADNI

ADNI Neuropsychological Battery

Handout

• There is a handout that summarizes these tests and provides the variable names for the variables in the dataset The

Boston Naming Test

presents the participant with 30 black and white line drawings; the participant is asked to identify the object in the drawing. Only the odd numbered drawings from the original 60 item version are administered (30 points max). Each drawing is accompanied by 1) semantic and 2) phonemic cues that are provided to the participant if they do not spontaneously produce the name. The item is scored correct if the participant either spontaneously produces the correct name or if they do so after the semantic clue. (Both scores are in the data set.) Words provided after the phonemic cue are provided are not counted as correct.

bntspont bntstim bntcstim bntphon bntcphon bnttotal

"Number of spontaneously given correct responses" "Number of semantic cues given" "Number of correct responses following a semantic cue" "Number of phonemic cues given" "Number of correct responses following a phonemic cue" "Total Number Correct (1+3)"

ADNI Neuropsychological Battery

MCI

Alternate Word Lists There are also two versions of the Rey AVLT that are alternated NL MCI AD BL A A A M6 B B B M12 A A A M18 -- B -- M24 A A A M30 -- -- -- M36 B B --

Summary

• Repeated administration of a rich neuropsychological battery at 6 month intervals for 2 (AD) or 3 (NC, MCI) years • How do we drink from that fire hose?

Strategies for analyzing these data • Pick a couple of tests and ignore the others – ADAS-Cog and MMSE – CDR and CDR-SB • Modifications of those tests – ADAS-Tree – ADAS-Rasch • Composite scores for specific domains – Z score – Something fancier using latent variable approach

Outline

• ADNI neuropsychological battery • Latent variable approaches – SEM and IRT • ADAS-Cog in ADNI • Memory in ADNI • Executive functioning in ADNI

Latent variable approach

• “Items” not intrinsically interesting, only as indicators of the underlying thing measured by the test • Many nice properties follow

Parallel development 1: SEM

• “Measurement part” of the model specifies how latent constructs are modeled • “Structural part” of the model specifies relationships between latent constructs and each other, and between latent constructs and other covariates

http://sites.google.com/site/lvmworkshop/home/downloads-general/2010-downloads

limmtotal avtot1 avtot2 avtot3 avtot4 avtot5 avtotb avtot6 ldeltotal avdel30min avdeltot cot1scor cot2scor cot3scor cot4totl mmballdl mmflagdl mmtreedl

Bunch of indicators

limmtotal avtot1 avtot2 avtot3 avtot4 avtot5 avtotb avtot6 ldeltotal avdel30min avdeltot cot1scor cot2scor cot3scor cot4totl mmballdl mmflagdl mmtreedl

“Memory”

Memory Underlying single factor with many indicators

LM story Rey word list ADAS word list MMSE words limmtotal avtot1 avtot2 avtot3 avtot4 avtot5 avtotb avtot6 ldeltotal avdel30min avdeltot cot1scor cot2scor cot3scor cot4totl mmballdl mmflagdl mmtreedl Memory

limmtotal avtot1 avtot2 avtot3 avtot4 avtot5 avtotb avtot6 ldeltotal avdel30min avdeltot cot1scor cot2scor cot3scor cot4totl mmballdl mmflagdl mmtreedl HC volume Memory* This is what we care about!

… and typically don’t care whether memory is modeled this way or with all that secondary structure

And sometimes we care about it for 600,000 SNPs, or for voxels – we need to move outside of an SEM package for some of the analyses we want to do

Parallel development 2: IRT

• Models are nested within SEM • Single factor confirmatory factor analysis model • Initially worked out with binary indicators • Extended 1960s to polytomous items (Samejima) • It’s only the measurement part • Attention to the quality of measurement and the quality of scores

Typical SEM example

Indicator 1 Indicator 2 Indicator 3 Indicator 4 Construct

Beck Zung CESD PHQ-9

Depression

Beck Zung CESD PHQ-9 PHQ-9 Depression

A closer look at PHQ-9

• A 9 item depression scale • Standard scores totalled up • Typical SEM model would take that total score and treat it as a continuous indicator by using a linear link (single loading parameter)

PHQ-9 measurement properties

Beck Zung CESD PHQ-9

IRT approach to PHQ-9

Depression Item 1 Item 2 Item 3 Item 4 Item 5 Item 6 Item 7 Item 8 Item 9 Depression*

SEM and IRT, then and now

• SEM was initially about total scores as indicators of constructs measured in common across tests • IRT was initially about item level data that had to satisfy assumptions • More recently: merging the strengths of the two approaches – Computational rather than conceptual advances, or maybe computational advances have fueled conceptual advances

Categorical data in SEM

• Runmplus code: • That little code snippet tells Mplus to treat all of the elements in the local `vlist’ as categorical data • Mplus default is WLSMV – Other appropriate ways of handling • Major reason Mplus is dominant SEM software used at FH

What about IRT?

• Array of tools to address measurement precision • Explicit focus on measurement properties and measurement precision differentiates it from SEM

Pretend for a moment that a single factor model was appropriate… • Item response theory (IRT) developed middle of last century – Lord and Novick / Birnbaum (1968) • Polytomous extension Samejima 1969 • Lord 1980 • Hambleton et al. 1991 • XCALIBRE, Parscale, Multilog – All variations on a single factor CFA model

4 items each at 0.5 increments 50 45 40 35 30 25 20 15 10 5 0 -3 -2 -1 0 1 2 3

Comments on that test

• Essentially linear test characteristic curve – Immaterial whether the standard score or the IRT score is used in analyses • No ceiling or floor effect – People at the extremes of the thing measured by the test will get some right and get some wrong • Pretty nice test!

2 items each at 0.5 increments

50 45 15 10 5 0 40 35 30 25 20 -3 -2 -1 0 1 2 3

Comments on that test

• But that’s what we said about the last one and it had twice as many items!

Why might we want twice as many items?

• Measurement precision / reliability • CTT: summarized in a single number: alpha • IRT: conceptualized as a quantity that may vary across the range of the test • Information • Mathematical relationship between information and standard error of measurement • Intuitively makes sense that a test with 2x the items will measure more precisely / more reliably than a test with 1x the items

Test information curves for those two tests 16 14 12 10 8 6 4 2 0 -3 -2 -1 0 1 2 3

Standard errors of measurement for the two tests 0.5

0.4

0.3

0.2

0.1

0 -3 -2 -1 0 1 2 3

Comments about these information and SEM curves • Information curves look more different than the SEM curves – Inverse square root relationship – TIC 100  – TIC 25  – TIC 16  – TIC 9  – TIC 4  SEM 0.10

SEM 0.20

SEM 0.25

SEM 0.33

SEM 0.50

(1/10) (1/5) (1/4) (1/3) (1/2) • Trade off between test length and measurement precision

These were highly selected “tests” • It would be possible to design such a test if we started with a robust item pool • Almost certainly not going to happen by accident / history • What are more realistic tests?

Test characteristic curves for 2 26 item dichotomous tests 30 25 20 15 10 5 0 -3 -2 -1 0 1 2 3

Comments on these TCCs

• Same number of items but very different shapes • Now it may matter whether you use an IRT score or a standard score in analyses • Both ceiling and floor effects

14 12 10 8 6 4 2 0 -3 -2 -1

TICs

0 1 2 3

1 0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0 -3 -2 -1

SEMs

0 1 2 3

Comments on the TICs and SEMs • Comparing the red test and the blue test: the red test is better for people of moderate ability (more items close to where they are) – For people right in the middle, measurement precision is just as good as a test twice as long • Items far away from your ability level don’t help your standard error • The blue test is better for people at the extremes (more items close to where

they

are)

Where do information curves come from?

• Item information curves use the same parameters as the item characteristic curves (difficulty level,

, and strength of association with latent trait or ability,

) (see next slides) • Test information is the sum of all of the item information curves – We can do that because of local independence

I( θ) = D

a

P( θ)Q(θ)

1 0.75

0.5

0.25

0 -3

I( θ) = D

a

* P( θ) *Q( θ)

-2 -1 0 1 2 3

I( θ) = D

a

P( θ) Q( θ)

1 0.75

0.5

0.25

0 -3 -2 -1 0 1 2 3

I( θ) = D

a

* P( θ)*Q(θ)

1 0.75

0.5

0.25

0 -3 -2 -1 0 1 2 3

1 0.75

0.5

0.25

0 -3

I( θ) = D

a

P( θ)Q(θ)

-2 -1 0 1 2 3

More thoughts on IICs

• The b parameters shift the location of the curve left and right (where is the bead on the string?) • The a parameters modify the height of the curve – But not tons; location location location • IRT gives us tools to evaluate and manage measurement precision

What about cognitive tests?

Cognitive screening tests

Typical SEM example

Indicator 1 Indicator 2 Indicator 3 Indicator 4 Construct Transformation of the scale of the indicator only works if it happens to be the inverse of the function from Cecile’s model! (Blom comment)

How about we fix the tools?

• Testlet models in IRT – Wainer, Bradlow, Wang (2007) – Li, Bolt, Fu (2006) • Embed everything in hierarchical Bayesian framework or SEM framework – Curtis (2010) • Modify IRT tools to integrate over nuisance factors for measurement precision – Curtis and Crane (2011 under review) • Sample from posterior – Mislevy (1992) – Crane, Feldman, et al., (2011 under development)

Outline

• ADNI neuropsychological battery • Latent variable approaches – SEM and IRT • ADAS-Cog in ADNI • Memory in ADNI • Executive functioning in ADNI

ADAS-Cog in ADNI

• Brief cognitive test • Commonly used outcome in drug trials • Described in handout • Two parts: cognitive part, rater part – Ignored in several manuscripts I have reviewed • Eliminating rater items does not reduce respondent burden!

Does it meet IRT assumptions?

6 5 4 3 2 1 0 1 2 3 4 5 6 7 8 9 10 11 12 13

q1 ADAS Cog 1 q9 q10 q11 q12 q14 q2 q3 q4 q5 q6 q7 q8 ADAS 1

More on assumptions: 1 factor CFA TESTS OF MODEL FIT Chi-Square Test of Model Fit Value 273.016* Degrees of Freedom 39** P-Value 0.0000

Chi-Square Test of Model Fit for the Baseline Model Value 4011.878

Degrees of Freedom 29 P-Value 0.0000

CFI/TLI CFI 0.941

TLI 0.956

Number of Free Parameters 63 RMSEA (Root Mean Square Error Of Approximation) Estimate 0.086

WRMR (Weighted Root Mean Square Residual) Value 1.515

More on assumptions

TESTS OF MODEL FIT Chi-Square Test of Model Fit Value 273.016* Degrees of Freedom 39** P-Value 0.0000

Chi-Square Test of Model Fit for the Baseline Model Value 4011.878

Degrees of Freedom 29 P-Value 0.0000

CFI/TLI

CFI 0.941

TLI 0.956

Number of Free Parameters 63

RMSEA (Root Mean Square Error Of Approximation) Estimate 0.086

WRMR (Weighted Root Mean Square Residual) Value 1.515

0.90 / 0.95

0.08 / 0.05

BUT the details

• Item 1 is an average across three learning trials – Already assumes we can just total them up • What if we treat the data as more granular, as 3 different indicators

Scree

6 5 8 7 4 3 2 1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Scree

8 7 6 5 4 3 2 1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

q1a q1b q1c q9 q10 q11 q12 q14 q2 q3 q4 q5 q6 q7 q8

ADAS Cog 2

ADAS 2

ADAS Cog 2 vs standard scores

-4 -2 0 single 2 4

Typical SEM example

Indicator 1 Indicator 2 Indicator 3 Indicator 4 Construct Transformation of the scale of the indicator only works if it happens to be the inverse of the function from Cecile’s model! (Blom comment)

Broad range of scores for each standard score -4 -2 0 single 2 4

Typical SEM example

Indicator 1 Indicator 2 Indicator 3 Indicator 4 Construct 1. Transformation of the scale of the indicator only works if it happens to be the inverse of the function from Cecile’s model! (Blom comment) 2. Transformations will only work at the level of granularity included in the model

Returning to the ADAS story

Item 1 Item 1a, 1b, 1c TESTS OF MODEL FIT Chi-Square Test of Model Fit Value 273.016* Degrees of Freedom 39** P-Value 0.0000 Chi-Square Test of Model Fit for the Baseline Model Value 4011.878 Degrees of Freedom 29 P-Value 0.0000 CFI/TLI CFI 0.941 TLI 0.956 Number of Free Parameters 63 RMSEA (Root Mean Square Error Of Approximation) Estimate 0.086 WRMR (Weighted Root Mean Square Residual) Value 1.515 TESTS OF MODEL FIT Chi-Square Test of Model Fit Value 372.634* Degrees of Freedom 48** P-Value 0.0000 Chi-Square Test of Model Fit for the Baseline Model Value 7559.487 Degrees of Freedom 28 P-Value 0.0000 CFI/TLI CFI 0.957 TLI 0.975 Number of Free Parameters 83 RMSEA (Root Mean Square Error Of Approximation) Estimate 0.091 WRMR (Weighted Root Mean Square Residual) Value 1.522 • Lot more parameters (20 more)

0.90 / 0.95

• CFI, TLI a bit better • RMSEA a bit worse

0.08 / 0.05

Secondary domain structure

• Methods effects – Same word list multiple times – Rater items • Subdomain effects

Word list

ADAS Cog 2 bi-factor

q1a q1b q1c q9 q10 q11 q12 q14 q2 q3 q4 q5 q6 q7 q8 ADAS 2

Fit for single vs bi-factor

Chi squared DF Single 372.634* 48** Bifactor 277.210* 46** Baseline Chi squared DF CFI TLI RMSEA 7559.487 7559.487

28 28 0.957

0.975

0.091

0.969

0.981

0.078

0.90 / 0.95

0.08 / 0.05

PQ01A PQ01B PQ01C PQ02 PQ03 PQ04 PQ05 PQ06 PQ07 PQ08 PQ09 PQ10 PQ11 PQ12 PQ14 Single

Loadings

Bifactor difference % difference -0.84

-0.91

-0.89

0.43

0.44

0.86

0.51

0.46

0.65

0.61

0.58

0.63

0.46

0.65

0.55

-0.72

-0.75

0.46

0.80

0.55

0.49

0.70

0.66

0.63

0.68

0.49

0.69

0.59

0.12

0.16

0.14

0.03

-0.06

0.03

0.05

0.04

-17% -22% -19% 6% 6% -7% 6% 7% 7% 8% 8% 7% 7% 6% 7%

Word list

ADAS Cog 2 bi-factor

q1a q1b q1c q9 q10 q11 q12 q14 q2 q3 q4 q5 q6 q7 q8 ADAS 2

Single factor vs. secondary factor structure • Single factor easier to explain but not consistent with theory, may or may not fit data as well – Single factor model has a nice history of use in educational testing – Nice tools to understand the quality of measurement • Bi-factor harder to explain, more consistent with theory, may fit data better – Tools not widely available to understand the quality of measurement – Scores often highly HIGHLY correlated between single factor and bi-factor models

Scores correlate 0.95

-2 -1 0 bifact2 1 2 3

Word list Ratings

ADAS Cog 3 bi-factor

q1a q1b q1c q9 q10 q11 q12 q14 q2 q3 q4 q5 q6 q7 q8 ADAS 3

Further improves fit

Chi squared DF Single 372.634* 48** Bifactor 2 Bifactor 3 277.210* 145.486* 46** 46** Baseline Chi squared DF CFI TLI RMSEA 7559.487 7559.487 7559.487

28 28 28 0.957

0.975

0.091

0.969

0.981

0.078

0.987

0.992

0.051

0.90 / 0.95

0.08 / 0.05

Differences in loadings

PQ01A PQ01B PQ01C PQ02 PQ03 PQ04 PQ05 PQ06 PQ07 PQ08 PQ09 PQ10 PQ11 PQ12 PQ14 Single -0.84

-0.91

-0.89

0.43

0.44

0.86

0.51

0.46

0.65

0.61

0.58

0.63

0.46

0.65

0.55

Bifactor 2 difference % difference Bifactor 3 Difference % difference -0.72

0.12

-17% -0.75

0.09

-13% -0.75

-0.75

0.16

0.14

-22% -19% -0.77

-0.78

0.14

0.11

-18% -14% 0.46

0.46

0.80

0.55

0.03

-0.06

0.03

6% 6% -7% 6% 0.46

0.46

0.83

0.54

0.03

-0.03

0.03

6% 6% -4% 6% 0.49

0.70

0.66

0.63

0.68

0.49

0.69

0.59

0.03

0.05

0.04

7% 7% 8% 8% 7% 7% 6% 7% 0.49

0.70

0.65

0.59

0.56

0.40

0.56

0.59

0.03

0.05

0.02

-0.07

-0.06

-0.08

0.04

7% 7% 7% 3% -12% -15% -15% 6%

Information content: single factor 12 1.2

10 8 6 1 0.8

0.6

4 2 0 -3 0.4

0.2

3 0 -2 -1 0 1 2

ADAS Single vs. Bi-factor

• Single factor is sure nice – Easy to explain – But it sure fits less well – And over-estimates measurement precision somewhat • Bi-factor is sure nice – Not so hard to explain – More consistent with our theory – Fits better • Measurement precision appropriately estimated at least within the SEM framework

I( θ) in presence of nuisance factors

So we integrate over the nuisance dimensions….

What about this?

Multiple versions longitudinally

• A bit of a challenge • But nothing we can’t handle!

• Does require some assumptions

0.79

0.90

0.77

0.78

1_3

0.86

Different versions of the word list 1_2

0.77

0.88

-2.5

1_1 -2 -1.5

-1 -0.5

0 0.5

1 1.5

0.71

0.72

0.81

2.5

0.79

0.90

0.77

0.78

1_3

0.86

Different versions of the word list 1_2

0.77

0.88

-2.5

1_1 -2 -1.5

-1 -0.5

0 0.5

1 1.5

0.71

0.72

0.81

2.5

Different versions of the word list 4 1_3

0.79

0.90

0.77

0.78

0.86

1_2

0.77

0.88

-2.5

1_1 -2 -1.5

-1 -0.5

0 0.5

1 1.5

0.71

0.72

0.81

2.5

Mplus ADAS model constraints 1

ADAS model constraints 2

ADAS model constraints 3

ADAS model constraints 4

ADAS model constraints 5

ADAS model constraints 6

And there’s a cost

• 320 lines of code Beginning Time: 14:09:56 Ending Time: 15:01:36 Elapsed Time: 00:51:40

Limitations

• Only tried this with the single factor model – Thresholds likely to be unaffected – But loadings may vary • Also relies on a crucial assumption that may not hold

Essentially co-calibration over time • Cross sectional co-calibration assumption: people are people. Relationship of items to the underlying domain, controlling for the latent ability level, is invariant across people • This seems problematic with memory vs. other domains when we have people with MCI and AD – Memory declines faster – Maybe not hugely so over 6 months

Outline

• ADNI neuropsychological battery • Latent variable approaches – SEM and IRT • ADAS-Cog in ADNI • Memory in ADNI • Executive functioning in ADNI

limmtotal avtot1 avtot2 avtot3 avtot4 avtot5 avtotb avtot6 ldeltotal avdel30min avdeltot cot1scor cot2scor cot3scor cot4totl mmballdl mmflagdl mmtreedl Scr/0

limmtotal avtot1 avtot2 avtot3 avtot4 avtot5 avtotb avtot6 ldeltotal avdel30min avdeltot cot1scor cot2scor cot3scor cot4totl mmballdl mmflagdl mmtreedl Scr/0 avtot1 avtot2 avtot3 avtot4 avtot5 avtotb avtot6 limmtotal avtot1 avtot2 avtot3 avtot4 avtot5 avtotb avtot6 avtot1 avtot2 avtot3 avtot4 avtot5 avtotb avtot6 limmtotal avtot1 avtot2 avtot3 avtot4 avtot5 avtotb avtot6 avdel30min avdeltot cot1scor cot2scor cot3scor cot4totl mmballdl mmflagdl mmtreedl 6 ldeltotal avdel30min avdeltot cot1scor cot2scor cot3scor cot4totl mmballdl mmflagdl mmtreedl 12 avdel30min avdeltot cot1scor cot2scor cot3scor cot4totl mmballdl mmflagdl mmtreedl 18 ldeltotal avdel30min avdeltot cot1scor cot2scor cot3scor cot4totl mmballdl mmflagdl mmtreedl 24

Strategies

• Ignore (likely a bad idea) • Use flexible approach we used for ADAS-Cog – Mplus to Paul: “That model looks hard. Perhaps you could give me an easier model.” – Paul to Mplus: “No, computers don’t date.” – Mplus: “Fine. I’m not talking to you anymore.” • Use dummy variables for study visit and regress versions on visit (Rich Jones) – But: single parameter does not address all differences in ADAS-Cog versions • ???

The “ignore” strategy

Executive functioning in ADNI

• Report is in the documentation • Bifactor scores

Acknowledgements • R01 AG 029672, “Improving cognitive outcome precision & responsiveness with modern psychometrics” • UW – Betsy – Elena – Elizabeth – Joey – Jonathan – Laura – McKay – RJ • Hebrew SeniorLife, UC Davis, Washington U – Alden – Chengjie – Dan – Danielle – Doug – Frances – Rich

Item 1 Item 2 Item 3 Item 4 Item 5 Item 6 Item 7 Item 8 Item 9 Depression*

Psychometric analysis of ADNI data

Transcript Psychometric analysis of ADNI data

Psychometric analyses of ADNI data

Disclaimer

Outline

Handout

Summary

Outline

Latent variable approach

Parallel development 1: SEM

Bunch of indicators

“Memory”

Parallel development 2: IRT

Typical SEM example

Depression

A closer look at PHQ-9

PHQ-9 measurement properties

IRT approach to PHQ-9

SEM and IRT, then and now

Categorical data in SEM

What about IRT?

Comments on that test

2 items each at 0.5 increments

Comments on that test

Comments on these TCCs

TICs

SEMs

I( θ) = D

a

*P( θ)*Q(θ)

I( θ) = D

a

* P( θ) *Q( θ)

I( θ) = D

a

*P( θ)* Q( θ)

I( θ) = D

a

* P( θ)*Q(θ)

I( θ) = D

a

*P( θ)*Q(θ)

More thoughts on IICs

What about cognitive tests?

Cognitive screening tests

Typical SEM example

How about we fix the tools?

Outline

ADAS-Cog in ADNI

Does it meet IRT assumptions?

More on assumptions

BUT the details

Scree

Scree

ADAS Cog 2

ADAS Cog 2 vs standard scores

Typical SEM example

Typical SEM example

Returning to the ADAS story

Secondary domain structure

ADAS Cog 2 bi-factor

Fit for single vs bi-factor

Loadings

ADAS Cog 2 bi-factor

Scores correlate 0.95

ADAS Cog 3 bi-factor

Further improves fit

Differences in loadings

ADAS Single vs. Bi-factor

What about this?

Multiple versions longitudinally

ADAS model constraints 2

ADAS model constraints 3

ADAS model constraints 4

ADAS model constraints 5

ADAS model constraints 6

And there’s a cost

Limitations

Outline

Strategies

P( θ)Q(θ)

P( θ) Q( θ)

P( θ)Q(θ)