Transcript Testing Theoretical Conjectures Strongly

Common Factors versus Components:
Principals and Principles,
Errors and Misconceptions
Keith F. Widaman
University of California at Davis
Presented at conference “Factor Analysis at 100”
L. L. Thurstone Psychometric Lab, University of
North Carolina at Chapel Hill, May 2004
Goal of the Talk
• Flip rendition
– (With apologies to Will) I come not to praise principal
components, but to bury them
– Thus, we might inter the procedure beside its creator
• More serious
– To outline several key assumptions, usually implicit, of
the “simpler” principal components approach
– Compare and contrast common factor analysis and
principal component analysis
2
Organization of the Talk
• Principals
– Major figures/events
– Important dimensions – factors/components
• Principles
– To organize our thinking
– Lead to methods to evaluate procedures
• Errors
– Structures of residuals
– Unclear presentations
• Misconceptions
3
Principal Individuals & Contributions
• Spearman (1904)
– First conceptualization of the nature of a common
factor – the element in common to two or more
indicators (preferably three or more)
– Stressed presence of two classes of factors –
• general (with one member) and
• specific (with a potentially infinite number)
– Key: Based evaluation of empirical evidence on the
tetrad difference criterion (i.e., on patterns in
correlations among manifest variables) with no
consideration of diagonal
4
Principal Individuals & Contributions
• Thomson (1916)
– Early recognition of elusiveness of theory – data
connection
– Single common factor implies hierarchical pattern of
correlations, but so does an opposite conceptualization
– Key for this talk: Focus was still on the patterns
displayed by off-diagonal correlation. Diagonal
elements were of no interest or importance
5
Principal Individuals & Contributions
• Thurstone (1931)
– First foray into factor analysis
– Devised a “center of gravity” method for estimation of
loadings
– Led to centroid method
– Key: Again, diagonal values explicitly disregarded
6
Principal Individuals & Contributions
• Hotelling (1933)
– Proposed method of principal components
– Method of estimation
• Least squares
• Decomposition of all of the variance of the manifest
variables into dimensions that are:
(a) orthogonal
(b) conditionally variance maximized
– Key 1: Left unities on diagonal
– Key 2: Interpreted unrotated solution
7
Principal Individuals & Contributions
• Thurstone (1935) – The Vectors of Mind
– “It is a fundamental criterion for a valid method of
isolating primary abilities that the weights of the
primary abilities for a test must remain invariant when
it is moved from one test battery to another test
battery.”
– “If this criterion is not fulfilled, the psychological
description of a test will evidently be as variable as the
arbitrarily chosen batteries into which the test may be
placed. Under such conditions no stable identification
of primary mental abilities can be expected.”
8
Principal Individuals & Contributions
• Thurstone (1935)
– This implies invariant factorial description of a test
(a) across batteries and (b) across populations
– Again, diagonal values explicitly disregarded
– Developed rationale for necessity for rotation
– Contra Hotelling:
• Unities on diagonal – imply manifest variables are
perfectly reliably
• Need for # dimensions = # manifest variables
• No rotation! This appears, to me, to be the most
important criticism of Hotelling by Thurstone.
9
Principal Individuals & Contributions
• McCloy, Metheny, & Knott (1938)
– Published in Psychometrika
– Sought to compare Common FA (Thurstone’s method)
vs. Principal Components Analysis (Hotelling’s
method)
– Perhaps the first comparison of the two methods
10
Principal Individuals & Contributions
• Thomson (1939)
– Clear statement of the differing aims of
• Common factor analysis – to explain the offdiagonal correlations among manifest variables
• Principal component analysis – to re-represent the
manifest variables in a mathematically efficient
manner
11
Principal Individuals & Contributions
• Guttman (1955, 1958)
– Developed lower bounds for the number of factors
– Weakest lower bound was number of “factors” with
eigenvalues greater than or equal to unity
• With unities on diagonal
• With population data
– Other bounds used other diagonal elements (e.g.,
strongest lower bound used SMCs), but these did not
work as well
12
Principal Individuals & Contributions
• Kaiser (1960, 1971)
– Described the origin of the Little Jiffy
• Principal components
• Retain components with eigenvalues >= 1.0
• Rotate using varimax
– Later modifications – Little Jiffy Mark IV – offered
important improvements, but were not followed
13
Principles – Mislaid or Forgotten
• Principle 1: Common factor analysis and principal
component analysis have different goals – à la
Thomson (1939)
– Common factor analysis – to explain the off-diagonal
correlations among manifest variables
– Principal component analysis – to re-represent the
original variables in a mathematically efficient manner
• (a) in reduced dimensionality, or
• (b) using orthogonal, conditionally variance
maximized way
14
Principles – Mislaid or Forgotten
• Principle 2: Common factor analysis was as much
a theory of manifest variables as a theory of latent
variables
– Spearman – doctrine of the indifference of the indicator,
so any manifest variable was a more-or-less good
indicator of g
– Thurstone – test one’s theory by developing new
variables as differing mixtures of factors and then
attempt to verify presumptions
– Today, focus seems largely on the latent variables
– Forgetting about manifest variables can be problematic
15
Principles – Mislaid or Forgotten
• Principle 3: Invariance of the psychological/
mathematical description of manifest variables is a
fundamental issue
– “It is a fundamental criterion for a valid method of
isolating primary abilities that the weights of the
primary abilities for a test must remain invariant when
it is moved from one test battery to another test battery”
– Much work on measurement & factorial invariance
– But, only similarities between common factors and
principal components are stressed; differences are not
emphasized
16
Principles – Mislaid or Forgotten
• Principle 4: Know data and model
– Should know relation between data and model
– Should know all assumptions (even implicit) of model
– Frequently told:
• information in correlation matrix is difficult to
discern
• so, don’t look at data
• run it through FA or PCA
• interpret the results
– This is not justifiable!
17
Common FA & Principal CA Models
• Common Factor Analysis
– R = FF’ + U2 = PΦP’ + U2
– where
• R is (p x p) correlation matrix among manifest vars
• F is a (p x k) unrotated factor matrix, with loadings
of p manifest variables on k factors
• U2 is a (p x p) matrix (diagonal) of unique factor
variances
• P is a (p x k) rotated factor matrix, with loadings of
p manifest variables on k rotated factors
• Φ is a (k x k) matrix of covariances among factors
(may be I, usually diag = I)
18
Common FA & Principal CA Models
• Principal Component Analysis
–
–
–
–
R = FcFc’
= PcΦcPc’
R = FcFc’ + GG’
= PcΦcPc’ + GG’
R = FcFc’ + Δ
= PcΦcPc’ + Δ
where
• Fc, Pc, & Φc have same order as like-named matrices
for CFA, but with c subscript to denote PCA
• G is a (p x [p-k]) matrix of loadings of p manifest
variables on the (p-k) discarded components
• Δ (= GG’) is a (p x p) matrix of covariances among
residuals
19
Present Day: Advice to Practicing Scientist
• Velicer & Jackson (1990): CFA vs. PCA
– Four “practical” issues
• Similarity between solutions
• Issues related to # of dimensions to retain
• Improper solutions in CFA
• Differences in computational efficiency
– Three “theoretical” issues
• Factorial indeterminacy in CFA, not PCA
• CFA can be used in exploratory and confirmatory
modes, PCA only exploratory
• CFA is latent procedure, PCA is manifest
20
Present Day: Advice to Practicing Scientist
• Goldberg & Digman (1994) and Goldberg &
Velicer (in press): CFA vs. PCA
– Results from CFA and PCA are so similar that
differences are unimportant
– If differences are large, “data are not well-structured
enough for either type of analysis”
– Use “factor” to refer to factors and components
– Aim is to explain correlations among manifest vars
21
Present Day: Quantitative Approaches
• Recent paper in Psychometrika (Ogasawara, 2003)
– Based work on oblique factors & components with:
• Equal number of indicators per dimension
• Independent cluster solution
• Sphericity (equal “error” variances), hence equal
factor loadings
– Derived expression for SEs (standard errors) for factor
and component loadings and intercorrelations
– SEs for PCA estimates were smaller than those for CFA
estimates, implying greater stability of (i.e., lower
variability around) population estimates
22
An Apocryphal Example
• Researcher wanted to develop a new inventory to assess
three cognitive traits
• Knew to collect data in at least two initial, derivation
samples
• Use exploratory procedures to verify initial, a priori
hypotheses
• Then, move on to confirmatory techniques
• So, Sample 1, N = 1600, and 8 manifest variables
• 3 Components explain 51% of total variance
23
Oblique Components, Sample 1
Variable
V1
V2
V3
N1
N2
N3
S1
S2
Fac 1
.704
.704
.704
.105
–.002
–.116
–.005
–.005
Fac 2
.002
.002
.002
.715
.725
.670
.002
.002
Fac 3
–.005
–.005
–.005
.065
.014
–.089
.735
.735
Fac 1
Fac 2
Fac 3
1.0
.256
.147
1.0
.147
1.0
. h2
.496
.496
.496
.575
.538
.417
.540
.540
24
.
Orthogonal Components, Sample 1
Variable
V1
V2
V3
N1
N2
N3
S1
S2
Fac 1
.698
.698
.698
.211
.104
–.025
.050
.050
Fac 2
.079
.079
.079
.717
.716
.643
.046
.046
Fac 3
.044
.044
.044
.127
.070
–.045
.732
.732
Fac 1
Fac 2
Fac 3
1.0
.000
.000
1.0
.000
1.0
. h2 .
.496
.496
.496
.575
.538
.417
.540
.540
25
An Apocryphal Example
• After confirming a priori hypotheses in Sample 1, the
researcher collected data from Sample 2
– Same manifest variables
– Sampled from the same general population
– Same mathematical approach – principal components
followed by oblique and orthogonal rotation
– Got same results!
• Decided to “test” the theory in Sample 3 – using “replicate
and extend” approach
– Major change: Switch to Confirmatory Factor Analysis
26
Confirmatory Factor Analysis, Sample 3
Variable
V1
V2
V3
N1
N2
N3
S1
S2
Fac 1
2.50 (.18)
3.00 (.21)
3.50 (.25)
.0
.0
.0
.0
.0
Fac 2
.0
.0
.0
2.10 (.13)
2.00 (.14)
1.50 (.16)
.0
.0
Fac 3
.0
.0
.0
.0
.0
.0
2.40 (.44)
2.70 (.50)
Fac 1
Fac 2
Fac 3
1.0
.50 (.04)
.50 (.10)
1.0
.50 (.10)
1.0
. θ2 .
18.75
27.00
36.75
4.59
12.00
22.75
58.24
73.71
27
Fully Standardized Solution, Sample 3
Variable
V1
V2
V3
N1
N2
N3
S1
S2
Fac 1
Fac 2
Fac 3
Fac 1
.50
.50
.50
.0
.0
.0
.0
.0
1.0
.50
.50
Fac 2
.0
.0
.0
.70
.50
.30
.0
.0
1.0
.50
Fac 3
.0
.0
.0
.0
.0
.0
.30
.30
1.0
. h2
.25
.25
.25
.49
.25
.09
.09
.09
28
.
Oblique Component Solution, Sample 3
Variable
V1
V2
V3
N1
N2
N3
S1
S2
Fac 1
.704
.704
.704
.105
–.002
–.116
–.005
–.005
Fac 2
.002
.002
.002
.715
.725
.670
.002
.002
Fac 3
–.005
–.005
–.005
.065
.014
–.089
.735
.735
Fac 1
Fac 2
Fac 3
1.0
.256
.147
1.0
.147
1.0
. h2
.496
.496
.496
.575
.538
.417
.540
.540
29
.
An Early Comparison
• McCloy, Metheny, & Knott (1938)
– Published in Psychometrika
– Sought to compare Common FA (Thurstone’s method)
vs. Principal Components Analysis (Hotelling’s)
– Stated that Principal Components can be rotated
– So, both techniques are different means to same end
– Principal difference:
• Thurstone inserts largest correlation in row in the
diagonal of each residual matrix
• Hotelling begins with unities and stays with residual
values in each residual matrix
30
Hypothetical Factor Matrix (McCloy et al.)
Variable
Fac 1
Fac 2
Fac 3
1
2
3
4
5
6
7
8
9
10
.900
.800
.0
.0
.0
.0
.0
.566
.495
.520
.0
.0
.700
.800
.0
.0
.424
.566
.0
.520
.0
.0
.0
.0
.900
.600
.424
.0
.495
.520
. h2
.810
.640
.490
.640
.810
.360
.360
.640
.490
.810
31
.
Rotated Factor Matrix (McCloy et al.)
Variable
Fac 1
Fac 2
Fac 3
1
2
3
4
5
6
7
8
9
10
.860
.819
.014
.023
.010
–.008
–.011
.587
.516
.489
–.033
–.025
.726
.766
–.004
.029
.434
.548
–.038
.471
.035
.023
.000
.004
.808
.645
.466
–.014
.512
.537
. h2
.742
.672
.527
.587
.653
.417
.406
.645
.530
.749
32
.
Rotated Component Matrix (McCloy et al.)
Variable
Fac 1
Fac 2
Fac 3
1
2
3
4
5
6
7
8
9
10
.906
.874
.034
.050
–.060
–.094
–.054
.653
.527
.527
–.055
–.053
.824
.859
.000
–.035
.519
.558
–.085
.477
.063
.046
–.021
–.006
.885
.773
.525
.029
.605
.552
. h2
.828
.769
.681
.740
.787
.608
.548
.739
.651
.810
33
.
An Early Comparison
• McCloy, Metheny, & Knott (1938)
– Argued that:
• both CFA and PCA were means to same end
• both led to similar pattern of loadings, but
• Thurstone’s method was more accurate (Δh2 = .056)
than Hotelling’s (Δh2 = .125) – [but these were
average absolute differences]
• I averaged signed differences, and Thurstone’s
method was much accurate (Δh2 = -.013) than
Hotelling’s (Δh2 = .120)
34
An Early Comparison
• McCloy, Metheny, & Knott (1938)
– Found similar pattern of high and low loadings from
PCA and CFA
– But, they found (but did not stress) that PCA led to
decidedly higher loadings
• Tukey (1969)
– Amount, as well as direction, is vital
– For any science to advance, we must pay attention to
quantitative variation, not just qualitative
35
Regularity Conditions or Phenomena
• Relations between population values of P and R
• Features of eigenvalues
• Covariances among residuals
– Need a “theory” of “errors”
– Recount my first exposure …
– Should have to acknowledge (predict? live with?) the
patterns in residuals
36
Practicing Scientists vs. “Statisticians”
• Interesting dimension along which researchers
fall:
Practicing
scientists
use CFA
use regression
analysis
“Statisticians”
(Dark side)
prefer PCA
warn of probs
errors in vars
37
Practicing Scientists vs. “Statisticians”
• At first seems odd
– Practicing scientist prefers
• CFA (which partials out errors of measurement and
specific variance)
• Regression analysis – despite the implicit
assumption of “perfect” measurement
– “Statistician” prefers
• To warn of ill-effects of errors in variables on results
of regression analysis
• PCA (despite lack of attention to measurement
error), perhaps due to elegant, reduced rank
representation
38
Practicing Scientists vs. “Statisticians”
• On second thought, is rational:
– Practicing scientist prefers
• Assumptions that residuals (in CFA or regression
analysis) are independent, uncorrelated, normally
distributed
– “Statistician” prefers
• To try to circumvent (or solve) problem of errors in
variables in regression
• To relegate “errors in variables” problems in PCA to
that part of solution (GG’) that is orthogonal to the
retained part, thereby circumventing (or solving)
this problem
39
Regularity Conditions or Phenomena
• In Common Factor Analysis,
– Char. of correlations  Char. of variables
– Char. of correlations  Char. of variables
1:1
1:1
• In Principal Component Analysis,
– Char. of correlations  Char. of variables
1:1 (??)
– Char. of correlations ~[] Char. of variables many:1
40
Manifest Correlations
Var
V1
V2
V3
V4
V5
V6
V1
1.00
.64
.64
.64
.64
.64
V2
V3
V4
V5
V6
1.00
.64
.64
.64
.64
1.00
.64
.64
.64
1.00
.64
.64
1.00
.64
1.00
41
Eigenvalues, Loadings, and Explained Variance
Var
V1
V2
V3
V4
V5
V6
P1
P2
EV
1.92
.0
.0
P1
.80
.80
.80
1.0
P2
h2
.64
.64
.64
EVc
2.28
.36
.36
Pc1
.87
.87
.87
Pc2
hc2
.76
.76
.76
1.0
42
Residual Covariances: CFA
Var
V1
V2
V3
V4
V5
V6
V1
.36
.00
.00
V2
.00
.36
.00
V3
.00
.00
.36
V4
V5
V6
Covs below diag., corrs above diag.
43
Residual Covariances: PCA
Var
V1
V2
V3
V4
V5
V6
V1
.24
-.12
-.12
V2
-.50
.24
-.12
V3
-.50
-.50
.24
V4
V5
V6
Covs below diag., corrs above diag.
44
Eigenvalues, Loadings, and Explained Variance
Var
V1
V2
V3
V4
V5
V6
P1
P2
EV
3.84
.0
.0
.0
.0
.0
P1
.80
.80
.80
.80
.80
.80
1.0
P2
h2
.64
.64
.64
.64
.64
.64
EVc
4.20
.36
.36
.36
.36
.36
Pc1
.84
.84
.84
.84
.84
.84
Pc2
hc2
.70
.70
.70
.70
.70
.70
1.0
45
Residual Covariances: CFA
Var
V1
V2
V3
V4
V5
V6
V1
.36
.00
.00
.00
.00
.00
V2
.00
.36
.00
.00
.00
.00
V3
.00
.00
.36
.00
.00
.00
V4
.00
.00
.00
.36
.00
.00
V5
.00
.00
.00
.00
.36
.00
V6
.00
.00
.00
.00
.00
.36
Covs below diag., corrs above diag.
46
Residual Covariances: PCA
Var
V1
V2
V3
V4
V5
V6
V1
.30
-.06
-.06
-.06
-.06
-.06
V2
-.20
.30
-.06
-.06
-.06
-.06
V3
-.20
-.20
.30
-.06
-.06
-.06
V4
-.20
-.20
-.20
.30
-.06
-.06
V5
-.20
-.20
-.20
-.20
.30
-.06
V6
-.20
-.20
-.20
-.20
-.20
.30
Covs below diag., corrs above diag.
47
Regularity Conditions or Phenomena
• In Common Factor Analysis,
– If (a) the model fits in the population, (b) there is one
factor, and (c) communalities are estimated optimally,
– Single non-zero eigenvalue
– Factor loadings and residual variances for first three
variables are unaffected by addition of 3 “identical”
variables
– Residuals = specific + error variance
– Residual matrix is diagonal
48
Regularity Conditions or Phenomena
• In Principal Component Analysis,
– If (a) the common factor model fits in the population,
(b) there is one factor, and (c) unities are retained on the
main diagonal,
– Single large eigenvalue, plus (p – 1) identical, smaller
eigenvalues
– Residual component matrix G is independent of the
space defined by Fc
– But, residual covariance matrix is clearly non-diagonal
– And, (a) “population” component loadings and (b)
residual variances and covariances vary as a function of
number of manifest variables!
49
Manifest Correlations
Var
V1
V2
V3
V4
V5
V6
V1
1.00
.36
.36
.36
.36
.36
V2
V3
V4
V5
V6
1.00
.36
.36
.36
.36
1.00
.36
.36
.36
1.00
.36
.36
1.00
.36
1.00
50
Eigenvalues, Loadings, and Explained Variance
Var
V1
V2
V3
V4
V5
V6
P1
P2
EV
1.08
.0
.0
P1
.60
.60
.60
1.0
P2
h2
.36
.36
.36
EVc
1.72
.64
.64
Pc1
.76
.76
.76
Pc2
hc2
.57
.57
.57
1.0
51
Residual Covariances: CFA
Var
V1
V2
V3
V4
V5
V6
V1
.64
.00
.00
V2
.00
.64
.00
V3
.00
.00
.64
V4
V5
V6
Covs below diag., corrs above diag.
52
Residual Covariances: PCA
Var
V1
V2
V3
V4
V5
V6
V1
.43
-.21
-.21
V2
-.50
.43
-.21
V3
-.50
-.50
.43
V4
V5
V6
Covs below diag., corrs above diag.
53
Eigenvalues, Loadings, and Explained Variance
Var
V1
V2
V3
V4
V5
V6
P1
P2
EV
2.16
.0
.0
.0
.0
.0
P1
.60
.60
.60
.60
.60
.60
1.0
P2
h2
.36
.36
.36
.36
.36
.36
EVc
4.80
.64
.64
.64
.64
.64
Pc1
.68
.68
.68
.68
.68
.68
Pc2
hc2
.47
.47
.47
.47
.47
.47
1.0
54
Residual Covariances: CFA
Var
V1
V2
V3
V4
V5
V6
V1
.64
.00
.00
.00
.00
.00
V2
.00
.64
.00
.00
.00
.00
V3
.00
.00
.64
.00
.00
.00
V4
.00
.00
.00
.64
.00
.00
V5
.00
.00
.00
.00
.64
.00
V6
.00
.00
.00
.00
.00
.64
Covs below diag., corrs above diag.
55
Residual Covariances: PCA
Var
V1
V2
V3
V4
V5
V6
V1
.53
-.11
-.11
-.11
-.11
-.11
V2
-.20
.53
-.11
-.11
-.11
-.11
V3
-.20
-.20
.53
-.11
-.11
-.11
V4
-.20
-.20
-.20
.53
-.11
-.11
V5
-.20
-.20
-.20
-.20
.53
-.11
V6
-.20
-.20
-.20
-.20
-.20
.53
Covs below diag., corrs above diag.
56
Regularity Conditions or Phenomena
• So, the difference between “population” parameters from
CFA and PCA diverge more:
– (a) the fewer the number of indicators per
dimension, and
– (b) the lower the true communality
• But, some regularities still seem to hold (although these
vary with the number of indicators)
– “regular” estimates of loadings
– “regular” magnitude of residual covariance
– “regular” magnitude of residual covariance
– “regular” form of eigenvalue structure
57
Regularity Conditions or Phenomena
• But, what if we have variation in loadings?
58
Manifest Correlations
Var
V1
V2
V3
V4
V5
V6
V1
1.00
.64
.64
.48
.48
.48
V2
V3
V4
V5
V6
1.00
.64
.48
.48
.48
1.00
.48
.48
.48
1.00
.36
.36
1.00
.36
1.00
59
Eigenvalues, Loadings, and Explained Variance
Var
V1
V2
V3
V4
V5
V6
P1
P2
EV
3.00
.0
.0
.0
.0
.0
P1
.80
.80
.80
.60
.60
.60
1.0
P2
h2
.64
.64
.64
.36
.36
.36
EVc
3.47
.64
.64
.53
.36
.36
Pc1
.83
.83
.83
.68
.68
.68
Pc2
hc2
.69
.69
.69
.47
.47
.47
1.0
60
Residual Covariances: CFA
Var
V1
V2
V3
V4
V5
V6
V1
.36
.00
.00
.00
.00
.00
V2
.00
.36
.00
.00
.00
.00
V3
.00
.00
.36
.00
.00
.00
V4
.00
.00
.00
.64
.00
.00
V5
.00
.00
.00
.00
.64
.00
V6
.00
.00
.00
.00
.00
.64
Covs below diag., corrs above diag.
61
Residual Covariances: PCA
Var
V1
V2
V3
V4
V5
V6
V1
.31
-.05
-.05
-.09
-.09
-.09
V2
-.15
.31
-.05
-.09
-.09
-.09
V3
-.15
-.15
.31
-.09
-.09
-.09
V4
-.21
-.21
-.21
.53
-.11
-.11
V5
-.21
-.21
-.21
-.20
.53
-.11
V6
-.21
-.21
-.21
-.20
-.20
.53
Covs below diag., corrs above diag.
62
Regularity Conditions or Phenomena
• So, with variation in loadings
• One piece of approximate stability
– “regular” estimates of loadings
• But, sacrifice
– “regular” magnitude of residual covariance
– “regular” magnitude of residual covariance
– “regular” form of eigenvalue structure
63
Regularity Conditions or Phenomena
• But, what if we have multiple factors?
• Let’s start with
– (a) equal loadings
– (b) orthogonal factors
64
Eigenvalues, Loadings, and Explained Variance
Var
V1
V2
V3
V4
V5
V6
P1
P2
EV
1.08
1.08
.0
.0
.0
.0
P1
.60
.60
.60
.0
.0
.0
1.0
.0
P2
.0
.0
.0
.60
.60
.60
1.0
h2
.64
.64
.64
.64
.64
.64
EVc
1.72
1.72
.64
.64
.64
.64
Pc1
.76
.76
.76
.0
.0
.0
Pc2
.0
.0
.0
.76
.76
.76
1.0
.0
1.0
hc2
.57
.57
.57
.57
.57
.57
65
Residual Covariances: PCA
Var
V1
V2
V3
V4
V5
V6
V1
.43
-.21
-.21
.00
.00
.00
V2
-.50
.43
-.21
.00
.00
.00
V3
-.50
-.50
.43
.00
.00
.00
V4
.00
.00
.00
.43
-.21
-.21
V5
.00
.00
.00
-.50
.43
-.21
V6
.00
.00
.00
-.50
-.50
.43
Covs below diag., corrs above diag.
66
Regularity Conditions or Phenomena
• So, “strange” result:
– Same factor inflation as with 1-factor, 3 indicators
– Same within-factor residual covariances as for 1-factor,
3 indicators
– But, between-factor residual covariances = 0!
• Let’s go to
– (a) equal loadings, but
– (b) oblique factors
67
Eigenvalues, Loadings, and Explained Variance
Var
V1
V2
V3
V4
V5
V6
P1
P2
EV
1.62
.54
.0
.0
.0
.0
P1
.60
.60
.60
.0
.0
.0
1.0
.5
P2
.0
.0
.0
.60
.60
.60
1.0
h2
.64
.64
.64
.64
.64
.64
EVc
2.26
1.18
.64
.64
.64
.64
Pc1
.76
.76
.76
.0
.0
.0
Pc2
.0
.0
.0
.76
.76
.76
hc2
.57
.57
.57
.57
.57
.57
1.0
.31 1.0
68
Residual Covariances: PCA
Var
V1
V2
V3
V4
V5
V6
V1
.43
-.21
-.21
.00
.00
.00
V2
-.50
.43
-.21
.00
.00
.00
V3
-.50
-.50
.43
.00
.00
.00
V4
.00
.00
.00
.43
-.21
-.21
V5
.00
.00
.00
-.50
.43
-.21
V6
.00
.00
.00
-.50
-.50
.43
Covs below diag., corrs above diag.
69
Regularity Conditions or Phenomena
• So, “strange” result:
– Same factor inflation as with 1-factor, 3 indicators
– Reduced correlation between factors
– But, residual covariances matrix is identical!
• Let’s go to
– (a) unequal loadings, and
– (b) orthogonal factors
70
Eigenvalues, Loadings, and Explained Variance
Var
V1
V2
V3
V4
V5
V6
P1
P2
EV
1.16
1.16
.0
.0
.0
.0
P1
.80
.60
.40
.0
.0
.0
1.0
.0
P2
.0
.0
.0
.80
.60
.40
1.0
h2
.36
.64
.84
.36
.64
.84
EVc
1.70
1.70
.79
.79
.51
.51
Pc1
.83
.78
.64
.0
.0
.0
Pc2
.0
.0
.0
.83
.78
.64
1.0
.0
1.0
hc2
.68
.61
.41
.68
.61
.41
71
Residual Covariances: PCA
Var
V1
V2
V3
V4
V5
V6
V1
.32
-.16
-.21
.00
.00
.00
V2
-.47
.39
-.26
.00
.00
.00
V3
-.48
-.55
.59
.00
.00
.00
V4
.00
.00
.00
.32
-.16
-.21
V5
.00
.00
.00
-.47
.39
-.26
V6
.00
.00
.00
-.48
-.55
.59
Covs below diag., corrs above diag.
72
Regularity Conditions or Phenomena
• So, “strange” result:
– Different factor inflation than with 1-factor, 3 indicators
– Reduced correlation between factors
– But, residual covariances matrix has unequal
covariances and correlations among residuals, but
between-factor covariances = 0!
• Let’s go to
– (a) unequal loadings, and
– (b) oblique factors
73
Eigenvalues, Loadings, and Explained Variance
Var
V1
V2
V3
V4
V5
V6
P1
P2
EV
1.74
.58
.0
.0
.0
.0
P1
.80
.60
.40
.0
.0
.0
1.0
.5
P2
.0
.0
.0
.80
.60
.40
1.0
h2
.36
.64
.84
.36
.64
.84
EVc
2.27
1.16
.79
.77
.52
.49
Pc1
.77
.77
.71
.11
.00
-.12
Pc2
.11
.00
-.12
.77
.77
.71
hc2
.66
.59
.46
.66
.59
.46
1.0
.32 1.0
74
Residual Covariances: PCA
Var
V1
V2
V3
V4
V5
V6
V1
.34
-.14
-.21
-.04
-.04
.00
V2
-.38
.41
-.28
-.04
-.02
.03
V3
-.49
-.59
.54
.00
.03
.08
V4
-.13
-.11
.01
.34
-.14
-.21
V5
-.11
-.04
.07
-.38
.41
-.28
V6
.01
.07
.16
-.49
-.59
.54
Covs below diag., corrs above diag.
75
Regularity Conditions or Phenomena
• So, “strange” result:
– Extremely different factor inflation than with 1-factor, 3
indicators
– Largest loading is now UNderrepresented
– Very different population factor loadings (.8, .6, & .4)
have very similar component loadings
– Now, between-factor covariances are not zero, and
some are positive!
76
R from Component Parameters
• All the preceding from a CFA view:
–
–
–
–
–
Develop parameters from a CF model
Analyze using CFA and PCA
CFA procedures recover parameters
PCA procedures exhibit failings or anomalies
So What? What else could you expect?
• Challenge (to me):
– Generate data from a PC model
– Analyze using CFA and PCA
– PCA should recover parameters, CFA should exhibit
problems and/or anomalies
77
R from Component Parameters
• Difficult to do
• Leads to
– Impractical, unacceptable outcomes, from the point of
view of the practicing scientist
– Crucial indeterminacies with the PCA model
78
R from Component Parameters
• Impractical, unacceptable outcomes, from the
point of view of the practicing scientist
79
Manifest Correlations
Var
V1
V2
V3
V4
V5
V6
V1
1.00
.46
.46
V2
V3
1.00
.46
1.00
V4
V5
V6
First principal component has 3 loadings of .8
First principal factor has 3 loadings of (.46)1/2, or about .67
80
Manifest Correlations
Var
V1
V2
V3
V4
V5
V6
V1
1.00
.568
.568
.568
.568
.568
V2
V3
V4
V5
V6
1.00
.568
.568
.568
.568
1.00
.568
.568
.568
1.00
.568
.568
1.00
.568
1.00
First principal component has 6 loadings of .8
First principal factor has 6 loadings of (.568)1/2, or about .75
But, one would have to alter the first 3 tests, as their “population”
correlations are altered
81
R from Component Parameters
• Crucial indeterminacies with the PCA “model”
– Consider case of well-identified CFA model: 6 manifest
variables loading on a single factor
– One could easily construct the population matrix as FF’
+ uniquenesses to ensure diag(R) = I
– With 6 manifest variables, 6(7)/2 = 21 unique elements
of covariance matrix
– 12 parameter estimates
– therefore 9 df
82
R from Component Parameters
• Crucial indeterminacies with the PCA “model”
– Consider now 6 manifest variables with defined
loadings on first PC
– To estimate the correlation matrix, must come up with
the remaining 5 PCs
– A start: [Fc | G]’ [Fc | G] = diag, so orthogonality
constraint yields 6(5)/2 = 15 equations
– Sum of squares across rows = 1, so 6 more equations
– In short, 15 equations, but 30 unknowns (loadings of 6
variables on the 5 components in G)
– Therefore, an infinite # of R matrices will lead to the
stated first PC
83
R from Component Parameters
• Crucial indeterminacies with the PCA “model”
– Related to the Ledermann number, but in reverse
– For example, with 10 manifest variables, one can
minimally overdetermine no more than 6 factors (so use
6 or fewer factors)
– But, here, one must specify at least 6 components (to
ensure more equations than unknowns) to ensure a
unique R
– If fewer than 6 components are specified, an infinite
number of solutions for R can be found
84
Conclusions: CFA
• CFA factor models may not hold in the population
• But, if they do (in a theoretical population):
– The notion of a population factor loading is realistic
– The population factor loading is unaffected by presence
of other variables, as long as the battery contains the
same factors
– In one-factor case, loadings can vary from 0 to 1
(provided reflection of variables is possible)
– This generalizes to the case of multiple factors
85
Conclusions: CFA
• CFA factor models may not hold in the population
• But, if they do:
– Residual (i.e., unique) variances are uncorrelated
– Magnitude of unique variance for a given variable is
unaffected by other variables in the analysis
86
Conclusions: PCA
• PCA factor models cannot hold in the population (because
all variables have measurement error)
• Moreover:
– The notion of “the population component loading” for a
particular manifest variable is meaningless
– The “population” component loading is affected
strongly by presence of other variables
– SEs for component loadings have no interpretation
– In the one-component case, component loadings can
only vary from (1/m)1/2 to 1, where m is the number of
indicators for the dimension
– Generalizes to multiple component case
87
Conclusions: PCA
• PCA factor models cannot hold in the population (because
all variables have measurement error)
• Moreover:
– Residual variables are correlated, often in unpredictable
and seemingly haphazard fashion
– Magnitude of unique variance and covariances for a
given manifest variable are affected by other variables
in the analysis
88
Conclusions: PCA
• PCA factor models cannot hold in the population (because
all variables have measurement error)
• Moreover:
– Finally, generating data from a PC model leads either to
• Impractical, unacceptable outcomes
• Indeterminacies in the parameter – R relations
89
90
91