Transcript Creating Valid and Effective Measures

Creating Valid and Effective
Measures
Using –optifact- to Create Better
Summated Rating Scales
North American Stata Users' Group Meetings
Boston, July 11-12, 2005
Are measures important?
Measures are how theory is translated into
data for modeling
Unlike art, where we want to engage
individual interpretation as a major
component of the work
Measures need to be unambiguously
interpreted in precisely the same way by all
Why Scales?
Combining the results of several questions
is more reliable and precise, and reduces
measurement error (Spector, 1992)
What makes a good scale?
 It is reliable
• The same measurement, produces similar results across time
and persons
• The questions or items that make up the scale are consistent
with one another
 It is valid
• It measures the concept that it is intended to measure
 It measures one concept
 It is regularly distributed
Reliability
Internal Reliability
• Internal consistency: the items should intercorrelate (Cronbach’s Alpha)
Test-Retest Reliability
• Same measure taken again should produce
consistent results
Validity
Rationale
• The items must be consistent with theory and the
concept under consideration
Criterion Validity
• The items should vary in the same way to other
variables that are theoretically related to the
concept
• Canonical correlation
Uni-dimensional, Efficient
Scales that measure a single concept are
more useful in establishing causal
relationships
Efficient scales use only enough items to
reliably measure a concept
Regular distributions allow standard
modeling techniques
Research Question
If my scales are not uni-dimensional, are
there sub-scales which are and meet the
criteria associated with good scales?
Investigated using data gathered for the
Canadian National Longitudinal Survey of
Children and Youth by Statistics Canada.
Data and Method
The NLSCY cycles 1, 2 and 3
• Measures on parenting and child behaviour taken
every two years (allows the investigations of testretest reliability) N > 13,000.
Common factor analysis
• Dimensions, alpha
Canonical correlation
• Criterion validity
Problems
There are a lot of candidate scales
• 2k – 1, if scales of all sizes are considered
The testing procedure is lengthy
•
•
•
•
•
Factor analysis (dimensionality)
Alpha (internal reliability)
Canonical Correlation (external validity)
Re-test (all of the above, twice)
More than 13 scales
Solution: optifact
A Stata software component to analyze a
list of candidate items for the creation of a
summated rating scale
-optifact- Specification
optifact varlist [weight] [if exp] [in range]
[, top(#) smallest(#) largest(#) maxfact(#) minslope(#)]
Varlist: the list of candidate items (numeric)
Options:
Top: the number of candidate scales to list (10)
Smallest: the smallest k allowed (3)
Largest: the smallest k allowed (all items)
Maxfact: the maximum number of factors allowed (1)
-optifact- Output
optifact abecq6b abecq6n abecq6w abecq6p abecq6i abecq6qq abecq6hh abecq6s
[aweight=awtcw01c], top(20) criteria(childsex ammcq01)
8 items will be processed
There are 219 potential scales
56 combinations of
Top 5 scales using
Avg.
K Alpha Cov.
- ----- ----1 3 0.774 0.201
2 3 0.769 0.203
3 3 0.744 0.231
4 3 0.742 0.169
5 3 0.734 0.213
3 items had
3 items
Can
Chk Items
--- ----Yes abecq6p
Yes abecq6p
Yes abecq6b
Yes abecq6p
Yes abecq6n
55 combinations of
Top 5 scales using
Avg.
K Alpha Cov.
- ----- ----1 4 0.802 0.185
2 4 0.784 0.199
3 4 0.776 0.191
4 4 0.775 0.198
4 items had
4 items
Can
Chk Items
--- ----Yes abecq6p
Yes abecq6n
Yes abecq6n
Yes abecq6b
one factor, 0 had more than one factor
abecq6i abecq6qq
abecq6i abecq6hh
abecq6n abecq6hh
abecq6qq abecq6hh
abecq6p abecq6i
one factor, 15 had more than one factor
abecq6i
abecq6p
abecq6p
abecq6p
abecq6qq abecq6hh
abecq6i abecq6hh
abecq6i abecq6qq
abecq6i abecq6hh
Behaviour Scales
Variable
Concepts
k
Alpha
Hyperactivity
(becs06)
3
8
0.84
Prosocial Behaviour
(becs07)
4
10
0.76
Emotional Disorder
/Anxiety (becs07)
3
8
0.78
Aggression
(becs09
2
6
0.78
Indirect Aggression
(becs10)
1
5
0.78
Property Offences
(becs11)
2
6
0.62
Parenting Scales
Variable
Concepts
k
Alpha
Positive interaction (0-23
months) (prcs01)
2
5
0.73
Ineffective parenting style
(0-23 months) (prcs02)
1
2
0.37
Positive interaction (2-11
years) (prcs03)
2
5
0.81
Ineffective parenting style
(2-11 years) (prcs04)
3
7
0.70
Consistency (2-11 years)
(prcs05)
2
5
0.68
Rational parenting style
(2-11 years) (prcs06)
2
6
0.62
PMK Depression
(dpps01)
4
12
0.82
Results
Of these 13 scales only one was uni-
dimensional
Most had larger number of items than
desirable for the value of alpha
New Behaviour Scales
Concepts
k
Alpha
Concepts’
k’
Alpha’
Hyperactivity
(becs06)
3
8
0.84
1
3
0.80
Prosocial Behaviour
(becs07)
4
10
0.76
1
4
0.77
Emotional Disorder
/Anxiety (becs07)
3
8
0.78
1
3
0.62
Aggression
(becs09)
2
6
0.78
1
3
0.67
Indirect Aggression
(becs10)
1
5
0.78
1
3
0.70
Property Offences
(becs11)
2
6
0.62
1
2
0.63
Variable
New Parenting Scales
Concepts
k
Alpha
Concepts’
k’
Alpha’
Positive interaction (0-23
months) (prcs01)
2
5
0.73
1
3
0.63
Ineffective parenting style (023 months) (prcs02)
1
2
0.37
Positive interaction (2-11
years) (prcs03)
2
5
0.81
1
3
0.78
Ineffective parenting style (211 years) (prcs04)
3
7
0.70
Consistency (2-11 years)
(prcs05)
2
5
0.68
1
3
0.66
Rational parenting style (2-11
years) (prcs06)
2
6
0.62
PMK Depression (dpps01)
4
12
0.82
1
3
0.75
Variable
Implications
These scales are part of a large survey
conducted every 2 years in Canada on
more than 20,000 children
The survey has completed 6 cycles of data
collection
Each question has been asked more than
120,000 times
Implications (Cont’d)
The scales required 85 questions
The revised scales require 30 questions
More than 55,000,000 questions
Caveats
Most scales in the NLSCY are not the best
that can be made from the available data
Some of these scales should not be used
The NLSCY is an excellent survey,
conducted by competent statisticians
Similar results might be expected in other
major surveys
Conclusions
-optifact- can help find better measures
• Uni-dimensional
• Valid
• Reliable
-optifact- can reduce costs
• Equivalent or better measure for same money