Transcript Correcting for Indirect Range Restriction in Meta

CORRECTING FOR INDIRECT RANGE
RESTRICTION IN META-ANALYSIS:
DETERMINING THE UT DISTRIBUTION
Huy Le
University of Central Florida
In-Sue Oh
University of Iowa
THE IMPORTANCE OF CORRECTING FOR
STUDY ARTIFACTS IN META-ANALYSIS
•
•
The major goal of meta-analysis is to estimate the true
relationships between variables (constructs) from
observed correlations.
These observed correlations, however, are influenced
by the effects of study artifacts  Meta-analysts need
to take these effects into account in order to accurately
estimate the true correlations.
RANGE RESTRICTION AS A STUDY ARTIFACT

What is Range restriction?


Occurs when the variance of a variable in a sample
is reduced due to pre-selection or censoring in some
way (Ree, Carretta, Earles, & Albert, 1994).
Effects of range restriction

Statistics estimated in such a restricted sample
(incumbent sample) are biased, attenuated
estimates of parameters in the unrestricted
population (applicant sample).
TWO TYPES OF RANGE RESTRICTION
Direct Range Restriction
Explicit Selection on X resulting
in distortion of correlation between
X and Y
Indirect Range Restriction
Explicit Selection on a third variable Z
resulting in distortion of correlation
between X and Y (which are correlated
with Z); Always the case for concurrent
validation studies
X
Y
Suitability scores
Z
X
GATB
Multivariate Range Restriction
An extension of indirect range restriction where explicit selections
occur on several variables
Y
EFFECTS OF DIRECT AND INDIRECT RANGE
RESTRICTION ON CORRELATIONS
Direct Range Restriction
(Sr= 20%)
Unrestricted Correlation
Rho=.60
Rxx =.90
Ryy =.52
R = .22
Indirect Range Restriction
(Sr= 20% on Z; Rxz=.66; Ryz=.30)
R = .41
R = .32
CORRECTION FOR RANGE RESTRICTION

Direct Range Restriction: Thorndike Case II
RXY 

U X rXY
2
1  (U X2  1)rXY
Indirect Range Restriction: Thorndike Case III
R XY 
rXY  rXZ rYZ (U Z2  1)
2
1  (U Z2  1)rXZ
1  (U Z2  1)rYZ2
Notes:
Ux = 1/uX; uX = sdx/SDx = Range restriction ratio of X (the ratio of standard deviation of the
independent variable X in the restricted sample to its standard deviation in the unrestricted
population).
UZ = 1/uZ ; uZ = Range restriction ratio of Z (the third variable where explicit selection occurs).
CORRECTION FOR RANGE RESTRICTION
•
Problems related to correcting for the effect of range
restriction in Meta-Analysis:
–
Most studies are affected by indirect range restriction.
–
However, information required to correct for this effect of
indirect range restriction (shown in the previous) is often
not available.
–
The problem is even worst for meta-analysts who have to
rely on information reported by primary researchers.
NEW RANGE RESTRICTION CORRECTION METHODS
•
•
Recently, Hunter, Schmidt, and Le (2006) introduced a new
procedure (CASE IV) to correct for range restriction.
The procedure requires information about:
uT: Range restriction ration on the true score T underlying X
– Rxxa : Reliability of X estimated in the unrestricted population.
–
•
•
Simulation study shows that the method is accurate (Le &
Schmidt, 2006), outperforming traditional approach of using direct
range restriction correction (when range restriction is actually
indirect) in most situations.
Using this procedure, the researchers showed that traditional
estimates of the validity of the GATB were underestimated from
24% - 45%!
NEW RANGE RESTRICTION CORRECTION METHODS
Hunter, Schmidt, & Le (2006) model for the combined effects
of indirect range restriction and measurement error:
uS
S
RTP
uT
T
P
RTX=(RXX )1/2
uX
X
RXY
Y
New Method for Range Restriction (Case IV)
Two key characteristics:
Before applying Thorndike’s
Case II,
+ Ut (instead of Ux)
+ Correction for measurement
error before RR correction
+ Applying Thorndike’s Case II
+ Reintroducing unreliability
in predictor to estimate
true validity
APPLYING THE NEW CORRECTION
APPROACH TO META-ANALYSIS



Problem: The information needed to apply the
new procedure is not available in every primary
study.
In the past, meta-analysts addressed that
problem by using artifact distributions.
This approach allows corrections to be made even
when information of the artifacts is not available
in each primary study.
DIFFICULTIES IN ESTIMATING THE UT
DISTRIBUTION
Problem when applying the artifact distribution
approach to correct for indirect range restriction in
meta-analysis:

Need the uT artifact distribution but uT is unknown
(unobservable – unlike uX)!

Hunter et al. (2006) suggested uT be estimated from uX and
Rxx (reliability of the independent variable in the
unrestricted population) using the formula:
uT 
u
2
X

 (1  RXX a ) / RXX a
THE ARTIFACT DISTRIBUTION OF UT
Individual studies
Study 1:
Study 2:
Study 3:
Study 4:
…………
Rxxa1
Rxxa2
........
Ryy4
…….
Study k:
Rxxak
Ryyk
Very Rare Case! Less representative! Dependent
on Rxxa! Sometimes, cannot be computed even
when Rxxa and Ux are simultaneously available
uX1
uT1
……
……
…..
rxy1
rxy2
rxy3
rxy4
……
uXk
uTk
Nk
rxyk
uX
uT
uX3
Artifact Distributions
of
Rxx
Ryy
N1
N2
N3
N4
DIFFICULTIES IN ESTIMATING THE UT
DISTRIBUTION

Doing so, however, renders the resulting distribution
of uT is highly dependent to Rxxa  the assumption of
independence of the artifact distributions is violated.

Further, there are values in the distributions of uX
and Rxxa which cannot be combined. For example,
when uX = .56 (equivalent to selection ratio of 40%)
and Rxxa =.60, we cannot estimate the corresponding
value of uT . Current practice is to disregard these
values.
 The uT distribution estimated by the current approach may not be
appropriate  Meta-analysis results may be affected.
ESTIMATING THE UT DISTRIBUTION
•
Our solution:
–
To go backward: Instead of combining values uX and RXX in
their respective distributions to estimate the values of the uT
distribution, we systematically examine the appropriateness
of different “plausible uT distributions” in term of how closely
they can reproduce the original uX distribution when
combined with the RXX distribution.
–
This approach is logically appropriate because uX results from
uT and RXX (see Hunter et al., 2006; Le & Schmidt, 2006), not
the other way around as seemingly suggested by the formula.
ESTIMATING THE UT DISTRIBUTION
•
Procedure: Five steps
(1) Selecting a “plausible distribution” for uT ( uˆT ). This
distribution includes a number of representative values of uT ,
together with their respective frequencies.
(2) The values of uˆT are then combined with all the values of Rxx in
its distribution using the following equation to calculate the
corresponding uX values (equation 8, p. 422, Le & Schmidt,
2006):
uˆ X  RXX a uˆT2  RXX a  1
(3) The resulting values of uX form a distribution with frequency of
each value being the product of the corresponding frequencies
of uˆT and Rxx in their respective distributions.
ESTIMATING THE UT DISTRIBUTION
•
Procedure: (cont.)
(4) This uX distribution is then compared to the observed
(original) distribution of uX, based on a pre-determined
criterion. If they are close enough, as determined by the
criterion, the process terminates and the “plausible
distribution” of uˆT specified in step (1) becomes the
estimated uX distribution. Otherwise, the process
continues in step (5);
(5) A new plausible distribution is constructed by keeping
the original values of uˆT but systematically changing
their frequencies. A new iteration is then started (by
returning to step 2 above).
A SAS program was developed to implement the procedure
(the program is available from Huy Le).
ESTIMATING THE UT DISTRIBUTION

•
Demonstration of the procedure:
Note that the uX distribution for cognitive tests derived by
Alexander et al. (1989) and the Rxxa distribution derived by
Schmidt and Hunter (1977) were used.
RESULT: THE UT DISTRIBUTION FOR
COGNITIVE MEASURES
Alexander et al. (1989)’s Distribution of uX
(Cognitive)
Selection
uX
Frequency
Ratio
1.00
.849
.766
.701
.649
.603
.599
uX
.05
.15
.20
.20
.20
.15
.05
100%
90%
80%
70%
60%
50%
40%
=.718; SDu = .107
X
(Skewness =0.80; Kurtosis = 0.33)
Estimated Distribution of uT
uT
Frequency
Selection
Ratio
1.00
.807
.710
.632
.562
.499
.432
.05
.13
.15
.17
.18
.29
.03
100%
85.8%
71.4%
56.5%
41%
26.6%
13.3%
uT =.628;
SDuT = .139
(Skewness = 0.94; Kurtosis = 0.36)
RESULT: THE UT DISTRIBUTION FOR
EDUCATIONAL TESTS
Alexander et al. (1989)’s Distribution of uX
(Education)
Selection
uX
Frequency
Ratio
.849
.766
.701
.649
.603
.599
uX
.15
.20
.20
.20
.15
.05
90%
80%
70%
60%
50%
40%
=.704; SDu = .084
X
(Skewness =0.28; Kurtosis = -0.84)
Estimated Distribution of uT
uT
Frequency
Selection
Ratio
.807
.710
.632
.562
.499
.432
.11
.15
.23
.23
.23
.05
85.8%
71.4%
56.5%
41%
26.6%
13.3%
uT =.607;
SDuT = .105
(Skewness = 0.43; Kurtosis = 0.69)
DISCUSSION
•
Procedures to correct for indirect range restriction are
necessarily complicated, but the procedure described
in this paper will allow better, more accurate
estimation of the uT distribution  More accurate
meta-analysis results.
•
The uT distributions estimated here can be used by
researchers in their future research.
•
Alternatively, meta-analysts can apply the current
procedure to any situations where there are only
sparse information about range restriction and
reliabilities in their data (i.e., primary studies).
THANK YOU!
 Any
Questions or Comments?