Transcript Comparative Analysis Using DNA Microarrays: Sensitivity

Analysis of DNA Microarray
Data: Sensitivity, Specificity,
and Other Real-World Issues
1. Definitions and basic
considerations
DNA microarrays
 Major advantage

Simultaneous measurement of level of
expression for nearly all transcribed genes
within given cell or tissue
 Major disadvantage

Cost
Therefore, to get the most bang
for the buck, it is imperative to
understand the role of
uncertainty in measurement…
Categorical tests
(yes/no, based upon threshold)
 Gene arrays


Is gene expressed or not?
Is gene differentially expressed under two
different experimental conditions?
 Medical tests

Does patient have disease or not?
Key concepts for categorical
tests
 Specificity


true negative rate
1 – FPR (false positive rate)
 Sensitivity

TPR (true positive rate)
Specificity provides the answer to
questions like…
 What fraction of patients who are
disease-free are correctly classified as
disease-free?
 What fraction of genes that are not
differentially expressed are correctly
classified as being non-differentially
expressed?
Specificity
 Specificity is defined as true negative rate

Probability that disease-free patient will be correctly
categorized as disease-free
 False positive rate (FPR) = 1 – specificity

Probability that disease-free patient will be
incorrectly categorized as having disease
Sensitivity and specificity deal with
distinct sets of patients or genes
 Specificity



Healthy patients lacking the disease
Non-expressed genes
Non-differentially expressed genes
 Sensitivity



Sick patients having the disease
Expressed genes
Differentially expressed genes
Sensitivity provides the answer to
questions like…
 What fraction of patients who have a
given disease are correctly classified as
diseased?
What fraction of genes that are
differentially expressed are correctly
classified as being differentially
expressed?
Sensitivity
 Sensitivity is defined as true positive
rate

Probability that diseased patient will be
correctly categorized as having the disease
Yin and yang of sensitivity and
specificity
 Improving specificity always worsens
sensitivity
 Improving sensitivity always worsens
specificity
Since when is the world ever
ideal?
Frequency
0.4
Non-differentially expressed
Diffe rentially e xpresse d
0.3
0.2
0.1
0
-2
0
2
SLR
4
6
If we choose a threshold l of
1.5, then...
Frequency
0.4
Non-differentially expressed
Diffe rentially e xpre sse d
0.3
0.2
0.1
0
-2
0
2
SLR
4
6
And if we choose a threshold l
of 0.5, then...
Frequency
0.4
Non-differentially expressed
Diffe rentially e xpre sse d
0.3
0.2
0.1
0
-2
0
2
SLR
4
6
2. Sources of uncertainty in
categorical tests
SMEASURE  STRUE N
SMEASURE = measured signal
STRUE
= true signal
N
= noise (error)
Noise-to-Signal (N:S) Ratio
 N : S << 1

reliable and trustworthy measurement
N~S

unreliable measurement
N>S

highly unreliable measurement
Sources of uncertainty in
categorical measurements
 Measurement uncertainty


SMEASURE does not necessarily equal STRUE
N ~ S or N > S
 “Overlap” uncertainty


Some patients with disease truly have positive test
values
Some patients without disease truly have negative
test values
Gene arrays and medical tests
have distinct and different
sources of uncertainty
Variability in medical tests is
mostly “overlap”
 Measurement variability



Essentially none (error is of no clinical significance)
N : S << 1
Hence, perform test once and only once
 “Overlap” variability


Ubiquitous and essentially unavoidable
Feature of all medical tests to one degree or another
 So what’s the solution?

Search for a better test
Variability in DNA microarrays is
mostly measurement uncertainty
 Measurement variability


Ever-present
N > S for many genes
 “Overlap” variability



None
Absent gene has expression level of zero, whereas
present gene has expression level of non-zero
Differentially expressed gene…
 So what’s the solution?

Repeated measurements
So how do we improve the
N:S ratio?
Take mean of repeated
measurements...
S MEASURE
1
= [ SMEASURE(1) + ... + SMEASURE(n) ]
n
1
= [( STRUE + N1 ) + ... + ( STRUE + Nn )]
n
= STRUE + N
Benefits of repeated
measurements
 Assuming that noise N has a normal
(Gaussian) distribution, then the error
decreases with square root of number n
of measurements
 Example: to reduce N : S by half, take
mean of 4 measurements
3. Measurements using Affymetrix
(MSV 5.0)
Affymetrix Microsoft Suite
Version 5.0 (MSV 5.0)
Single array Two arrays
(absolute) (comparative)
Qualitative
present vs.
absent
increased vs.
decreased
Quantitative
log signal
signal log ratio
(SLR)
For our analysis, we used...
Single array Two arrays
(absolute) (comparative)
Qualitative
Quantitative
present vs.
absent
signal log ratio
(SLR)
Signal Log Ratio (SLR)
 SLR = logarithm to base 2 of the ratio of the signal
for gene under experimental condition A (SA1) to
that for the same gene under experimental condition
N (SN1)
 SLR A 1N1
SA1
= log2
= log2 SA 1 – log2 SN1
SN1
Examples of SLR
SA1 = 4000
SN1 = 1000
SA1 = 2
SN1 = 16
SLR = log2 (4) = 2
SLR =log2 (1/8) = –3
4. Specificity of MSV 5.0
To get a handle on specificity,
perform same-versus-same
comparisons
 SLRTRUE must be zero

log2 (1) = 0
 Hence, SLRMEASURE is all noise
Perform separate analyses for
“present” and “absent” genes
 Present genes
 N : S << 1
 Absent genes
 N : S ≥ 1
Experimental system
 Primary cultures of peritoneal macrophages from mice
of 3 strains



BALB/c (normal)
MRL/+ (autoimmune lupus)
MRL/lpr (autoimmune lupus)
 Each array represents mRNA pooled from distinct sets
of ~ 6 mice harvested on separate days
 Macrophages were stimulated with bacterial endotoxin
(lipopolysaccharide, LPS) for 8 or 24 hours
Present genes:
same-vs.-same comparison
(single array)
0.5
0.4
Frequency
Expe rimental
BALB/c vs. BALB/c
(n=5141)
Normal Dist
0.3
0.2
0.1
0
-4
-3
-2
-1
0
SLR
1
2
3
4
Present genes:
same-vs.-same comparison
(single array)
0.5
Frequency
0.4
MRL/+ vs. MRL/+
(n=5102)
Expe rimental
Normal Dist
0.3
0.2
0.1
0
-4
-3
-2
-1
0
SLR
1
2
3
4
Present genes:
same-vs.-same comparison
(single array)
0.5
0.4
Frequency
Expe rimental
MRL/lpr vs. MRL/lpr
(n=5222)
Normal Dist
0.3
0.2
0.1
0
-4
-3
-2
-1
0
SLR
1
2
3
4
T able 1. Experimental dist ribut ions for "present " versus "absent " genes in single array-based
same-versus-same comparisons.
"PRESENT GENES"
No. genes
Mean SLR
 N

BALB/c vs. BALB/c
4022
–0.1
1.1
5133
0.0
0.3
4118
–0.1
1.1
5141
0.0
0.6
5140
0.0
0.9
5461
0.0
0.3
5040
0.0
1.0
5628
0.0
0.6
MRL/+ vs. MRL/+
MRL/lpr vs. M RL/lpr
Mean ± SD
4512
5104
0.0
0.0
0.9
0.6
5224
5364
0.0
0.0
0.5
0.4
4990 ± 507
0.02 ± 0.04*†
0.69 ± 0.30§
Present genes:
Same-vs.-same comparison (single array)
 Average SLR = ~ 0.02 + 0.04 (~ 1.014-fold)


not different from zero
that’s good!
 Standard deviation = ~ 0.69 + 0.30



~ 32% genes have SLR > 0.69 (1.61-fold induction)
~ 4% genes have SLR > 1.38 (2.60-fold induction)
that’s not good
Present genes:
Statistical distribution of SLR
 Entire distribution

Not normal (p < 0.01, by D statistic)
 Central 95%



Normal (p > 0.2, by D statistic)
Highly noteworthy, since D statistic detects
tiny tiny deviations from normality
5% at tails overestimate the SLR
Present genes:
same-vs.-same comparison
(single array)
0.5
Frequency
0.4
MRL/+ vs. MRL/+
(n=5102)
Expe rimental
Normal Dist
0.3
0.2
0.1
0
-4
-3
-2
-1
0
SLR
1
2
3
4
If we compare genes in central
95% versus genes in 5% tails…
 Center (95% genes)

Mean signal intensity = 1493
 Tails (5% genes)

Mean signal intensity = 620 (p < 10-19, t-test)
• Consistent with intuitive idea that measurement
variability is inversely related to level of gene’s
expression
Absent genes:
same-vs.-same comparison
(single array)
0.5
BALB/c vs. BALB/c
(n=7347)
Frequency
0.4
Expe rimental
Normal Dist
0.3
0.2
0.1
0
-4
-3
-2
-1
0
SLR
1
2
3
4
Absent genes:
same-vs.-same comparison
(single array)
0.5
MRL/+ vs. MRL/+
(n=7385)
Frequency
0.4
Expe rimental
Normal Dist
0.3
0.2
0.1
0
-4
-3
-2
-1
0
SLR
1
2
3
4
Absent genes:
same-vs.-same comparison
(single array)
0.6
MRL/lpr vs. MRL/lpr
(7264)
Frequency
0.5
Expe rimental
Normal Dist
0.4
0.3
0.2
0.1
0
-4
-3
-2
-1
0
SLR
1
2
3
4
T able 1. Experimental dist ribut ions for "present " versus "absent " genes in single array-based
same-versus-same comparisons.
"ABSENT GENES"
No. genes
Mean SLR
 N

BALB/c vs. BALB/c
8466
+0.9
1.4
7355
0.0
0.8
8370
+0.9
1.3
7347
+0.1
1.1
7348
+0.5
1.4
7027
+0.0
0.8
7448
+0.4
1.3
6860
+0.1
1.1
MRL/+ vs. MRL/+
MRL/lpr vs. MRL/lpr
Mean ± SD
7976
7384
+0.2
+0.1
1.3
1.2
7264
7124
+0.4
+0.4
0.9
0.8
7497 ± 507
0.33 ± 0.31*‡
1.12 ± 0.24§
Absent genes:
Same-vs.-same comparison (single array)
 Average SLR = ~ 0.33 + 0.31 (~ 1.26-fold
induction)

definitely not good
 Standard deviation = ~ 1.12 + 0.24



> 35% genes have SLR > 1.0 (2-fold induction)
> 5% genes have SLR > 2.0 (4-fold induction)
even worse!
Absent genes:
Statistical distribution of SLR
 Entire distribution

Not normal (p < 0.01, by D statistic)
 Central 95%

Not normal (p < 0.01, by D statistic)
 Central 60%

Not normal (p < 0.01, by D statistic)
Summary of same-vs.-same
comparisons (single array)
 Use SLR only for genes that are actually
expressed (i.e., “present” genes)


Central 95% normally distributed with standard
deviation of ~ 0.69
2.5% at each tail exceeds normal distribution
 Do not use SLR for genes that are marginally,
if at all, expressed (i.e., “absent” genes)


Most of measured signal is noise
SLR is therefore ratio of two small randomly
distributed values
Specificity of single array
comparisons
SLR
Threshold (l)
"Present"
genes
"Absent"
genes
l = 1.0
FPR = ~14%
FPR = >35%
l = 2.0
FPR = 0.4%
FPR = >5%
5. Sensitivity of MSV 5.0
Experimental evaluation of
sensitivity of gene arrays is not
nearly so simple as specificity
 With same-vs.-same, we had a large set
of equivalently expressed genes whose
SLRTRUE was, by definition, equal to zero
But what to do for differentially
expressed genes?


which genes are they?
what is their SLRTRUE?
Experimental approach to sensitivity
 Step #1: Compare gene expression by macrophages
from BALB/c vs. MRL/lpr mice
 Step #2: Use strategy of recurrence (n > 2) on
“present” genes to determine those that are
differentially expressed at threshold of 0.8 (1.74-fold
induction)


strategy of recurrence is more specific
218 differentially expressed genes identified
 Step #3: Determine the mean measured SLR for each
of these 218 genes
Experimental approach to sensitivity
 Step #4: Group genes according to mean
measured SLR in increments of 0.1
 Step #5: Assess normality of distribution by D
statistic for 3 largest gene groups



Mean measured SLR = -1.5 (n = 11 genes)
Mean measured SLR = -1.6 (n = 10 genes)
Mean measured SLR = +1.2 (n = 10 genes
• using D statistic, none of 3 differed from normality
(p > 0.20)
Experimental approach to sensitivity
 Step #6: Determine standard deviation of the
distribution of measured SLR within each
mean measured SLR group

running average (e.g., standard deviation for mean
measured SLR of 1.4 combines genes whose mean
measured SLR was 1.3, 1.4, or 1.5)
Mean variability (
N )
2
Unsm oot hed
1.5
Moving average
  SLRTRUE = 0
1
0.5
0
-3
-2
-1
0
Mean SLR(MEASURED)
1
2
3
Properties of differentially
expressed genes
 Distribution is also normal (Gaussian)
Standard deviation (SD) appears to
depend on SLRTRUE


SD increases roughly linearly with increasing
SLRTRUE
SD ~ 1.0 for SLRTRUE = 3.0
Sensitivity = TPR
Sensitivity of single arraybased comparisons
SLRTRUE = 1.0
1
SLRTRUE = 1.5
SLRTRUE = 2.0
0.5
SLRTRUE = 2.5
SLRTRUE = 3.0
0
0
1
2
Threshold ( l)
3
Sensitivity as a function of threshold l
for single array comparisons
l = 1.0 (2-fold l = 2.0 (4-fold
induction)
induction)
SLRTRUE = 1.0
TPR = 50%
TPR = <5%
SLRTRUE = 3.0
TPR = ~98%
TPR = >85%
Specificity of single array
comparisons
SLR
Threshold (l)
"Present"
genes
"Absent"
genes
l = 1.0
FPR = ~14%
FPR = >35%
l = 2.0
FPR = 0.4%
FPR = >5%
6. Strategies for combining
multiple measurements
Strategies for combining data
from replicate array-based
comparisons
 Strategy of means


Mean SLR from n individual comparisons
must exceed threshold l
Standard deviation decreases as square root
of number of replicates
 Strategy of recurrence

SLR for all n comparisons must exceed
threshold l
Example
 Set threshold SLR at l = 0.9
 4 replicate measurements:

0.7, 1.1, 1.6, 1.0
 Strategy of means

Mean SLR = 1.1 => include gene
 Strategy of recurrence

1 of the 4 SLR (0.7) does not exceed 0.9 =>
exclude gene
Strategies of recurrence vs. means:
Effect of replicates on specificity
 Strategy of recurrence is more specific than
strategy of means (i.e., fewer false positives)
 Strategy of recurrence: benefit of replicates


Keep multiplying FPR
Example: FPR (n = 4) ~ [ FPR (n = 1) ]4
 Strategy of means: benefit of replicates


Standard deviation reduced by square route of n
Example: SD for FPR (n = 4) ~ 1/2 SD for FPR (n =
1)
False Positive Rates (FPR) using
strategy of recurrence and means
n=2
0
1
2
3
FPR = 1 - specificity
SLR threshold ( l)
1
0.1
0.01
0.001
0.0001
1E-05
1E-06
1E-07
1E-08
n=4
0
1
2
SLR threshold ( l)
3
FPR = 1 – Specificity
1
0.1
0.01
0.001
0.0001
1E-05
1E-06
1E-07
1E-08
Means
FPR = 1 – Specificity
FPR = 1 - specificity
Recurrence
1
0.1
0.01
0.001
0.0001
1E-05
1E-06
1E-07
1E-08
n=2
0
1
2
3
SLR threshold ( l)
1
0.1
0.01
0.001
0.0001
1E-05
1E-06
1E-07
1E-08
n=4
0
1
2
SLR threshold ( l)
3
FPR using strategy of recurrence
or means (threshold l = 1.0)
# Replicates
Recurrence
Means
n=1
FPR = ~ 14.0%
FPR = ~ 14.0%
n=2
FPR = ~ 2.0%
FPR = ~ 4.0%
n=4
FPR = ~ 0.04%
FPR = ~ 0.4%
Sensitivity (TPR) using strategy of
recurrence and means
Means
n =2
1
Sensitivity = TPR
Sensitivity = TPR
Recurrence
0.5
0
0
1
2
3
n=2
1
0.5
0
0
1
Sensitivity = TPR
Sensitivity = TPR
n=4
1
0.5
0
1
2
Threshold ( l)
3
Threshold ( l)
Threshold ( l)
0
2
3
4
n=4
1
0.5
0
0
1
2
Threshold ( l)
3
Strategies of recurrence vs. means:
Effect of replicates on sensitivity
 Strategy of recurrence is more specific than
strategy of means
 BUT strategy of recurrence is also less sensitive
than strategy of means
 The inevitable trade-off between sensitivity
and specificity
The benefit of multiple
measurements…
 By increasing the number of replicates
n, one can achieve the same level of
specificity at a lower threshold l
 By using a lower threshold l, one can
achieve a higher level of sensitivity
7. Receiver-operator characteristic
(ROC) curves
Receiver-operator
characteristic (ROC) curves
 Recall that increasing the threshold l
always increases specificity and decreases
sensitivity
 ROC curves are means of depicting
overall performance of a test as the
threshold l is varied
ROC for single array-based
comparisons
1
TPR
0.75
0.5
SLRTRUE = 1.0
SLRTRUE = 1.5
0.25
SLRT RUE = 2.0
0
0
0.25
0.5
0.75
FPR = 1 – Specificity
1
In an ideal world….
0.4
Frequency
Non-differentially expressed
0.3
Diffe rentially expressed
0.2
0.1
0
-2
0
2
4
SLR
6
8
10
ROC for single array-based
comparisons
1
TPR
0.75
0.5
SLRTRUE = 1.0
SLRTRUE = 1.5
0.25
SLRT RUE = 2.0
0
0
0.25
0.5
0.75
FPR = 1 – Specificity
1
If we choose a threshold l of
1.5, then...
Frequency
0.4
Non-differentially expressed
Diffe rentially e xpre sse d
0.3
0.2
0.1
0
-2
0
2
SLR
4
6
ROC for single array-based
comparisons
1
TPR
0.75
0.5
SLRTRUE = 1.0
SLRTRUE = 1.5
0.25
SLRT RUE = 2.0
0
0
0.25
0.5
0.75
FPR = 1 – Specificity
1
So how do strategies of means and
recurrence compare?
 For any given threshold l, the strategy
of recurrence is more specific than
strategy of means, but it is also
unfortunately less sensitive
 As one increases the number n of
replicates, strategy of recurrence gains
more in specificity, but also loses more in
sensitivity
ROC curves show that strategies
of recurrence and means may
perform nearly equivalently
n =2
1
TPR
0.75
0.5
Recurrence, SLRTRUE = 1.0
Mean, SLRTRUE = 1.0
Recurrence, SLRTRUE = 1.5
0.25
Mean, SLRTRUE = 1.5
0
0
0.25
0.5
0.75
FPR = 1 – Specificity
1
Strategy of means versus strategy
of recurrence
 As assessed by ROC curves, these two
strategies seem to perform more or less
equivalently
 To achieve this equivalence in
performance, one need only choose a
lower threshold l for a strategy of
recurrence
Example: Gene whose SLRTRUE is
1.5 (~3-fold induction) with n = 2
replicates
Sensit ivit y
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
0.95
0.99
n = 2 replicat es
Strat egy of Recurrence
T hreshold l()*
Specificit y
1.55
>0.9999
1.45
>0.9999
1.35
>0.9999
1.30
>0.9999
1.20
>0.9999
1.10
>0.9999
1.00
>0.9999
0.85
0.9993
0.70
0.9968
0.45
0.9640
Strat egy
T hreshold l()
1.75
1.65
1.60
1.50
1.40
1.35
1.25
1.10
1.00
0.80
of Means
Specificit y
>0.9999
>0.9999
>0.9999
>0.9999
>0.9999
>0.9999
>0.9999
0.9998
0.9996
0.9954
8. Strategy of recurrence: Getting
the most bang for your buck
The dark side of gene arrays...
 Gain in certainty by performing
multiple measurements (i.e.,
experimental repeats) is counterbalanced
by high cost of each experiment
 Therefore, imperative that you extract
maximal information from each set of
experiments
The mathematical problem...
 For any given gene, we combine 2
independent measurements (i.e., data
from 2 separate arrays) to obtain a single
comparative index
 Thus, although we use 2n arrays and
therefore perform 2n independent
measurements, we obtain only n
comparative measurements
The mathematical problem…
 So, can one extract additional
independent comparative measurements?
 How many of the n2 possible pair-wise
comparisons are linearly independent?
Using linear algebra, one can
prove that the maximal number of
linearly independent comparative
measurements obtainable from n
individual comparisons (2n
arrays) is equal to 2n–1
Heuristic proof
 Number of linearly independent pair-wise
comparisons cannot exceed number of
independent measurements or variables

therefore, number of independent pair-wise
comparisons can not exceed 2n
 System has one degree of freedom



i.e., our pair-wise comparisons tell us nothing about
absolute level of expression
once fix absolute level of expression for any one of
2n arrays, all others fall out
therefore, 2n – 1 independent pair-wise comparisons
Traditional approach
n=2
N1
N2
Ñ1
Ñ2
{SLRN1Ñ1, SLRN2Ñ2 }
4 arrays, 2 SLR comparisons
Maximal utilization of available
resources
N1
N2
Ñ1
Ñ2
{SLRN1Ñ1, SLRN2Ñ2, SLRN1Ñ2, SLRÑ1N2}
4 arrays, 4 possible SLR comparisons
Which 2n-1 should we use?
2n -1 = 3
N1
N2
Ñ1
Ñ2
{SLRN1Ñ1, SLRN2Ñ2, SLRN1Ñ2, SLRÑ1N2}
To avoid weighting the arrays unequally,
we should use all 2n comparisons
Application within strategy of
recurrence (first approximation)
 Let p denote probability that
SLRMEASURE for a given gene will exceed a
threshold l on a single comparison
 Then the probability that SLRMEASURE
will exceed l on all 2n-1 independent
measurements will be ~ p2n–1
Experimental test for strategy of
recurrence in same-vs.-same
comparisons
500
B
# false positive genes
400
n = 2, experimental
2 n -1 = 3, experimental
300
200
p 2 , predicted
p 3 , predicted
100
0
0.5 0.6 0.7 0.8 0.9 1 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2
Threshold SLR
Nothing is gained within a
strategy of means
 To see this, let us take the following
example:




SLRMEASURE (1) = log2 (4000/1000) = log2 (4) = 2
SLRMEASURE (2) = log2 (5000/2500) = log2 (2) = 1
» mean = 1.5
SLRMEASURE (3) = log2 (4000/2500) = log2 (1.6) = 0.68
SLRMEASURE (4) = log2 (5000/1000) = log2 (5) = 2.32
» mean = 1.5
Experimental lack of benefit
within strategy of means
1000
A
n = 2, strategy of means
#false positive genes
750
2 n -1 = 3, strategy of means
n = 2, strategy of recurrence
500
2 n -1 = 3, strategy of recurrence
250
0
0.5 0.6 0.7 0.8 0.9 1
1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2
Threshold SLR
The following mathematical
features can be proven…
 Assuming you choose wisely the order of
comparisons, then the plot of FPR is
composed of 3 distinct segments



initial n comparisons (i.e., n individual
experiments)
n – 1 additional linearly independent
comparisons
remaining (n – 1)2 non-independent
comparisons
The following mathematical
features can be proven…
 1st two segments are linear (i.e., probability that
SLR will exceed threshold is constant for all
comparisons within each of 1st two segments)

probability of surviving any given comparison within
each of 1st two segments is function solely of the chosen
threshold and the underlying probability distribution
of SLR

probability of surviving any given comparison is
greater for 2nd segment (i.e., comparisons in 2nd
segment are less effective in eliminating FPR)
Can prove following mathematical
features
 3rd segment


not linear
comparisons in this segment much less effective
in eliminating FPR
Heuristic proof of these features
 1st segment


each of 2n arrays used one and only one time
n independent experiments, so if probability of surviving one
experiment is p, then probability of surviving n experiments is
pn
 2nd segment



distributions of SLR now constrained by fact that gene has
survived one comparison (probability of survival q > p)
choose additional n – 1 comparisons carefully, so each array is
used a 2nd time only
linear independence assures probabilistic independence of next
n – 1 comparisons, qn - 1
 3rd segment

Because of linear dependence, every additional comparison
further constrains the underlying SLR distributions (ripple
effect)
Some caveats for using additional 2n – 1
linearly independent comparisons within a
strategy of recurrence
 Statistical algorithms of Affymetrix MSV 5.0
are only approximately linear

close enough not to be a problem
 Implicitly assumed that across-experiment
noise (different days) is small compared to
within-experiment noise (biologic and technical
variability)

but we assume this all the time anyway
Some caveats for using additional 2n – 1
linearly independent comparisons within a
strategy of recurrence
 Must choose additional n – 1 comparisons
wisely, so the complete set of 2n – 1
comparisons is linearly independent


Each array should be used only twice
One possibility (if comparing Normal N vs.
Autoimmune A would be: N1 vs. A2, N2 vs. A3, …., Nn
-1 vs. An
 Use of the remaining (n – 1)2 non-independent
comparisons can reduce the FPR yet further
Some caveats for using additional 2n – 1
linearly independent comparisons within a
strategy of recurrence
 Best also to perform same-versus-same
comparisons corresponding to additional
independent comparisons



for example, if comparing normal N3 from 3rd
experiment against A1 from 1st
then perform N1 vs. N3 and A1 vs. A3
do not use N3 vs. A1, if these same-vs.-same
comparisons have disproportionately increased SD
or SLRMEAN  0
Strategy of recurrence versus
strategy of means: Final verdict
 For an equal number n of experimental
replicates, as judged by ROC curves, the two
strategies seem to perform fairly equivalently


strategy of recurrence achieves this equivalence at a
lower threshold l
NOTE: we’ve shown this only for the case where the
distribution of SLR is normal
 The advantage of a strategy of recurrence is
that one obtains an additional n – 1
independent comparisons
9. The biological problem
Murine Models of Lupus
MRL/+, MRL/lpr
NZW, NZB, NZB/W F1
BXSB
LG
-/gld, -/lpr
IMMUNOLOGIC
ABNORMALITIES IN
AUTOIMMUNE DISEASE
?
Manifestation of Disease
?
Predisposing Factor
SUMMARY OF IL -1 DATA
Under expression of IL - 1 apparent by:
1. BIOSSAY:
Secreted IL - 1
Intracellular IL - 1
Membrane-associated IL - 1
2. WESTERN:
Total cellular IL - 1
Total cellular IL - 1
3. NORTHERN:
IL - 1 mRNA
IL - 1 mRNA
NORMAL STRAINS
A/J
AKR/J
A. Thy
B. 10
B. 10A
B. 10BR
BALB/c
C3HeB/FeJ
C3H/HeN
C57BL/6
CBA/J
DBA/2J
SWR
Serum-induced changed in m cytokine mRNA
Down regulated
Up regulated
Unchanged
IL-1
IL-1
IL-6
IL-12, p35
IL-12, p40
GM-CSF
MIP-1
RANTES
TNF-
IL-10
M-CSF
MIF
TGF-1
TGF-2
TGF-3
10. Application of arrays to
biology
The question…
 Are there other genes in autoimmune
mice that show a similar dysregulation to
that of IL-1 and other cytokines?


Normally expressed in absence of serum
(FBS)
Abnormally expressed in presence of serum
(FBS)
Experimental system
 Primary cultures of peritoneal macrophages from mice
of 3 strains



BALB/c (normal)
MRL/+ (autoimmune lupus)
MRL/lpr (autoimmune lupus)
 Each array represents mRNA pooled from distinct sets
of ~ 6 mice harvested on separate days
 Macrophages were stimulated with bacterial endotoxin
(lipopolysaccharide, LPS) for 8 or 24 hours
T able1. Microarraycom parisons.
Strains
com parisons*
MRL/lprvs. BALB/c
+/– FBS
DurationLP S stim ulat ion
Number
+
+
–
–
8h
24 h
8h
24 h
2
2
2
2
MRL/+ vs. BALB/c
+
+
8h
24 h
2
2
BALB/c vs. BALB/c
+
+
8h
24 h
2
2
MRL/+ vs. MRL/+
+
+
8h
24 h
1
1
MRL/lprvs. MRL/lpr
+
+
8h
24 h
1
1
of
Detection of differentially
expressed genes
 Threshold SLR of 0.8 (1.74-fold
induction) within strategy of recurrence
(n = 2)


Specificity ~99.9% (FPR ~ 0.5-1.0 gene per
1000)
Sensitivity depends on true SLR of gene, and
ranges from 0.33 (SLR of 1.0, 2-fold
induction) to >0.96 (SLR of 2.0, 4-fold
induction)
within
Expt.: #1
across
#2
Expt.: #1
#2
MRL/+
MRL/+
MRL/+
MRL/+
BALB/c
BALB/c
BALB/c
BALB/c
4684
“present”
genes
“present”
genes
3833
4777
intersection
4052
intersection
3456
“present”
genes
3671
469
differentially
expressed genes
(n=2 comparisons)
661
intersection
217
217 differentially
expressed genes
(n=3 linearly independent comparisons)
T able 2. Summ ary of the result s of microarray-based comparison of gene expression.
"Within" comparisons*
Comparison
P resent
genes†
MRL/+ vs. BALB/c
R.10, 8 h
MRL/+ vs. BALB/c
R.10, 24 h
MRL/lpr vs. BALB/c
R.10, 8 h
MRL/lpr vs. BALB/c
R.10, 24 h
MRL/lpr vs. BALB/c
R.0, 8 h
MRL/lpr vs. BALB/c
R.0, 24 h
BALB/c vs. BA LB/c
R.10, 8 h
BALB/c vs. BA LB/c
R.10, 24 h
MRL vs. MRL,
R.10, 8 h
MRl vs. M RL,
R.10, 24 h
4684
3833
5217
4984
5063
5413
5246
5709
5257
5523
5289
5830
4022
5133
5140
5461
4512
5224
5104
5364
In ters ecti o‡n
Differential
expression §
3456
469
4444
244
4831
208
5064
225
5006
193
5138
162
3813
62
4686
49
4087
136
4626
51
"Across" com parisons*
P resent
genes†
4777
4052
4820
5637
5199
5424
5640
5428
5416
5409
5702
5534
4118
5141
5040
5628
4782
4991
5298
5332
Intersection
of "within"
& "across"
differential
expression ¶
In ters ecti o‡n
Differential
expression §
3671
661
217
4563
366
109
4936
189
131
5161
199
151
5031
202
149
5231
210
121
3935
259
11
4757
293
9
4226
494
36
4778
310
8
Union of 8
& 24 h
differential
expression 
270
223
201
18
42
* "Within"com parisonsrefer t o com parisonsbet ween RNA samples obt ainedfrom m cultured on t he same day, whereas "across"
com parisonsrefer t o com parisonsbet weenRNA samples obt ainedfrom m culturedon differentdays as part sof replicat eexperiments.
† As det erminedby AffymetrixMSV 5.0.
‡ Refers t o t hosegenes which were presentin bot h replicat eexperiments.
§ Refers t o t hosegenes whose SLR • 0.8 on bot h com parisonsor whose SLR Š 0.8 on bot h com parisons.
¶ Refers t o t hosegenes t hatfulfilledt he criteriafor differentialexpressionin bot h t he "wit hin"and t he "across" com parisons.
 Refers t o t hosegenes that were differentiallyexpressedat 8 h and/or 24 h.
T able 3. Mean SLR magnit ude for different ially expressed genes.
Different ially expressed gene set
Comparison
MRL/+ vs. BALB/c
R.10, 8 h
MRL/+ vs. BALB/c
R.10, 24 h
MRL/lpr vs. BALB/c
R.10, 8 h
MRL/lpr vs. BALB/c
R.10, 24 h
Mean SLR magnitude for each differentially expressed gene set in
same-vs-same com parisons
BALB/c vs. BALB/c, BALB/c vs. BALB/c,
MRL vs. MRL
MRL vs. MRL
8h
24 h
8h
24 h
No.
genes
Mean SLR
magnitude
217
2.1± 1.2
0.3 ± 0.3
0.4 ± 0.3
0.5 ± 0.6
0.3 ± 0.2
109
2.0 ± 1.3
0.4 ± 0.4
0.4 ± 0.4
0.6 ± 0.6
0.3 ± 0.2
131
1.9 ± 1.3
0.4 ± 0.3
0.4 ± 0.4
0.6 ± 0.6
0.3 ± 0.3
151
1.7 ± 1.1
0.5 ± 0.4
0.5 ± 0.4
0.7 ± 0.7
0.3 ± 0.3
Expected FPR
 We expect FPR of 0.5-1.0 per 1000 genes
 ~ 5000 genes were expressed or
‘present’

therefore, 2.5-5 genes per comparison are
FPR (out of 100-200 genes)
 Same-vs.-same comparisons gave FPR
about twice expected
Summary of differentially
expressed genes
 Differentially expressed only in presence of
FBS

280 genes
 Differentially expressed only in absence of FBS

80 genes
 Differentially expressed both in presence and
in absence of FBS

112 genes
3
FBS on l y
SLR (FBS) / SLR (FBS-free)
FBS & FBS -fre e
FBS -free on l y
2
1
0
8h
24 h
Dur ation of LPS Stim ulation
Global expression patterns of
genes differentially expressed only
in presence of FBS
5
MRL/lpr vs. BALB/c
+ FBS
4
3
SLR, 24 h
2
1
0
-1
-2
-3
-4
-5
mean distance = 1.6
-5
-4
-3
-2
-1
0
SLR, 8 h
1
2
3
4
5
5
MRL/lpr vs. BALB/c
FBS-free
4
3
SLR, 24 h
2
1
0
-1
-2
-3
-4
-5
mean distance = 0.8
-5
-4
-3
-2
-1
0
SLR, 8 h
1
2
3
4
5
5
BALB/c vs. BALB/c
+ FBS
4
3
SLR, 24 h
2
1
0
-1
-2
-3
-4
-5
mean distance = 0.7
-5
-4
-3
-2
-1
0
SLR, 8 h
1
2
3
4
5
5
MRL/lpr vs. MRL/lpr
+ FBS
4
3
SLR, 24 h
2
1
0
-1
-2
-3
-4
-5
mean distance = 0.8
-5
-4
-3
-2
-1
0
SLR, 8 h
1
2
3
4
5
5
5
MRL/lpr vs. BALB/c
+ FBS
3
3
2
2
1
1
0
-1
0
-1
-2
-2
-3
-3
-4
-5
-4
mean distance = 1.6
-5
-4
-3
-2
-1
0
1
2
3
4
MRL/lpr vs. BALB/c
FBS-free
4
SLR, 24 h
SLR, 24 h
4
-5
5
mean distance = 0.8
-5
-4
-3
-2
SLR, 8 h
1
2
3
4
5
5
BALB/c vs. BALB/c
+ FBS
4
3
3
2
2
1
1
0
-1
0
-1
-2
-2
-3
-3
-4
-4
mean distance = 0.7
-5
-4
-3
-2
-1
0
SLR, 8 h
1
2
3
4
MRL/lpr vs. MRL/lpr
+ FBS
4
SLR, 24 h
SLR, 24 h
0
SLR, 8 h
5
-5
-1
5
-5
mean distance = 0.8
-5
-4
-3
-2
-1
0
SLR, 8 h
1
2
3
4
5
Global expression pattern of genes
differentially expressed in both presence
and absence of FBS
8
10
MRL/lpr vs. BALB/c
+ FBS
8
MRL/lpr vs. BALB/c
FBS-free
6
6
4
SLR, 24 h
SLR, 24 h
4
2
0
-2
-4
2
0
-2
-4
-6
-6
-8
-10
-10
mean distance = 2.6
-8
-6
-4
-2
0
SLR, 8 h
2
4
6
8
-8
10
mean distance = 2.5
-8
-6
-4
-2
0
SLR, 8 h
2
4
6
8
Acknowledgments
 Angelika Longacre
 L. Ridgway Scott
 My lab personnel and collaborators


Hanli Fan
Joyce Rauch
Jason Koh
 University of Chicago Bioinformatics and
Computational Biology Core Facility


Richard Quigg
Terry Clark
 University of Chicago Functional Genomics Facility



Xinmin Li
Jaejung Kim
Jamie Zhou
Chris Dyanov
Miglena Petkova