Transcript A Blind Analysis - Department of Physics & Astronomy

A Blind Analysis
You are not allowed to peek!
Prof. Donald Koetke
Senior Research Professor of Physics
Valparaiso University
Outline
1. What is a “blind analysis”?
•
What is a “bias”?
•
Does “bias” = “systematic error”?
•
What is the special (subtle) bias?
2. What is the history of the blind analysis”?
3. What are examples of “blind analysis” in
physics, astronomy, astrophysics, and, … ?
4. Some thoughts and reflections -3/30/07
VU Colloquium
2
Outline
1. What is a “blind analysis”?
•
What is a “bias”?
•
Does “bias” = “systematic error”?
•
What is the special (subtle) bias?
2. What is the history of the blind analysis”?
3. What are examples of “blind analysis” in
physics, astronomy, astrophysics, and, … ?
4. Some thoughts and reflections -3/30/07
VU Colloquium
3
Outline
1. What is a “blind analysis”?
•
What is a “bias”?
•
Does “bias” = “systematic error”?
•
What is the special (subtle) bias?
2. What is the history of the blind analysis”?
3. What are examples of “blind analysis” in
physics, astronomy, astrophysics, and, … ?
4. Some thoughts and reflections -3/30/07
VU Colloquium
4
Outline
1. What is a “blind analysis”?
•
What is a “bias”?
•
Does “bias” = “systematic error”?
•
What is the special (subtle) bias?
2. What is the history of the blind analysis”?
3. What are examples of “blind analysis” in
physics, astronomy, astrophysics, and, … ?
4. Some thoughts and reflections -3/30/07
VU Colloquium
5
Outline
1. What is a “blind analysis”?
•
What is a “bias”?
•
Does “bias” = “systematic error”?
•
What is the special (subtle) bias?
2. What is the history of the blind analysis”?
3. What are examples of “blind analysis” in
physics, astronomy, astrophysics, and, … ?
4. Some thoughts and reflections -3/30/07
VU Colloquium
6
Outline
1. What is a “blind analysis”?
•
What is a “bias”?
•
Does “bias” = “systematic error”?
•
What is the special (subtle) bias?
2. What is the history of the blind analysis”?
3. What are examples of “blind analysis” in
physics, astronomy, astrophysics, and, … ?
4. Some thoughts and reflections -3/30/07
VU Colloquium
7
A Blind Analysis
What is a “bias”?
“A prejudice in favor of or against” -- may be
due to computer codes, equipment performance
or setting, decisions/selections/cuts imposed, etc.
Does “bias” = “systematic error”?
Yes - if it causes a systematic shift in a result
(This is not a “mistake” or “blunder”.)
What is the special (subtle) bias?
Experimenter bias in making decisions/choices to
achieve a desired answer. Examples …
3/30/07
VU Colloquium
8
A Blind Analysis
You do an experiment and…
1. You know what the answer “should be” i.e., you know what the “accepted”
answer is -- but your answer is different outside of errors.
What do you do?
2. You make a measurement and within errors
it agrees with the “accepted” answer -What do you do?
3/30/07
VU Colloquium
9
A Blind Analysis
The goal of a “blind analysis” is to prevent
the experimenter from (unconsciously or consciously) making decisions in the
analysis that would affect the result based on:
• The predictions of a model or theory (e.g., the
Standard Model for particle physics)
• Previous measurements known to the experimenter
• The experimenter’s intuition or other predisposition
These are all examples of personal bias
3/30/07
VU Colloquium
10
A Blind Analysis
1. What is a “blind analysis”?
A “blind analysis” is an analysis of measured
data in which the final answer is kept hidden from the
experimenters until all of the decisions about the
analysis have been made:
• Computer codes have been developed and tested
• Decisions about the number of events (trials) needed
have been made (cuts have been selected)
• Apriori agreements are reached about what to do
when the real answers from the experiment are
revealed; no further analysis!
3/30/07
VU Colloquium
11
Outline
1. What is a “blind analysis”?
•
What is a “bias”?
•
Does “bias” = “systematic error”?
•
What is the special (subtle) bias?
2. What is the history of the blind analysis”?
3. What are examples of “blind analysis” in
physics, astronomy, astrophysics, and, … ?
4. Some thoughts and reflections -3/30/07
VU Colloquium
12
History
Blind analysis begins in ~1930s with -
Medical research -- blind tests!
Patients don’t know whether they are getting -a) the medicine/treatment, or,
b) a placebo
Patients are assigned to (a) or (b) ramdomly
Therefore - the patients are “blind” to their treatment
They may imagine symptoms or cure, etc., but only the
researcher/physician knows if these can be due to the
medicine/treatment.
3/30/07
VU Colloquium
13
History
Blind analysis begins in ~1930s with -
Medical research -- blind tests!
•
Patients talk to medical researcher
e.g., how are you feeling? is pain less or more?
are the other problems? serious or not so serious?
before or after you take the medicine? …etc.
•
Researcher hears what patient says
•
Researcher examines the patient
•
Researcher records all this as “data”
Do you see a
problem here?
c.f.,1937 JAMA 26 June 2178/2.
3/30/07
VU Colloquium
14
History
Blind analysis begins in ~1930s with -
Medical research -- blind tests!
The patients are “blind” to their treatment - but,
the researcher is NOT!
•
Therefore, the medical researcher can (and will)
unconsciously and unintentionally interpret and record
the information (data) with this bias (knowledge).
The solution is…
3/30/07
VU Colloquium
15
History
Blind analysis begins in ~1930s with -
Medical research -- The double blind test!
The patients are “blind” to their treatment - and,
the researcher is “blind” to who is getting treated!
•
The list of patients and their treatment is prepared and
maintained by someone who is NOT participating in the
research. The list is sealed in the “black box”.
•
The list can be retrieved from the “black box” only
after all the analysis is completed.
1948 Am. Heart Jrnl., XXXVI, 529. 1950 Am. Jrnl. Med., IX, 142/1.
3/30/07
VU Colloquium
16
History
Blind analysis begins in ~1930s with -
Medical research -- The double blind test!
Consider a simple analysis -- “Was the medicine effective?”
• The data on all patients will indicate whether the medication was
effective for each patient.
• When this data analysis is complete (and free from mistakes)
and any disagreement among the researchers have been
settled and any concerns about bias have been removed,
i.e., then (and only then) • The “black box” can be opened and the list can be retrieved.
• The data on all patients is now grouped into two groups:
group (a) and group (b)
3/30/07
VU Colloquium
17
History
Blind analysis begins in ~1930s with -
Medical research -- The double blind test!
The simple analysis -- “Was the medicine effective?”
• For what fraction of group (a) [fa] and group (b) [fb] was the
medicine effective?
• Do a statistical analysis to determine whether the difference in
these two fractions is statistically significant - or whether it is
consistent with a random occurrence. The result is now known!
• The researchers agree to accept the results obtained and no
further analysis of the data is permitted - unless there has
been a blunder (mistake) - in which case fix the blunder and
report both results and the nature of the “fix”.
3/30/07
VU Colloquium
18
History
Blind analysis has become the standard
methodology in clinical trials.
Blind analysis has been used in the physical
sciences only in recent years.
Physicists (and astronomers) are, of course --- Careful
-- Quantitative
-- Attentive to bias (systemtic errors)
Why should they need a blind analysis?
3/30/07
VU Colloquium
19
History
Ernest Rutherford (1934) -“It seems to me that in some way it is regrettable
that we had a theory of the positive electron before
the beginning of the experiments. Blackett* did
everything possible not to be influenced by the
theory, but the way of anticipating results must
inevitably be influenced to some extent by the
theory. I would have liked it better if the theory had
arrived after the experimental facts had been
established.” Ernest Rutherford, Proc. Solvay Conference,
(Gauthier-Villars, Paris 1934), p 177.
* Nobel prize in physics for discovering the positron (1948)
3/30/07
VU Colloquium
20
History
Gregor Mendel (1865) --
The classic case is Gregor Mendel’s
work on inheritance.
In every case the data agreed with the theoretical
ratios within less than the standard errors.
Taking the whole together, 2 was 41.6 on
84 degrees of freedom, giving a probability that he
would have measured this well to be only 7x10-5 !!
3/30/07
VU Colloquium
21
History
Modern examples --
From nuclear and subnuclear physics --Reason: It’s the field in which the blind analysis
techniques have been widely used - and It is the field with which I am most familiar.
3/30/07
VU Colloquium
22
History
3/30/07
VU Colloquium
23
History
3/30/07
VU Colloquium
24
History
B-meson lifetime ratio
8 Major experiments
Over 2000 physicists
Bottom line:
The agreement
appears to be
too good!!
The mean ratio has a 2 of 4.5 for 13 degrees of freedom; P=0.985
3/30/07
VU Colloquium
25
Outline
1. What is a “blind analysis”?
•
What is a “bias”?
•
Does “bias” = “systematic error”?
•
What is the special (subtle) bias?
2. What is the history of the blind analysis”?
3. What are examples of “blind analysis” in
physics, astronomy, astrophysics, and, … ?
4. Some thoughts and reflections -3/30/07
VU Colloquium
26
An example of a “search” for an uncommon occurrence
A reaction that violates the Standard Model
conservation of lepton number
If you do not find the reaction, you have not shown the
Standard Model to be incorrect
If you do find the reaction, you had better be very sure
that you have got it right!!!
3/30/07
VU Colloquium
27
Examples
MEGA
Search for   e
Signal: Ee = E = 52.8 MeV
Boxes represent 2 boundaries
Question: When and How are the
values of  determined?
3/30/07
VU Colloquium
28
Examples
MEGA
Search for   e
Signal: Ee = E = 52.8 MeV
Boxes represent 2 boundaries
Blind Analysis: The boundaries are
determined before the analysis of
(most of) the data!
3/30/07
VU Colloquium
29
3/30/07
VU Colloquium
30
Examples
p = 29.8 MeV/c
TWIST
Measure: pe()
High precision   ee
A search for physics beyond (not included in)
the Standard Model.
3/30/07
VU Colloquium
31
Examples
Number
Find a’ & b’ by
comparison of
real data
with
simulated
data
The detector system
distorts the distribution
Analyzed
expimental
data
Nexp(x) = a’ + b’x
N(x) = a + bx
a,b from
theory
X = E/Emax
3/30/07
VU Colloquium
Simulation of
the experiment
using a,b
Simulation = “Monte Carlo”
32
Examples
Muon decay spectrum
d 2
2
x0

(
3

3
x
)


(
4
x

3
)

3

(1  x)
2
x dxd(cos )
3
x
2


 P cos (1  x )   (4 x  3)
3


Current
 =
 =
 =
P
3/30/07
-0.007 ± 0.013
0.7518 ± 0.0026
0.7486 ± 0.0026 ± 0.0028
= 1.0027 ± 0.0079 ± 0.0030
VU Colloquium
SM
0
3/4
3/4
1
33
Examples
Muon decay spectrum from Standard Model
3/30/07
VU Colloquium
34
Examples
TWIST measured spectrum
Standard Model spectrum
Monte Carlo computer code
,,
,
3/30/07
VU Colloquium
35
Examples
Compare spectra  ’,’,’,’
TWIST measured spectrum
TWIST simulated spectrum
’,’,
’,’
Problem!
,,
,
This spectrum is the S.M.
spectrum -- nothing is
hidden; the experimenter
is not “blinded”.
3/30/07
VU Colloquium
36
Examples
TWIST measured spectrum
Standard
Unknown
Model
spectrum
spectrum
Monte Carlo computer code
o,o,o,o are generated
randomly, are encrypted,
stored secretly, and used to
generate the simulated
data. Nobody knows what
the offsets from ,,, are.
3/30/07
VU Colloquium
o,o,
o,o
37
Examples
Compare spectra to get  , , , 
TWIST measured spectrum
’,’,
’,’
3/30/07
TWIST simulated spectrum
 = ’ - o
 = ’ - o
 = ’ - o
 = ’ - o
VU Colloquium
?
38
Examples
Before you can open the black box:
o,o,
o,o
• Get all computer codes working and tested
• Identify all sources of systematic error and evaluate
the size of each one
• Take all of the data you will need including data
to help estimate the systematic errors
• Be sure that the Monte Carlo programs accurately
simulate your experiment or you will have false values
• Analyze all of the data you intend to use to get the result
3/30/07
VU Colloquium
39
Examples
o,o,
o,o
 = ’ - o
 = ’ - o
 = ’ - o
 = ’ - o
Measure
Calculate the results
Compare with Standard Model predictions
Write the paper
3/30/07
VU Colloquium
40
Examples
Let’s look at one of your experiments
from PHYS-245
3/30/07
VU Colloquium
41
Examples
A measurement of the speed of light

2
T
Present accepted value:
c = 299,792,458 m/s
c  3 x 108 m/s
1


t
T
2
osc
D
 c
t
D

3/30/07
VU Colloquium
42
Examples
Blinded
A measurement of the speed of light

2
T
Present accepted value:
c = 299,792,458 m/s
c  3 x 108 m/s
1

T o  T  T
 
D
T
c
t
To  co 
2
t
osc
D

3/30/07

VU Colloquium

43
Outline
1. What is a “blind analysis”?
•
What is a “bias”?
•
Does “bias” = “systematic error”?
•
What is the special (subtle) bias?
2. What is the history of the blind analysis”?
3. What are examples of “blind analysis” in
physics, astronomy, astrophysics, and, … ?
4. Some thoughts and reflections -3/30/07
VU Colloquium
44
Reflections
1. Guide to a blind analysis: If my answer
were to come out to be six standard
deviations from the expected result, what would I do?
Make the list, and then do all of that before you look at the answer!
2. A blind analysis is intended to guard against
experimenter bias. It will not guard against fraud;
that requires integrity and honesty.
3. A blind analysis is likely not the fastest way to
an answer.
3/30/07
VU Colloquium
45
Reflections
4. It is not always possible to achieve perfect
“blindness” -- e.g., drug testing.
5. Don’t need to plan everything in the analysis
before beginning; just keep the answer
hidden.
6. A blind analysis may not work for every experiment but it is worth investigating before you begin.
7. In a blind analysis you want to hide the answer
from anyone else who might want to offer advice
that may be based on the answer.
3/30/07
VU Colloquium
46
Reflections
8. A blind analysis removes the
“comfort factor” - knowing what answer
you are getting so you can make changes, do
more analysis, repeat measurements, etc., if
the answer is not what you expect.
Thank
you
But, that may not be the best science!
3/30/07
VU Colloquium
47
3/30/07
VU Colloquium
48