Transcript Document
Lecture 8
Read Taylor
Hypothesis Testing
Ch 6 and Section 10.8
Introduction
l l The goal of hypothesis testing is to set up a procedure(s) to allow us to decide if a mathematical model ("theory") is acceptable in light of our experimental observations.
Examples: u Sometimes its easy to tell if the observations agree or disagree with the theory.
u u n n A certain theory says that Columbus Ohio will be destroyed by an earthquake in May 1992.
A certain theory says the sun goes around the earth.
n A certain theory says that anti-particles (e.g. positron) should exist.
Often its not obvious if the outcome of an experiment agrees or disagrees with the expectations.
n A theory predicts that a proton should weigh 1.67x10
-27 kg, you measure 1.65x10
-27 kg.
n A theory predicts that a material should become a superconductor at 300K, you measure 280K.
Often we want to compare the outcomes of two experiments to check if they are consistent.
n Experiment 1 measures proton mass to be 1.67x10
-27 kg, experiment 2 measures 1.62x10
-27 kg.
Types of Tests
l l
Parametric Tests
: compare the values of parameters.
u Example: Does the mass of the proton = mass of the electron?
Non-Parametric Tests
: compare the "shapes" of distributions.
u Example
:
Consider the decay of a neutron. Suppose we have two theories that predict the energy spectrum of the electron emitted in the decay of the neutron (beta decay): Theory 1 predicts n pe (decays to two particles) Theory 2 predicts n pev (decays to three particles, v=neutrino) 1 Richard Kass S07 P416 Lecture 8
n pe n pev n n Both theories might predict the same average energy for the electron.
A parametric test might not be sufficient to distinguish between the two theories.
The shapes of their energy spectrums are quite different: Theory 1: the spectrum for a neutron decaying into two particles (e.g. n p + e).
Theory 2: the spectrum for a neutron decaying into three particles (p + e + ??).
We would like a test that uses our data to differentiate between these two theories.
In previous lectures we have run across the chi-square ( c 2 ) probability distribution and saw that we could use it to decide (subjectively) if our data was described by a certain model.
c 2
i n
1
(
y i
f
(
x i
i
2
,
a
,
(y i i , x i ) are the data points (
n b
...))
2 of them)
f
(
x i , a, b
..) is a function (“model”) that relates
x
and
y
P416 Lecture 8 Richard Kass S07 2
Example: We measure a bunch of data points (x, y ) and we believe there is a linear relationship between x and y:
y=a+bx
If the y’s are described by a Gaussian
then we saw previously that minimizing the c 2 function (or LSQ or MLM methods) gives us an estimate for a and b.
Assume: We have 6 data points. Since we used the 6 data points to find 2 quantities (a, b) we have 4 degrees of freedom (dof). Further, assume that: c 2
i
6 1 (
y i
(
a
i
2
bx i
)) 2 15
What can we say about our hypothesis, the data are described by a straight line?
To answer this question we find (look up) the probability to get c 2 15 for 4 degrees of freedom: P( c 2 15, 4 dof) 0.006
Thus in only 6 of 1000 experiments would we expect to get this result ( a c 2 15) by “chance”.
Since this is a such a small probability we could reject the above hypothesis or we could accept the hypothesis and rationalize it by saying that we were unlucky.
It is up to you to decide what at probability level you will accept/reject the hypothesis.
3 Richard Kass S07 P416 Lecture 8
Confidence Levels (CL)
l An informal definition of a confidence level (CL): CL = 100 x [probability of the event happening by chance] l The 100 in the above formula allows CL's to be expressed as a percent (%).
We can formally write for a continuous probability distribution
p
: CL 100
prob
(
x
1
X
x
2 ) 100
x
1
x
2
p
(
x
)
dx
For a CL we know
p
(x), x 1 , and x 2 l Example: Suppose we measure some quantity (
X
) and we know that distribution with mean u m = 0 and standard deviation What is the CL for measuring x ≥ 2 (2 = 1.
above the mean)?
X
is described by a Gaussian l u CL 100
prob
(
X
2) 100 1 2 2
e
(
x
m 2 2 ) 2
dx
100 2 2
e
x
2 2
dx
2.5% To do this problem we needed to know the underlying probability distribution function
p
.
u u If the probability distribution was not Gaussian (e.g. binomial) we could have a very different CL.
If you don’t know
p
you are out of luck!
Interpretation of the CL can be easily abused.
u Example
:
We have a scale of known accuracy (Gaussian with n We weigh something to be 20 gm.
= 10 gm).
n Is there really a 2.5% chance that our object really weighs ≤ 0 gm??
F probability distribution must be defined in the region where we are trying to extract information.
l Interpretation of the meaning of a CL depends on Classical or Baysian viewpoints.
Baysian and “Classical” are two schools of thought on probability and its applications.
P416 Lecture 8 Richard Kass S07 4
Confidence Intervals (CI)
l For a given Confidence Level, confidence interval is the range [
x
1
,
x
2
].
u Confidence Interval’s are not always uniquely defined.
l u We usually seek the minimum or symmetric interval.
Example: Suppose we have a Gaussian distribution with m = 3 and = 1.
u What is the 68% CI for an observation?
u u We need to find the limits of the integral [
x
1
,
x
2
] that satisfy:
x
2 0 .
68
p
(
x
)
dx
For a Gaussian distribution the area enclosed by ±1
x x
2 1
= = m m 1 + 1
x
1 = 2 = 4 is 0.68.
The Confidence Interval is [2,4].
For a CI we know
p
(x) and CL. We want to determine x 1 and x 2
Upper Limits/Lower Limits
l Example: Suppose an experiment observed no events of a certain type they were looking for.
u u What is the 90% CL upper limit on the expected number of events?
CL 0.90
n
1
e
n
!
n
If we expect to observe zero 1 CL 2.3
0.10
1
n
1
e
n
!
n
n
0
e
n
!
n
e
=2.3 then 10% of the time events even though there is nothing wrong with the experiment!
If the expected number of events is greater than 2.3 events, l the probability of observing one or more events is greater than 90%.
Example: Suppose an experiment observed one event.
u What is the 95% CL upper limit on the expected number of events?
5 Richard Kass S07 P416 Lecture 8
CL 0 .
95
n
2
e
n n
!
1 CL 0 .
05 1
n
2
e
n
4 .
74
Procedure for Hypothesis Testing
n
!
n
1 0
e
n n
!
e
e
a) Measure something. b) Get a hypothesis (sometimes a theory) to test against your measurement.
c) Calculate the CL that the measurement is from the theory.
d) Accept or reject the hypothesis (or measurement) depending on some minimum acceptable CL.
Problem: How do we decide what is an acceptable CL?
u Example: What is an acceptable definition that the space shuttle is safe?
One explosion per 10 launches or per 1000 launches or…?
Hypothesis Testing for Gaussian Variables
If we want to test whether the mean of some quantity we have measured (
x
measurements) is consistent with a known mean ( = average from
n
m
0
) we have the following two tests: m Test = m
0
Condition
2
known Test Statistic
x
/ m 0
n
Test Distribution m Gaussian = 0, = 1 m = m
0
2
unknown
x
m
s
/
n s
n
Richard Kass S07 0
t
(
n
measurements.
– 1)
t
(
n
– 1): Student’s “t-distribution” with
n
– 1 degrees of freedom.
Student is the pseudonym of statistician W.S. Gosset who was employed by a famous English brewery.
P416 Lecture 8 6
Procedure for Hypothesis Testing
a) Measure something. b) Get a hypothesis (sometimes a theory) to test against your measurement (“null hypothesis”, H 0 ).
c) Calculate the CL that the measurement is from the theory.
d) “Accept” or “reject” the hypothesis (or measurement) depending on some minimum acceptable CL.
Problem: How do we decide what is an acceptable CL?
u Example: What is an acceptable definition that the space shuttle is safe?
One explosion per 10 launches or per 1000 launches or…?
Reject H
0
is True Type 1 error H
0
is False
OK
Accept
OK
Type 2 error
In hypothesis testing we are assuming that H
0
is true. We never disprove H
0
.
If the CL is low all we can say is that our data do not support H
0
.
If our CL is
a%
(e.g. 5%) then we make a type 1 error
a
% of the time if H
0
is true!
In a trial H 0 = innocent. Convicting an innocent person is a type 1 error while letting a guilty person go free is a type 2 error.
P416 Lecture 8 Richard Kass S07 7
l Example: Do free quarks exist? Quarks are nature's fundamental building blocks and are thought to have electric charge (|q|) of either (1/3)e or (2/3)e (e = charge of electron). Suppose we do an experiment to look for |q| = 1/3 quarks.
u u u Measure: |q| = 0.90 ± Quark theory: |q| = 0.33 = m 0 Test the hypothesis m = Use the first line in the table:
z
m
0
x
when / 0.2 (This gives m 0
n
is known: 0.9
0.33
0.2/ 1 m and 2.85
) n Assuming a Gaussian distribution, the probability for getting a
z
≥ 2.85,
prob
(
z
2.85) 2.85
P
( m , ,
x
)
dx
2.85
P
(0,1,
x
)
dx
1 2 2.85
e
x
2 2
dx
0.002
n CL is just 0.2%!
n If we repeated our experiment 1000 times, two experiments would measure a value |q| ≥ 0.9 if the true mean was |q| = 1/3.
This is not strong evidence for |q| = 1/3 quarks!
If instead of |q| = 1/3 quarks we tested for |q| = 2/3 what would we get for the CL?
n m = 0.9 and = 0.2 as before but m
0
= 2/3.
z
= 1.17
prob(
z
≥ 1.17) = 0.13 and CL = 13%.
Quarks are starting to get believable!
P416 Lecture 8 Richard Kass S07 8
l l Consider another variation of |q| = 1/3 problem. Suppose we have 3 measurements of the charge q: q 1 = 1.1, q 2 = 0.7, and q 3 = 0.9
u We don't know the variance beforehand so we must determine the variance from our data. use the second test in the table: m 1 3 (
q
1
q
2
q
3 ) 0.9
s
2
z
n
i
1 (
q i
n
1
x
s
/ m 0
n
m ) 2 0.2
2 0.9
0.33
0.2 / 3 ( 0.2) 2 2 4.94
0 0.04
n Need a
t
distribution table: Table 7.2 of Barlow: prob(
z
≥ 4.94) ≈ 0.02 for
n
– 1 = 2.
10X greater than the first part of this example where we knew the variance ahead of time.
Consider the situation where we have several independent experiments that measure the same quantity: We do not know the true value of the quantity being measured.
We wish to know if the experiments are consistent with each other.
m 1 – m 2 = 0 m m 1 1 – – Test m 2 m 2 = 0 = 0 Conditions 1 2 and 2 2 1 2 known 1 = 2 2 = 2 2 ≠ 2 2 Test Statistic
x
1
x
2 1 2
Q
/
n
2 2 /
m x
1
x
2 1/
n
1/
m s
1 2
x
1
x
2 /
n
s
2 2 /
m
Test Distribution m m Gaussian = 0, = 0, = 1
t
(
n
+
m
– 2) approx. Gaussian = 1
Q
2 (
n
1)
s
1 2 (
m n
m
2 1)
s
2 2 Richard Kass S07 P416 Lecture 8 9
l Example: We compare results of two independent experiments to see if they agree with each other.
Exp. 1 1.00
± 0.01
Exp. 2 1.04
± 0.02
Use the first line of the table and set
n
=
m
= 1 .
z
1 2
x
1 /
n
x
2 2 2 /
m
1.04
(0.01) 2 1.00
(0.02) 2 1.79
n n
z
is distributed according to a Gaussian with m 0, 1.
Probability for the two experiments to disagree by ≥ |0.04|:
prob
(
z
1.79) 1 1.79
P
1.79
( m , ,
x
)
dx
1 1.79
P
1.79
(0,1,
x
)
dx
1 1 2 1.79
1.79
e
x
2 2
dx
0.07
We don't care which experiment has the larger result so we use ±
z
.
7% of the time we should expect the experiments to disagree at this level.
Is this acceptable agreement?
P416 Lecture 8 Richard Kass S07 10