Transcript 6. Hypothesis Testing and the Comparison of 2 or More Populations

6. Hypothesis Testing and the
Comparison of 2 or More
Populations
ASW Chapter 9 + Chapter 10
A) Introduction

Estimating parameters of population
→ hypothesis testing on our model. Economic
Theory
Model or Theory
1
Deductions/Analysis
2
5
Testable Hypothesis
Modify Theory
Accept Theory
(For Now)
Reject Theory
4
Empirical Result
Tests
Ch. 9-11,
12, 13
3
Ch. 1-8,
12, 13
Data
2
Confidence Intervals and Hypothesis Testing
Confidence intervals → range that μ falls into.
 NOW: is μ > 0, or > 1, etc.
 OR: is μ1 > μ2?
 Testing for specific values of μ.
 We have a confidence interval for Saskatchewan
female wages by age.
 What could we test here?
 We will have a confidence interval for Bachelor’s
salaries in Saskatchewan.
 What could we test here?
 Others: gasoline prices? Stock Market fluctuations?

3
B) Developing Null and Alternative Hypotheses
Start with a testable hypothesis.
 Point of interest: do older women get paid more?
 Economic theory: is 0 < MPC < 1 and constant?
 Define it’s opposite:
 Older women’s salaries are < average.
 MPC is > 1.
 One is the Null hypothesis, the other is the Alternative
hypothesis.
 Use sample data to test the Null hypothesis.


What if it is not that simple to have 2 opposites?
4
Which is the Null?

General rule: the hypothesis with the = sign or the
< or the > sign is the Null.
 OR: the Null is something we assume is true
unless contradicted by the sample.
5
1.
Research hypotheses:
Testing an exception to the general rule, so it goes in
the alternative.
 E.g, testing if older women’s salaries (μ) > average:
H0: μ < μ(average)
HA: μ > μ(average)
 Results will tell us either:
 If testing shows H0 cannot be rejected (“accepted”)
→ implies that older women’s salaries are not
higher, but we cannot be sure.
 If testing shows H0 can be rejected → we can infer
HA is true, μ > μ(average).

6
2.
Testing the Validity of a Claim
Assume claim is true until disproven.
 E.G.: manufacturer’s claim of weight/container.
H0: μ > 100 grams.
HA: μ < 100 grams.
 Results will tell us either:
 If testing shows H0 cannot be rejected (“accepted”)
→ manufacturers claim not challenged.
 If testing shows H0 can be rejected → we can infer
manufacturer is lying.

7
3.
Testing in Decision-Making
Here, if either too high or too low, need to do
something.
 E.G.: is class length 75 minutes?
H0: μ = 75 minutes.
HA: μ ≠ 75 minutes.
 If H0 not rejected (accepted), no change in behaviour.
 If H0 rejected –> change behaviour.

8
C) Type I and Type II Errors

Sample data → could have errors.
9
Type I and Type II Errors
Population Condition
H0 True
H0 False
Correct
Decision
Type II Error
Type I Error
Correct
Decision
Conclusion
Accept H0
Reject H0
False Negative
False Positive
10
False Positive or False Negative?
New claims over bungled
shooting of Brazilian
By Mark Sellman, Times
Online, and Daniel McGrory
U.K. police defend shoot-to-kill after mistake
…
Blair said Menezes had emerged from an
apartment block in south London that had been
under surveillance in connection with Thursday’s
attacks, and refused police orders to halt. Menezes
had also been wearing an unseasonably heavy
coat, further raising police suspicions.
MSNBC, July 24th.
11
Other Type I and Type II Errors
Sampling songs.
 Health tests.
 Pregnancy tests.
 Jury decisions.

12
Level of Significance
Hypothesis testing is really designed to control the
chance of a Type I error.
 Probability of Type I error = the level of significance.
 Selecting  (= level of significance )
 select probability of Type I error.
 What is the level of significance for Jury trials?


We do not control for Type II errors
–> except by our language of stating “do not reject
H0”.
13
Level of Significance cont’d
 is picked by researcher –> normally 5%?
  = 5% → type I error happens only 5% of the time.

14
D) The probability value (p-value) approach
1.
2.
3.
Develop null and alternative hypothesis.
Select level of significance .
Collect data, calculate sample mean and test statistic.
p-value approach
4.
5.
Use test statistic to calculate p-value.
Compare: reject H0 if p-value < .
 The sample implies that the alternative (your
research hypothesis) is true.
15
Hypothesis Testing: The Critical Value Approach
1. Develop null and alternative hypothesis.
2. Select level of significance .
3. Collect data, calculate sample mean and test
statistic.
Critical-value approach
4.
5.
Use  to determine critical value and rejection rule.
Compare: if |test statistic| > |critical value|, reject H0.
 The sample implies that the alternative (your
research hypothesis) is true.
16
Hypothesis Testing cont’d…

This is essentially inverting our confidence index.
→ Is more than 2 standard deviations away from
some benchmark?
17
E) Population Mean, σ Known, One-Tailed Test
Same hypothesis
 p-value method and critical value method.
 Example: a new employment program initiative has
been introduced to reduce time spent being
unemployed.
 Goal: 12 weeks or less unemployed.
 Population standard deviation believed to be 3.2
weeks.
 Sample of 40 unemployed workers, average time
unemployed 13.25 weeks.
 Assuming a level of significance () of .05, is the
program goal being met?

18
First (Common) Steps

Hypothesis:
H0: μ < 12
HA: μ > 12

Clearly 13.25 > 12.
 This casts doubt on our program goal (the null), and
whether we should continue it.
 Key: is it enough more, given sample size and
standard deviation, or is it just a (small) random
fluctuation?
19
Computing the Test Statistic
Under our assumptions: use the standard normal.
 Use sample mean to calculate test statistic

x   13.25  12
z

 2.47
 / n 3.2 / 40
Is this z big enough to reject the null hypothesis?
 Next, go our two routes:
Calculate p-value OR z-critical.

20
Calculating the p-value

Given the z-value, what is the corresponding
probability?
p=??
0

Z=2.47
This is the probability that 13.25 > 12 by chance.
z
21
Calculating the p-value cont’d

Find 2.47 on the Standard Normal Distribution tables:
z
.00
.01
.02
.03
.04
.05
.06
.07
.08
.09
…
2.3
2.4 .4918 .4920 .4922 .4925 .4927 .4929 .4931 .4932 .4934 .4936
2.5
…
22
Calculating the p-value cont’d

.4932 is the probability of being between 0 and z=2.47.
0.4932
0

Z=2.47
z
p = 0.5 – 0.4932 = 0.0038.
23
Should We Reject the Hypothesis?

This says that the probability of getting a sample mean
of 13.25 when the true mean is 12 = .0038 or less than
½ of 1 percent.
 Our significance level was only 5%, so we reject the
null.
 We are 99.62% certain that the program has failed.
24
Rejection of the Null
0.0038
0
Z=2.47
z
Z.05
Sometimes
we say: “significant at the 0.38% level”.
25
Critical Value Approach
This is an alternative you often see in textbooks or
articles.
 Find the value of z.05, and compare it to the test value
of z (2.47).
 From the tables, z.05 = 1.645.
 Because 2.47 > 1.645, reject H0.

26
Rejection
0
Z=2.47
z
Z.05= 1.645
27
Excel

Let’s do this example in Excel.
 Look at Appendix 9.2 in text, especially Figure 9.8
28
F) Population Mean, σ Known, Two-Tailed Test
Null: μ = μ0 → Alternative is μ ≠ μ0.
 Must examine two areas of the distribution.
 Example:
 Price/earnings ratios for stocks.
 Theory: stable rate of P/E in market = 13.
 If P/E (market) < 13, you should invest in the
stock market.
 If P/E (market) > 13, you should take your
money out.

29
Estimate Steps
Can we estimate if the population P/E is 13 or not?
 Common steps:
 Set hypothesis:
H0: μ = 13
HA: μ ≠ 13
 Select  = .05.
 Calculate standard error.
 Calculate z-value.

30
Calculating Test Statistic

We have a sample of 50
 = 12.1.
 Historical σ = 3.0456.

3.0456
Standard Error x 

 0.4307
n
50
z  value 
x - 0
x
12.1  13

 2.0896
0.4307
31
p-value Approach

Calculate the p-value.
 We will calculate for the lower tail
→ then make an adjustment for the upper tail.
32
p-value, Two-Tailed Test
p(z > 2.09) = ??
p(z < –2.09) = ??
Z=–2.09
0
Z=2.09
z
We can just calculate one value, and double it.
33
Calculating the p-value cont’d

Find 2.090 on the Standard Normal Distribution
tables:
z
.00
.01
.02
.03
.04
.05
.06
.07
.08
.09
…
1.9
2.0 .4772 .4778 .4783 .4788 .4793 .4798 .4803 .4808 .4812 .4817
2.1
…
34
p-value, Two-Tailed Test
0.4817
p(z > 2.09) = 0.5 -.4817
= 0.0183
p(z < –2.09) = ??
Z=–2.09
0
Z=2.09
z
Doubling the value, we find the p-value = 0.0366
35
Should We Reject the Null Hypothesis?
Yes!
 p-value = 0.0366 <  = 0.05.
 There is only a 3.66% chance that the measured
price/earnings ratio sample mean of 12.1 is not equal to
the stable rate of 13 by random chance.

36
Critical Value Approach
Reject the null hypothesis if:
test z-value > critical value
or if
test z-value < critical value
 Two tailed test:  = 0.05 → need critical value for
/2 = 0.025.
 The tables tell us that this is 1.96.

37
G) Population Mean, σ Unknown

σ unknown → must estimate it with our sample too
→ use t-distribution, n – 1 degrees of freedom.
s
Standard Error x 
n
t  value 
x - 0
x
38
One-tailed Test, p-value Approach

Steps:
1. Set up hypothesis.
2. Decide on level of significance ()
3. Collect data, calculate sample mean and test
statistic.
4. Use test statistic & t-table/Excel to calculate pvalue.
5. Compare: reject H0 if p-value < .
39
Example: Highway Patrol
The RCMP periodically samples
vehicle speeds at various locations
on a particular roadway.
The sample of vehicle speeds
is used to test the hypothesis
H0:  < 65
The locations where H0 is rejected are deemed
the best locations for radar traps.
40
Example: Highway Patrol
Outside Lumsden:
A sample of 64 vehicles
–> average speed = 65.5 mph
–> standard deviation = 4.2 mph.
Use  = .05 to test the hypothesis.
41
Common to Both Approaches
1. Determine the hypotheses.
H0:  < 65
Ha:  > 65
2. Specify the level of significance.
 = .05
3. Compute the value of the test statistic.
t  value 
x - 0
x
65.5  65

 .9524
4.2 / 64
42
4. Estimate the p-value From t-Distribution Table

Must “interpolate” the value of t = 0.9524, df = 65
Degrees
of
Freedom
1
Area in Upper Tail
0.20
0.10
0.05
0.025
0.01
0.005
50
.849
1.299
1.676
2.009
2.403
2.678
60
.848
1.296
1.671
2.000
2.390
2.660
80
.846
1.292
1.664
1.990
2.374
2.639
…
…
0.9524
43
.10 < p–value < .20
p –Value Approach
5. Determine whether to reject H0.
Because p–value >  = .05, we do NOT reject H0.
We are at least 95% confident that the mean speed
of vehicles outside Lumsden is LESS than OR
EQUAL TO 65 mph.
44
Critical Value Approach
4. Determine the critical value and rejection rule.
For  = .05 and d.f. = 64 – 1 = 63, t.05 = 1.669
Reject H0 if t > 1.669
5. Determine whether to reject H0.
Because 0.9524 < 1.669, we do NOT reject H0.
45
H) Introduction: Comparing Population Differences
Do men get paid more than women?
 $46,452 for men vs. $35,122 for women (bachelors).
 Do more 100-level Economics courses help you in Econ
201?
 Has the crime rate risen?
 Are there more hurricanes
recently?

46
Key point: the role of standard deviation
Probably the
same
x1
x2
Probably
different
x1
x2
47
Comparing 2 Populations
True population means: 1 and 2.
 Random sample of n1 –> 1.
 Versus random sample of n2 –> 2.
 Transform into problem: is 1 – 2 = 0?


Assuming 1 and 2 known –> use z-test.
 If unknown –> estimate ’s from sample s’s, and
use t-test.
48
I) Confidence Intervals, 2 Means: ’s Unknown
How important is an extra introductory course in
determining your grade in Economics 201?
 Data:
 Natural experiment.
 59 students.
 43 had only one 10x course.
 16 had two 10x courses.
 Final exam grades:
 One 10x: average = 61.69%, s1 = 22.65, n1 = 43.
 Two 10x: average = 75.11%, s2 = 12.80, n2 = 16.

49
Confidence Interval Estimation
Point estimator: 1 – 2.
 Standard error of 1 – 2 is:

 x x 
1
2
2
1
2
2
s
s

n1 n2
50
Confidence Interval cont’d

Confidence interval of difference in means:
– 2 + Margin of Error
 Typically use α = 0.05.
 Margin of error:
1
x1  x2  t / 2
2
1
2
2
s
s

n1 n2
51
Degrees of Freedom

One UGLY formula:
2
s s 
  
n1 n2 

df 
2 2
2 2
1  s1 
1  s2 
  
 
n1  1  n1  n2  1  n2 
2
1
2
2
In this example: df = 47.36 → round down to 47.
 95% confidence interval → t0.025.
 For 47 degrees of freedom, table says: 2.012.

52
Confidence Interval cont’d.
x1  x2  t / 2
2
1
2
2
s
s

n1 n2
or
513.17 163.79
61.86  75.11 2.012

43
16
or
53
Confidence Interval cont’d.
- 13.25 2.012* 4.709
or
- 13.25 9.475
or
- 3.775 to - 22.725
54
Confidence Interval
We are 95% confident that students with only one 10x
course scored between 3.775% and 22.725% lower
than students with two 10x courses.
 Next step would be: why, how??

55
J) Hypothesis Tests, 2 Means: ’s Unknown
Two datasets –> is the mean value of one larger than
the other?
 Is it larger by a specific amount?
 μ1 vs. μ2 –> μ1 – μ2 vs. D0.
 Often set D0 = 0 –> is μ1 = μ2?

56
Example: Female vs. Male Salaries
Saskatchewan 2001 Census data:
- only Bachelor’s degrees
- aged 21-64
- work full-time
- not in school
 Men: M = $46,452.48, sM = 36,260.1, nM = 557.
 Women: W = $35,121.94, sW = 20,571.3, nW = 534.
 M – W = $11,330.44 } our point estimate.
 Is this an artifact of the sample, or do men make
significantly more than women?

57
Hypothesis, Significance Level, Test Statistic


1.
2.
3.
We will now ONLY use the p-value approach, and
NOT the critical value approach.
Research hypothesis: men get paid more:
H0: μM – μW < 0
H1: μM – μW > 0
Select  = 0.05
Compute test t-statistic:
t
( xM  xW )  D0
sM2 sW2

nM nW
(46452.48  35121.94)  0

 6.375
1314794928 426178351

557
534
58
4. a. Compute the Degrees of Freedom

Can compute by hand, or get from Excel:
2
s s 
  
n1 n2 

=
888
df 
2 2
2 2
1  s1 
1  s2 
  
 
n1  1  n1  n2  1  n2 
2
1
2
2
59
4. b. Computing the p-value
Degrees of
Freedom
…
100

0.20
0.10
0.02
0.025
0.01
0.005
.845
.842
1.290
1.282
1.660
1.645
1.984
1.960
2.364
2.326
2.626
2.576
6.375 up here somewhere
The p-value <<< 0.005.
60
5. Check the Hypothesis
Since the p-value is <<< 0.05, we reject H0.
 We conclude that we can accept the alternative
hypothesis that men get paid more than women at a
very high level of confidence (greater than 99%).

61
Excel
t-Test: Two-Sample Assuming Unequal Variances
Male
Female
Mean
46452.47935 35121.94195
Variance
Observations
1314794928
557
423178351.3
534
Hypothesized Mean Diff.
df
t Stat
P(T<=t) one-tail
0
888
6.381034789
1.41441E-10 .00000000014
t Critical one-tail
P(T<=t) two-tail
1.646571945
2.82883E-10
t Critical two-tail
1.962639544
62
Summary

Hypothesis tests on comparing two populations.
 Convert to a comparison of the difference to a
standard.
 More complex standard deviation and degrees of
freedom.
 Same methodology as comparing other hypothesis
tests.
63
K) Statistical vs. Practical Significance
Our tests: statistically significantly
 Real world interest:practical significance.

Men vs. women: the difference is statistically
significant AND practically:
$46,452.48 vs $35,121.94
 Saskatchewan, full-time, Bachelor’s:
Women make only 75.6% of men, same average
education level.

64
65
Source: Leader-Post, Oct. 31, 2008
L) Matched Samples
Controlled experiment –> match individuals in each
group.
 Matched samples –> each individual tries each method
in turn.
 Variation between samples not a problem.
 Focus on difference data.
 Independent samples –> the norm in economics.
 Regression analysis.

66
M) Introduction to ANOVA
What if we want to compare 3 or more sample means
(treatment means)?
 Example: total income, Saskatchewan females
employed full-time and full-year, by age, 2003

(Source: See Oct. 8th lectures)
Age group
Income in thousands of
dollars
Mean
25-34
35-44
33.3
40.3
45-54
55-64
45.1
40.1
Sample size
Standard
deviation
13.5
Overall
weighted
20.7
average
= 38.2
25.9
25.9
55
57
37
31
67
ANOVA’s Hypotheses
There is one true population mean and 4 sample variations.
H 0 : 1  2  3  4
H A : Not all populationmeansare equal
4 different populations.
68
N) Steps of ANOVA
1. Set up the Hypothesis Statements
H0: μ1 = μ2 = μ3 = μ4 = … = μk
HA: Not all population means are equal
2. Collect your sample data:
Means:
1,
2,
3,
4,
…
k
Variances:s21, s22, s23, s24 ,… s2k
Sample Sizes:
n1, n2, n3, n4,… nk
69
Steps of ANOVA Continued
4. Calculate the overall average:
n1 x1  n2 x2  n3 x3  ...  nk xk
x
n1  n2  n3  ...  nk
5. Create our two estimates of 2.
70
Step 5 a) Estimating 2 via SSTR

Between-treatments estimate of 2 or
sum of squares due to treatments (SSTR).
 This compares x j ' s to x , and constructs an estimate of 2
based on the assumption the Null Hypothesis is true:
SST R   j 1 n j ( x j  x )
k
2
 n1 ( x1  x )  n2 ( x2  x )  ...  nk ( xk  x )
2
2
71
2
Step 5 b) Estimating 2 via SSE

Within-treatments estimate of 2 or
the sum of squares due to error (SSE).
 This takes the weighted average of the sample sj2 as
an estimate of 2 and is a good estimate regardless
of whether the Null is true:
k
SSE   (n j  1) s
j 1
2
j
 (n1  1) s12  (n2  1) s22  ...  (nk  1) sk2
72
Step 6: Testing The Null
If Null true, both estimates should be similar, and
SSTR ≈ 1.
SST
 If ratio >>> 1  reject the Null, accept the
Alternative that there is multiple population
distributions.

73
Steps of ANOVA
1. Set up the Hypothesis Statement.
(Null: all means are equal)
2. Collect the sample data.
3. Select level of significance –> α = 0.05.
4. Calculate the overall average.
5. a) Estimate 2 via
sum of squares due to treatments (SSTR).
b) estimate of 2 via
sum of squares due to error (SSE).
6. If Null true, both estimates should be similar, and
STR ≈ 1.
SST
74
MSTR and MSE
MSTR = sum of squares due to treatment
df1
numerator degrees of freedom
= sum of squares due to treatment
no. of treatments – 1
= SSTR
k-1
 MSE =
Sum of squares due to error
df2
denominator degrees of freedom
=
Sum of squares due to error
total no. of obs. – no. of treatments
=
SSE
nT – k

75
F-test
F-statistic = MSTR  k-1 degrees of freedom (df1)
MSE  nT – k degrees of freedom (df2)
 If H0 is true, MSTR ≈ MSE → F-statistic ≈ 1.
 If H0 is false
→ p-value is < level of significance (α).
→ F-statistic is higher than critical value from the
table/Excel.

76
F-Distribution
Probabilit y that
0
Ftest-value
MSTR
1
MSE
F
MSTR
MSE
77
O) Saskatchewan Female Wages Example

Example: total income, Saskatchewan females
employed full-time and full-year, by age, 2003
(Source: See Oct. 8th lectures)
Age Group
Mean Income
Variance =
(St. Dev.)2
Sample Size
25-34
33.3
(13.5)2 = 182.25
55
40.3
(20.7)2 = 428.49
57
45.1
(25.9)2 = 670.81
37
40.1
(25.9)2 = 670.81
31
35-44
Overall
weighted
average
= 38.2
45-54
55-64
78
Calculating the MSTR, MSE
k
SST R   j 1 n j ( x j  x ) 2
 n1 ( x1  x ) 2  n2 ( x2  x ) 2  n3 ( x3  x ) 2  n4 ( x4  x ) 2
 55(33.3  38.2) 2  57(40.3  38.2) 2  37(45.1  38.2) 2
 31(40.1  38.2) 2
 3445.4
MSTR  SSTR/(k  1)  3445.4 / 3  1148.5
k
SSE   (n j  1) s 2j
j 1
 (n1  1) s12  (n2  1) s22  (n3  1) s32  (n4  1) s42
 54*182.25  56* 428.49  36* 670.81 30* 670.81
 78110.4
MSE  SSE /( NT  k )  78110.4 /(180 4)  443.8
79
Calculating F-Stat and p-value
Ftest-value = MSTR = 1148.5 = 2.59
MSE
443.8
P - value  ?
0
2.59
F
MSTR
MSE
80
F-Table for df1 = 3 and df2 = 176
Numerator degrees of freedom (df1)
Denomina(MSTR)
tor degrees
of freedom Area in
2
3
4
(df2) (MSE) Upper Tail
100
.10
…
2.14
Clearly the
p-value > 0.05
176 degrees of
.05
2.70
–> accept the Null of one
freedom,
.025
3.25
distribution
F=2.59
.01
3.98
in here.
1000
.10
…
2.09
.05
2.61
.025
3.13
.01
3.80
81
Excel F-test formula

=FDIST(F-value, df1, df2) –> yields value of .0544.
P - value  0.0544
0
2.59
F
MSTR
82
MSE
P) Econometrics for Dummies…

Instead of ANOVA, economists tend to use Regression
analysis + “dummy” variables.
 Gives us the direction and size of the differences in
mean values.
 But ANOVA is a useful first step.
83