Transcript Document

Evaluate Statistically Based
Reports ( AS 3.12)
Workshop AJ
Margin of Error :Clarifying the rules of thumb
Dru Rose (Westlake Girls High School)
The purpose of this workshop
• To clarify the rules of thumb for estimating
MoE and their relationship to theory .
• To demonstrate the power of technology for
developing the concept of margin of error
(making the topic accessible to a wider diversity of students
than a theoretical approach relying on the central limit
theorem and the normal distribution).
• To share two activities I have developed for clarifying
the rules of thumb with students.
Dru Rose
3 types of claim and rules of thumb:
• Single poll %
51% of young people agree there is too much
1
sex, violence and bad language on TV MoE ≈ √𝑛
• Comparison within one group
Young people are more likely to agree than
disagree
MoE for the difference ≈ 2 x MoE
• Comparison between independent groups
Young women are more likely to agree than young
men
MoE for the difference
≈ 1.5 x Average MoE
Dru Rose
Where do the rules of thumb come from?
1. Single poll %
1
√𝑛
MoE ≈
Media reports use a 95% level of confidence.
Usual theoretical formula for
standard error of a single
proportion
1.96≈2, 𝑝 = 0.5 for max MoE , 𝑛 − 1 ≈ n in large samples:
MoE (p) ≈ 2
Dru Rose
1 1
×
2 2
𝑛
≈
1
√𝑛
Where do the rules of thumb come from?
2. Comparison within one group
MoE for the difference ≈ 2 x MoE
𝑝1 and 𝑝2
𝑎𝑟𝑒 𝑡𝑤𝑜 𝑚𝑢𝑙𝑡𝑖 −
𝑛𝑜𝑚𝑖𝑎𝑙 𝑝𝑟𝑜𝑝𝑜𝑟𝑡𝑖𝑜𝑛𝑠
If p ≈ 0.5, (i.e. two main options with others having
very small support) then 𝑝1 + 𝑝2 ≈1 , 𝑝1 - 𝑝2 ≈ 0,
1.96 ≈ 2 , and for large samples n -1≈ n,
MOE(p1 − p2) ≈ 2 ×
Dru Rose
𝟏
𝒏
= 2 x MoE
Where do the rules of thumb come from?
3. Comparison between 2 independent groups
MoE for the difference ≈ 1.5 x Average MoE
When p1 and p2 ≈ 0.5 and n1 = n2= n, and 1.96 ≈2,
this formula reduces to :
2×
1
2𝑛
= 2×
𝟏
.
𝒏
We can show that “1.5 ×Average MoE” is a reasonable
approximation in most situations:
Dru Rose
Developing the rules of thumb with students:
Use of technology (Central Limit Theorem and normal
approximation to binomial distribution no longer in NZEA Level 3 )
1. Single poll % MoE ≈
• Wild’s animations show:
(i) need for large sample sizes to keep MoE around 5%
1
√𝑛
or below.
(ii) Max MoE when 𝑝=50% and rule of thumb is OK for
30% ≤𝑝 ≤ 70% (Outside this range MoE is much smaller1
dropping to almost half when 𝑝= 10% or 90%)
√𝑛
Dru Rose
Developing the rules of thumb with students:
1
Use of technology (Single poll % MoE ≈ )
√𝑛
• We can use the KareKare cards and bootstrap
and coverage VIT modules in iNZight to
develop the concept of a CI for a poll% and
1
the rule of thumb (see the 2012 Stats Day
√𝑛
presentation on Census at School)
n=100
MoE half as
long as CI
1
≈ 10% =
100
n = 500, CI length ≈ 9%,
1
500
= 4.5%
Developing the rules of thumb with students:
1
Use of technology (Single poll % MoE ≈ )
√𝑛
• We can use the coverage spreadsheet
developed by Chris Wild and Dave Smith to
show that the rule of thumb gives about 95%
coverage for realistic sample sizes of around
say n= 600
Developing the rules of thumb with students:
2. Comparison within one group 2 x MoE
• We cannot use iNZight this time.
• The rule hinges on the premise that there are two
main options with others having very small support.
When this is the case, the following argument is valid:
Suppose one option has 55% support and the second
has 45% support, with a poll MoE of 5%.
The first option could have as low as 50% support and
the second as high as 50% support.
We need a poll% difference between them of more
than 2 x MoE (i.e.>10%) to conclude that the first
option has more support.
Developing the rules of thumb with students:
2. Comparison within one group 2 x MoE
• Use of technology:
• We can use the coverage spreadsheet to
demonstrate that the 2 x MoE rule generally gives
about 95% coverage provided there are two main options
with close to 50% support and other options having very little
support.
• when support for other options is substantial, e.g. current
Green Party support, the 2 x MoE rule for the main
players over-estimates the MoE for the difference)
Developing the rules of thumb with students:
3. Comparison between 2 independent groups
1.5 x Average MoE
• We can use the KareKare cards and bootstrap VIT
module in iNZight, with the sample within groups
option.
(Students can watch the differences arrow flip direction
and note how often negative differences are produced.)
Developing the rules of thumb with students:
3. Comparison between 2 independent
groups
We can use the coverage spreadsheet to
demonstrate that the “1.5 x Average MoE”
rule gives about 95% coverage in most real
polling situations:
Developing the rules of thumb with students:
3. Comparison between 2 independent groups
1.5 x (1/ 𝑛1 + 1/ 𝑛2 )/2
• Consider the following extreme scenario:
Suppose 𝑛1 = 1000, 𝑛2 = 500 and 𝑝1 = 35.5%, 𝑝2 = 30%,
(extreme values for rule of thumb and sample sizes)
Correct formula: MoE difference = 5 perc. points
Rule of thumb: MoE difference = 5.7 perc. points
poll% difference = 5.5 percentage points
which is actually significant but rule of thumb would suggest
otherwise.
But generally p≈50% and 𝑛1 ≈ 𝑛2
How can we help students sort out which rule to apply when
testing claims in the media?
Testing claims in the media
1. The Herald Digipoll survey found 59.2 per cent support
across the Super City for the new no-mow policy.
Test the claim that:
“Most Aucklanders… appear to support the controversial
council decision to stop mowing roadside berms”
1
MoE =
= 4.5% 54.7%
54.7%
59.2%
√500
2. women were more in favour [of the policy] than men.
(You may assume that equal proportions of men and women were sampled).
MoE =
1
√250
= 6.3%
-1.2 p. pts
MoE diff = 1.5 × 6.3 = 9.5%
8.3p.pts
(63.2 – 54.9)
17.8 p pts
3. In the former Auckland City, a higher proportion of
residents disagreed than agreed with the new policy.
Diff = 46.3 – 43.6 = 2.7 p. pts
-13.5 p. pts
1
√152
= 8.1%
2× MOE = 16.2 %
2.7p.pts
19.1 p pts
Claim NOT supported
4. Can it be claimed that support is consistently higher outside the
old central Auckland area?
1
Franklin pop. Only 4.5% n = 22
=21.3%
1
√152
√22
Auckland
= 8.1%
Av. MoE = 14.7%
Diff: 63.2-43.6 = 19.6 p.pts
1.5× Av. MoE = 22%
-2.4 p. pts
19.6p.pts
Claim NOT supported
41.6 p pts
Research New Zealand Survey 28/02/2012: “Are Our Buildings Safe to Occupy?”