Transcript Slide 1

How to get a good sample
Thought Questions
1. Suppose a properly chosen sample of 1600 people across the United States was asked if
they regularly watch a certain television program, and 24% said yes.
How close do you think that is to the percentage of the entire country who watch the show?
Within 30%? 10%? 5%? 1%? Exactly the same?
2. Suppose you had a telephone directory listing all the businesses in a city, alphabetized by
type of business.
If you wanted to phone 100 of them to get a representative sampling of opinion on some
issue, how would you select which 100 to phone?
Why would it not be a good idea to simply use the first 100 businesses listed?
3. There are many professional polling organizations, such as Gallup and Roper. They often
report on surveys they have done, announcing that they have sampled 1243 adults, or some
such number.
How do you think they select the people to include in their samples?
How Gallup conducts polls
How to get a good sample
Sample Surveys
“It is better to know nothing than to know what ain't so.”
– Josh Billings (1818-1885), Humorist
You survey to know the general state of affairs: opinions, events, attitudes, intentions,
ownership, habits, purchases, demographic characteristics, and so forth.
Observation provides the basic data for a survey.
You count cars passing an intersection, and ask questions of consumers, voters, and
inhabitants.
A survey has three basic parts: the questions, the responses, and the analysis
Measurements, Mistakes, and Misunderstandings
Defining a Common Language
Determining Monthly Unemployment in the U.S.
Background: Bureau of Labor Statistics visits approx 60,000 households, chosen from list of all
known households in country. Each adult classified: employed, unemployed, “not in labor force.”
Unemployment rate: number of unemployed persons divided by the sum of the employed and
unemployed.
• Units: adults in the labor force
• Population of units: all adults in the labor force.
• Population of measurements: employment status (working or not working) of everyone in the
labor force.
• Sampling frame: list of all known households in country.
• Sample of units: people who were asked about their employment status.
• Sample of measurements: employment status of sample.
How to get a good sample
The beauty of sampling
•By its very nature, a sample cannot guarantee results that are the same as the results of a
census of all the individuals in the population of interest.
•The sample may give a proportion that is exactly the same as in the population, near the
value, or very far from the value.
•This uncertainty—which is inherent in the random process of choosing a sample—we call
error, variability, or uncertainty.
How to get a good sample
The beauty of sampling
If you did not know the true proportion of males in a population, took a random sample of 100
persons from a population to determine the proportion of males, and got a sample proportion of
.45, the margin of error would be
Margin of error 
1
n
= 1/square root of sample size = 1/10 = .10
Thus the 95% confidence interval is .45 plus or minus .10, or .35 to .55, or 45% plus or minus
10%.
What Gallup says
•Common sense -- and sampling theory -- tell us that a sample of 1,000 people probably is going
to be more accurate than a sample of 20.
•Gallup and other major organizations use sample sizes of between 1,000 and 1,500 because
they provide a solid balance of accuracy against the increased economic cost of larger and larger
samples.
•If Gallup were to -- quite expensively --use a sample of 4,000 randomly selected adults each
time it did its poll, the increase in accuracy over and beyond a well-done sample of 1,000 would
be minimal, and generally speaking, would not justify the increase in cost.
How to get a good sample
Unemployment Rate - Examples
What We Don’t Know About the Economy, Carl Bialik, WSJ, April 16th, 2010
•It isn't all that simple to work out how many Americans are out of work.
•The ranks of unemployed individuals grew by 134,000 last month from February, to 15
million, the Department of Labor's Bureau of Labor Statistics says.
•But it also is plausible, the agency says, that the number of unemployed rose by 500,000. Or,
it could have fallen by 200,000.
•In fact, at a time when high unemployment tops many people's worries about the economic
recovery, the BLS can say only that it is 90% confident that the true change in the number of
unemployed in March was somewhere between a drop of 243,000 and an increase of
511,000.
•In other words, it isn't even clear whether the number of unemployed rose or fell last month.
•The ranges are similarly broad for seven of the last 10 months—and for more than 75% of
the time in the past decade.
How to get a good sample
What We Don’t Know About the Economy, Carl Bialik, WSJ, April 16th, 2010
•sampling techniques help explain why there is so much uncertainty
• The jobless numbers are calculated by surveying a total of about 56,000 households in a small
number of U.S. counties.
•But the unemployment survey, conducted as a joint effort between the BLS and the Census
Bureau, doesn't use a random geographical sample.
•That would require either costly and time-consuming in-person interviews across the country,
or a telephone-only survey, which could risk providing unreliable information for a sensitive
report……
•Intentional deception by respondents, errors by survey takers or miscommunication between
interviewers and interviewees all can dent surveys' reliability.
•"It's one of the hazards of the job," Sean Snaith, an economist at the University of Central
Florida, says of flawed data. "You have to go with the data you have, not the data you wish
you have."
article
How to get a good sample
Sampling Methods
Probability sampling plans
Everyone in the population has a specified chance of making it into the sample .
•Simple Random Sampling(SRS)
•Stratified Random Sampling: Divide population into groups (strata) and take a simple random
sample from each.
Example: Separating 30,000 students into undergrad(27,000) and grad students(3,000)
SRS of 500 would contain how many grad students? Might want 300 undergrads and 200 grads.
•if only certain strata are (randomly) chosen to be used, and all subjects in these strata make up
the sample, then we have a cluster sample
•Example: Sample students living in a dorm at a college. College has 30 dorms, each dorm has 6
floors 180 floors form the clusters. Take a random sample of floors and measure everyone on
those floors.
•Other Methods: Systematic Sampling, Random Digit Dialing (why not use a phone book?)
Nonprobability method
•Convenience sampling is used in exploratory research where the researcher is interested in
getting an inexpensive approximation of the truth.
How to get a good sample
Difficulties and Disasters in Sampling
•Using the Wrong Sampling Frame: What are the two ways we could do this?
•Low response rate
•Using voluntary response
•Using a convenience or haphazard sample
How to get a good sample
Difficulties and Disasters in Sampling
Low response rate
The non respondent is the major problem in implementation.
In statistics, sampling bias is causing some members of the population to be less likely to be
included than others.
A low response rate can give rise to sampling bias
Pew Research Center Poll (2879 calls to get a sample of 1000 people)
Never answered phone 938
Answered but refused 678
Incomplete interview
Complete interview 1000
42
Not eligible 221
1221 responses of which 1000 were in the population they targeted
Non response rate = 2879 – 1221/2879 = 1658/2879 = 58%
How to get a good sample
How to live with a low response rate – Weighting the Responses
•Attempt to correct sources of bias
•If many urban households did not respond, survey gives more weight to those that did respond
•If too many women are in the sample, the survey gives more weight to the men
•The goal is to get results “as if” sample matched the population in age, gender, place of residence,
race and education.
Voluntary response
Literary Digest in 1936 . The magazine said that Alfred M. Landon would win the election.
Instead, Franklin D. Roosevelt was elected with 61% of the popular votes cast.
The Literary Digest survey sent mail questionnaires to its subscribers, car owners and those
who owned a telephone. Only 23% of the recipients responded.
How to get a good sample
Is America Islamaphobic? – Time Magazine, Aug. 19th 2010
How to get a good sample
New York Poll Finds Wariness for Muslim Site – NY Times, Sept. 2nd 2010
•Two-thirds of New York City residents want a planned Muslim community center and mosque
to be relocated to a less controversial site farther away from ground zero in Lower Manhattan,
including many who describe themselves as supporters of the project, according to a New York
Times poll
•Over all, 50 percent of those surveyed oppose building the project two blocks north of the
World Trade Center site, even though a majority believe that the developers have the right to do
so. Thirty-five percent favor it.
•Opposition is more intense in the boroughs outside Manhattan — for example, 54 percent in
the Bronx — but it is even strong in Manhattan, considered a bastion of religious tolerance,
where 41 percent are against it.
•The poll was conducted Aug. 27 to 31 with 892 adults. The margin of sampling error is plus or
minus three percentage points.
•More than half — 53 percent — of city residents with incomes over $100,000 back the center;
only 31 percent of those with incomes under $50,000 agree.
•Those under 45 are evenly divided (42 percent for, 43 percent against); among those over 45,
nearly 60 percent are opposed.
How to get a good sample
New York Poll Finds Wariness for Muslim Site – NY Times, Sept. 2nd 2010
•Richard Merton, 56, a real estate broker who lives on the Upper West Side of Manhattan,
exemplifies those mixed and seemingly contradictory feelings.
•“Freedom of religion is one of the guarantees we give in this country, so they are free to
worship where they chose,” Mr. Merton said. “I just think it’s very bad manners on their part to
be so insensitive as to put a mosque in that area.”……….
•The intensity of feeling is greater among opponents. Nearly three-quarters of respondents who
disapprove of the project say they feel strongly; only half of those who back it do so.
•“Give them an inch, they’ll take a yard,” Ms. Misetzis said. “They want to build a mosque
wherever they can. And once they start praying there, it is considered hallowed ground and can’t
be taken away. Ever. That’s why we’re having this tug of war between New Yorkers and the
Islamic people.”
•John Dewey, 65, of the Rego Park section of Queens, expressed his view in more practical terms.
•“We can’t say all Muslims are terrorists,” Mr. Dewey said. “There is a huge population of
Muslims throughout the world, and we will have to deal constantly with them in the future. If we
make enemies constantly, then we will constantly have war.”
New York Times Poll
How to get a good sample
Muslims and Islam Were Part of Twin Towers’ Life - NY Times, Sept. 11th 2010
Sometime in 1999, a construction electrician received a new work assignment from his union. The
man, Sinclair Hejazi Abdus-Salaam, was told to report to 2 World Trade Center, the southern of the
twin towers…..
Over the next few days, noticing some fellow Muslims on the job, Mr. Abdus-Salaam voiced an
equally essential question: “So where do you pray at?” And so he learned about the Muslim prayer
room on the 17th floor of the south tower.
On any given day, Mr. Abdus-Salaam’s companions in the prayer room might include financial
analysts, carpenters, receptionists, secretaries and ironworkers. There were American natives,
immigrants who had earned citizenship, visitors conducting international business — the whole
Muslim spectrum of nationality and race.
Leaping down the stairs on Sept. 11, 2001, when he had been installing ceiling speakers for a
reinsurance company on the 49th floor, Mr. Abdus-Salaam had a brief, panicked thought. He didn’t
see any of the Muslims he recognized from the prayer room. Where were they? Had they
managed to evacuate?...
“It is a shame, shame, shame,” Mr. Mamdouh, 49, said of the Park51 dispute. “Sometimes I wake up
and think, this is not what I came to America for. I came here to build this country together. People
are using this issue for their own agenda. It’s designed to keep the hate going.” - article
Final Review
Pentagon Sees Little Risk in Allowing Gay Men and Women to Serve Openly – NYT, Nov. 30, 2010
The Pentagon has concluded that allowing gay men and women to serve openly in the United
States armed forces presents a low risk to the military’s effectiveness, even at a time of war, and
that 70 percent of surveyed service members believe that the impact on their units would be
positive, mixed or of no consequence at all.
In an exhaustive nine-month study on the effects of repealing “don’t ask, don’t tell,” the 17-yearold policy that requires gay service members to keep their sexual orientation secret or face
discharge, the authors concluded that repeal would in the short run most likely bring about “some
limited and isolated disruption to unit cohesion and retention.”
Leaving aside those with moral and religious objections to homosexuality, the authors said the
concerns were “exaggerated and not consistent with the reported experiences of many service
members.” ……..
Mr. Obama, who campaigned for president on a promise to repeal the law, hailed the study.
“Today’s report confirms that a strong majority of our military men and women and their families
— more than two-thirds — are prepared to serve alongside Americans who are openly gay and
lesbian,” he said in a statement.
Final Review
Final Review
Don’t Ask Don’t Tell Study
Service member sample size and response rate.
The Service member sample population included 199,962 Active Duty Service members and
199,894 Reserve and National Guard members.
Survey administration for the Service member survey began on July 7,2010, and continued through
August 15, 2010. Five reminder notices were sent to Service member nonrespondents, with two of
the notices sent by both email and postal mail and the other three by email only.
The Defense Manpower Data Center (DMDC) used data from its personnel files to create sampling
frame strata. DMDC selected the samples from the frames and updated the sample files for both
surveys just prior to administration.
The process of weighting refers to the calculation of a sampling weight for each survey respondent.
Weighting is appropriate when the sample design is complex (that is, sample members do not all
have the same probability of selection) and there is nonresponse to the survey
The Service member survey had an overall weighted response rate of 28%.
How to get a good sample
Text Questions pg 75-77