Document

Transcript Document

A Journalist’s Guide to Survey Research and Election Polls

Cliff Zukin * Rutgers University AAPOR The American Association for Public Opinion Research Washington Press Club Briefing 9/24/12 *This presentation reflects the author’s views. It has not been vetted or endorsed by AAPOR.

First, can we trust polls: 2004?

http://www.realclearpolitics.com/epolls/2004/president/us/general_election_bush_vs_kerry-939.html

Can we trust polls: 2008?

http://www.realclearpolitics.com/epolls/2008/president/national.html

What AAPOR Thinks You Should Know

Who did the poll? Who paid for it?

When was it done?

Who was sampled?

How were respondents contacted?

What is sampling error?

Why are data weighted?

How is the question worded?

How are the questions ordered?

What You Want to Know

3. Discuss Internet polls and their validity.

4. Any new polling techniques for 2012?

6. What are the most common errors made by political journalists in writing about polls?

Our Roadmap

(Through the Methods Box. See the handout. ) • Sampling—why polls work: Good and Bad Samples • Developments and Challenges facing the SR profession: cell phones, IVR (robo-polls), etc.

• Question wording & ordering • Things you want to be wary of…SR for journalists • Election Polling—How it’s different & why polls differ 6

SAMPLING: The Science of Polling

http://faculty.elgin.edu/dkernler/statistics/ch01/3-1.html

All Scientific Polling is Based on the Notion of

Sampling

• Bloody W

iners

I’VE NEVER BEEN CALLED

• Polls are estimates • A sample is drawn to represent an underlying population • The sample must be

representative

• If you don’t do this well, the rest doesn’t matter 8

Telling Good Polls and Bad Polls Apart

Starts with the Sample

• Probability Samples are GOOD – Scientific—known chance of inclusion – – Random, or thereabouts, selection Generalizable from sample to population • Non-Probability Samples are BAD – Not representative: 1,200 people are not a sample of anything – – – Person in the street ALL Internet opt-in surveys Any self-selection 9

Journalistic Standards

• In order to represent the population statistically, a survey should be based on a probability sample.

-NY Times

• Methodologically, in all or nearly all cases we require a probability sample, with high levels of coverage of a credible sampling frame. Self-selected or so-called “convenience” samples, including internet, e-mail, “blast fax,” call-in, street intercept, and non-probability mail-in

samples do not meet our standards for validity and reliability, and we recommend against reporting them

-ABC/Washington Post

Probabilities

,

Better Known as

The Odds:

Why Polls are Accurate…We’re the House

Dice Total

2 3 4 5 6 7 8 9 10 11 12

How you can make this total

1,1 1,2; 2,1 1,3; 2,2; 3,1 1,4; 2,3; 3,2; 4,1 1,5; 2,4; 3,3; 4,2; 5,1 1,6; 2,5; 3,4; 4,3; 5,2; 6,1 2,6; 3,5; 4,4; 5,3; 6,2 3,6; 4,5; 5,4; 6,3 4,6; 5,5; 6,4 5,6; 6,5 6,6

How many ways to do it

1 2 3 4 5 6 5 4 3 2 1

36 Total Outcomes For every 100 times, you should get this outcome this many X

3 5 8 11 14 17 14 11 8 5 3 11

Dice Outcome

(Or, the Central Limits Theorem) 20 15 10 5 0 2 3 4 5 6 7

Dice Total

8 9 10 11 12 12

Probabilities: Why Polls are Accurate or “We’re the House”

•

Chances of getting a 7 = 17% average or mean

(of 100 rolls (100%)/ 6 ways = 17%) •

Chances of getting between 6 and 8 = 45%

(14% + 17% +14% = 45%) •

Chances of getting a 5 through 9 = 67%

(11% + 14% + 17% +14% + 11% = 67%) •

So, the chances of being within 4 points on either side of the expected (mean) is

95%. Or the margin of error is + 4 % points

Probability(good) Samples Depend on Mathematical Axioms:

The Central Limits Theorem & the Law of Large Numbers

1. Life (or a sample of life) is distributed normally and generally fits a bell-shaped curve

Central Limits

2. The more observations you have, the more your distribution will approximate a bell-shaped curve

Large numbers

3. After a certain point, sample size matters, but population size doesn’t.

Law of M&Ms

At a Secret Location in Hacketstown NJ… 15

…I Counted the M&Ms in the Last Slide

Color

Tan Brown Red Orange Green Yellow

Total Number

4 8 9 10 7 2

Relationship between Sample Size and Sampling Error

12,0% 10,0%

10,0%

8,0%

7,1%

6,0% 4,0% 2,0% 0,0%

5,8% 5,0% 4,5% 4,1% 3,8% 3,5% 3,3% 3,2% 3,0% 2,9%

100 200 300 400 500 600 700 800 900 1 000

Sample Size

1 100 1 200 17

Sampling Error

• • • Sampling Error is a theoretical minimum It is only one kind of error, but it is quantifiable It applies not to the GAP between candidates, but to each point estimate

MOE ± 3

Obama 47 Obama 47 Obama 44 + ± 3 50 Romney 44 Romney 44 ± 3 Romney 41 – 47 18

SO WHAT? Why is this Important?

•

INTERNET POLLS ARE NON-PROBABILITY SAMPLES

(WITH ONE EXCEPTION) •

It is not possible to calculate sampling error on a Non-Probability Sample

•

And even with lipstick, a pig is still a pig Ipsos Poll for Reuters DAILY ELECTION TRACKING 9/11/12:

These are findings from an Ipsos poll conducted for Thomson Reuters from Sept 7 11, 2012. For the survey, a sample of 1,269 American registered voters (age 18 and over) was interviewed online. The precision of the Reuters/Ipsos online polls is

measured using a credibility interval. In this case, the poll has a credibility interval of plus or minus 3.1 percentage points for all respondents. 19

It’s Not Just Sampling Error: Total Survey Error Comprises 4 Sources

• Sampling Error: Sample, not population surveyed • Coverage Error: Sample may not map to pop • Measurement Error: Question wording/ordering • Non-response Error: Many decline to be surveyed 20

Trends in Survey Research

Trends in the Survey Research Industry

• Cell phones • IVRs (Robo-Polls) • Fewer high-quality polls • Challenge of interviewing a representative sample • Increased cost of finding respondents and dialing 22

Percentage of Wireless Only Adults 2001-2005 35% 30% 25% 20% 15% 10% 5% 0%

1,3%

Jan-Jun 2001

2,1%

Jan-Jun 2002 *Data Source: CDC/NCHS National Health Interview Survey

2,8%

Jan-Jun 2003

4,4% 6,7%

Jan-Jun 2004 Jan-Jun 2005 23

Percentage of Wireless Only Adults 2001-2011 35% 30% 25% 20% 15% 10%

30,2%

5% 0% Jan-Jun 2001 Jan-Jun 2002 Jan-Jun 2003 Jan-Jun 2004 Jan-Jun 2005 Jan-Jun 2006 Jan-Jun 2007 Jan-Jun 2008 Jan-Jun 2009 Jan-Jun 2010 Jan-Jun 2011 *Data Source: CDC/NCHS National Health Interview Survey 24

Percent of Adults Ages 18-29

Source: Current Population Survey and Pew Research Center surveys 25

RDD Samples without Cell Phones – Age Bias 26

Phone Status of Adults, Dec 2011

Among those households with telephones… • There are four times more “cell phone only” (32%) in the population as “landline only” (8%) • Taken together,

fully half

of the public is “cell phone only” (32%) or “cell phone mostly” (18%) • Did you know: Most IVR Polls do not include cell phones in their samples? This is definitely a question you should ask before reporting 27

Journalistic Standards: NY Times

• Interactive Voice Response Polls — Interactive voice response (IVR) polls (also known as "robo-polls") employ an automated, recorded voice to call respondents who are asked to answer questions by punching telephone keys.

The Times does not publish IVR polls.

• Internet Polls — Non-probability samples are commonly used in Internet polls, call-in polls, blast e-mail polls and a variety of others. The entire population does not have an equal chance of being contacted. Most Internet polls are based on panels of self-selected respondents.

The Times does not publish most Internet polls.

Manipulating Public Opinion

Or can you make a poll find anything you want?

NO!

BUT question

order wording

and question are extremely important

Good Questions:

1. Are simple, direct, clear to all, and avoid jargon 2. Don’t presume information 3. Are balanced 4. Ask about only one thing 5. Don’t tax respondent’s memory or cognitive ability 30

Bad Questions:

1. Are complex or presume information 2. Are leading or unbalanced 3. Are double-barreled or double negative 4. Are loaded through emotional or red flag words 5. Give biasing or unequal information in the Q stem 31

Bad Questions:

1. Are

complex

presume information

Do you favor or oppose a 1 percent tax by the United Nations on international air travel which by itself would produce half of all the revenue the organization needs and a savings for all members, which for the United States would be about half a billion dollars a year. The tax itself would add about a billion dollars to America’s international travel and air freight bills. Do you favor or oppose that?

32% Strongly favor 30% Somewhat favor 11% Somewhat oppose 25% Strongly oppose 2% Don't know/No answer 32

Bad Questions:

1. Are

complex

presume information

President Obama has announced changes to federal student loan programs that would: allow some college graduates to limit federal student loan repayments to 10% of their discretionary income starting in January, two years before the cap was due to take effect under federal law; forgive remaining debt on the federal loans after 20 years, five years earlier than under current law; and allow those with more than one student loan to consolidate their debt. Do you approve or disapprove of these changes?

Getting Real: Citizen Knowledge and Attentiveness

• 17% follow “news about political figures and events in Washington very closely” • 28% named John Roberts as Chief Justice out of four choices (7/10) • 43% know Republicans control the House (11/11) • 53% Know it is the Republicans who want to reduce the size of government Source: Pew Research Center 34

Bad Questions:

2. Are

leading

(or

unbalanced

Are you in favor of the mandatory drug testing of professional athletes? BIASED

Rather,

Do you favor or oppose the mandatory… UNBIASED

… professional athletes, or not? UNBIASED

Bad Questions:

2. Are

leading

(or

unbalanced

Should the federal government increase its role in providing loans with a goal of making sure everyone who puts in the effort to graduate from college can afford to do so?

• It’s okay for you to ask leading questions of experts, but not for us to ask them of random folks Totals may not add up to 100% due to rounding.

Bad Questions:

3. Are

double-barreled

double negative

Did you vote in the 2008 and 2010 elections – yes or no?

Bad Questions:

3. Are

double-barreled

double negative

Do you want to see less money spent on defense and more on social problems?

Bad Questions:

3. Are

double-barreled

double negative

(Agree or disagree) Should the federal government no longer be involved in college loans, and instead leave that entirely to the private sector?

Bad Questions:

4. Are

loaded

through

emotional

red flag

words

Fifty state legislatures have passed resolutions calling on Congress to pass the flag amendment and send it to the states for ratification. Do you think members of Congress who may personally oppose the amendment should vote in favor of it anyway so that “we the people” can decide the issue?

Bad Questions:

4. Are

loaded

through

emotional

red flag

words

Do you believe in killing unborn babies?

Bad Questions:

5. Through giving

one-sided information If it would result in increased opportunities for educating New Jersey citizens, would you favor or oppose building a new TV transmitter at liberty science center?

Bad Questions:

5. Through giving

one-sided information Many gambling experts believe that internet gambling will continue no matter what the government does to try to stop it. Do you agree or disagree that the federal government should allocate government resources and spend taxpayer money trying to stop adult Americans from gambling online?

Balance is necessary:

Some people say…while others say…

ALASKA, or (I’ll Ask her)

• Zogby: for environmental interests

Do you think oil companies should be allowed to drill for oil in America’s Arctic National Wildlife Refuge ?

Allow 38% Not allow 55% 17 percentage points against drilling 44

ALASKA, or (I’ll Ask her)

• Luntz: for energy interests

And right now, if you had to choose, do you strongly support, somewhat support, somewhat oppose, or strongly oppose the exploration and production of oil and natural gas from the ANWR as one of the many ways to increase our nation’s energy supply?

Support 51% Oppose 34% 17 percentage points pro-drilling 45

ALASKA, or (I’ll Ask her)

• Harris poll:

How much would you support or oppose the following items that might be on the agenda for the new Congress? Energy reform to allow companies the ability to drill for oil in certain areas such as the Alaska National Wildlife Refuge to our reliance on foreign oil?

decrease

Support 53% Oppose 47% 6 percentage points pro-drilling 46

Maxims of Question Wording

Question wording matters most when respondents know the least 2.

Projections and hypotheticals are generally unreliable 3.

You can never really underestimate how much people know about politics and government 4.

Giving information before asking the question is a double edged sword 5.

It must be balanced--as easy to disagree as to agree 6.

No ONE question tells the story 47

CONTEXT EFFECTS

Each Question Affects the Following Ones • How important is Social Security to you?

• Do you think Social Security will be there when you need it?

• Are you worried about private investment options for Social Security?

•

What is the most important problem facing the country?

WHAT’S THE TOPIC HERE….

1. “The First Amendment of the United States applies to everyone regardless of gender, race, religion, age profession, or point of view.” (96%) 2. “The First Amendment protects the right of individuals to create a private organization consisting of a specific group of people based on age, gender, race, ethnicity, or interest.” (77%) 3. “The First Amendment protects the right of organizations like the Boy Scouts, the Girl Scouts, and the National Association for the Advancement of Colored People to exist.” (91%) 49

IT’S BEEN IN THE NEWS RECENTLY….

4. “Individuals have a right to join a private group, club, or organization that consists of people who share the same interests and personal backgrounds as they do if they so desire.” (91%) 5. “Private organizations that are not funded by the government should be allowed to decide who becomes a member and who does not become a member on their own, without being forced to take input from other outside people or organizations.” (76%) 50

The Masters Polling Co…

6. “Private clubs that have members only of the same gender are simply a harmless way for similar individuals to get together and associate with each other.” (71%) 23. “Although currently there are no women members of the Augusta National Golf Club, the Golf Club does allow women to play on their golf course, and visit the course for the Masters Tournament. In other words, women are welcome to visit the Club and they often play as guests.”

“Knowing this, would say you support or oppose the Augusta National Golf Clubs decision to keep their membership policy as it is?”

(60%) 51

Some Common Problems in Reporting Polls

What Do I Write if it’s 50 – 46?

AP Style Book:

• If the difference between the candidates is more than twice the sampling error margin, then the poll says one candidate is leading. • If the difference is less than the sampling error margin, the poll says that the race is close, that the candidates are "about even." – (Do not use the term "statistical dead heat," which is inaccurate if there is any difference between the candidates; if the poll finds the candidates are tied, say they're tied.) • If the difference is at least equal to the sampling error but no more than twice the sampling error, then one candidate can be said to be "apparently leading" or "slightly ahead" in the race.

Be Wary of the “Slight” or “Modest” Trends

From the January 14, 2005, Hartford Courant:

• …Rell's stellar approval rating crept even higher in a poll released Thursday,

rising from 80 percent to 83 percent

in the first survey since her cancer surgery and State of the State address….

• Telephone poll of 1,287 voters from 1/7-10/05. Margin of error of +/- 2.7 percentage points. 54

Beware Small Groups and Subgroups:

Size Does Matter Was Jesus married?

Inadequate Sample Size:

Independent Soccer Mom Voters in New Jersey 56

What’s the Difference?

18 16 14 12 10 8 6 4 2 0 Republicans and Democrats are Further Apart than Ever: Average Difference in Republican and Democratic Attitudes 17 15 12 7 10 7 12 9 7 11 7 14 9 14 9 11 1987 1988 1990 1994 1997 Political and Policy Attitudes 1999 2002 2003 Social and Personal Attitudes

An Alternative Perspective:

100 90 80 70 60 50 40 30 20 10 0

Republicans and Democrats are Further Apart than Ever?

1987 1988 1990 1994 1997 1999 2002 2003 Political and Policy Attitudes Social and Personal Attitudes

Comparing Two Different Polls

“After a day of action at the Republican convention in Tampa, GOP presidential candidate Mitt Romney is looking better to voters. Romney’s image has received a five-point bump after the convention’s first day, according to data presented at a Wednesday breakfast sponsored by The Hill at the law offices of Holland & Knight.” “Romney had a 43% favorable and 44% unfavorable rating in nine battleground states heading into the convention, according to the average compiled by Real Clear Politics. A survey conducted by Wilson Perkins Allen Opinion Research in nine battleground states Tuesday evening found Romney’s favorable rating among likely voters to have jumped to 48%. His unfavorable rating dipped to 39%.” 59

Beware the

Costless Question

:

Would you favor or oppose preserving more open land in New Jersey?

• • • • Favor Oppose Depends how much (VOL) Don´t Know / Refused (VOL) 60

Don’t Report Answers to Silly Questions

“Sixty-seven percent of American Catholics believe John Paul II was the best pope the church has ever had, according to a new CNN poll of 254 Catholics.” 61

About WEIGHTING

• It’s necessary in almost all surveys • Everyone does it • It corrects for the problem of not interviewing people in the sample in correct proportion to their size in the population 62

Weighting Example

Education

Less than High School High School Some College College Graduate

Population

10 25 35 30

Sample

5 10 35 50

Weight

2.0

2.5

1.0

0.6

Limitations of Weighting

• You can only weight to KNOWN population parameters • Weighting to “attitudes” (like Party ID) is risky—they are not fixed attributes • You can’t weight to something in the future (like turnout) 64

Election Polling

A Special Case

ELECTION POLLING: A Special Case

• Time and field dates/procedures • Type of sample used: random or listed • Resp. selection/screening; Identifying likely voters • Question wording and ordering • Weighting 66

Time, Field Dates and Procedures

• Polls are a snapshot (yada yada) • Number of days in the field: – Callbacks and refusal conversions • Tracking polls/rolling averages 67

Types of Samples: Advantages and Disadvantages

• Random digit dialing (RDD) • Addressed based sampling (ABS) • Listed based sample (LBS) 68

Source: Pew Research Center 69

Respondent Selection, Screening and Determining “Likely Voters”

• Hard screen to get into the survey, or remove low likelihood of voting respondents after the fact?

• The problem: There is a consistent over-report of intentions to vote by the public What do you do if 80 percent of your eligible voters tell you they will definitely vote for President, but you’re looking at the table on the right?

Election Year

1992 1996 2000 2004 2008

Percent of Eligible Voters

60.6

52.6

54.2

60.1

61.1

Pew Likely Voter Questions

• • • • • • • • • How much thought have you given to the coming presidential election?

How closely have you been following news about the candidates?

Do you plan to vote in the presidential election? How certain are you that you will vote?

Rate your chance of voting in November on a scale of 10 to 1, with 10 being “definitely will vote” and 1 “definitely will not vote.” How often do you follow what’s going on in government and public affairs?

Have you ever voted in your precinct or election district?

How often would you say you vote?

Do you happen to know where people in your neighborhood go to vote?

In the last election, did things come up that kept you from voting or did you vote?

Likely Voter Computation

Score

9 8 1 0 7 6 3 2 5 4

Percent (%)

40 8 7 12 20 15 4 3 1 0

Cumulative %

40 48 55 67 87 92 96 99 100 100 Target 72

Likely Voter Vote Division

Candidate

Obama Romney Don’t know Margin

Turnout (All)

50 40 10 10

Top 67%

50 43 7 7

Top 55%

53 48 4 5 73

Question wording and ordering

• • • • • Asking the vote question: The presidential candidates and ….? …VP candidates, …party labels, …randomized names, (3 rd pty if applic)?

• Where the vote question is put: What came before it that it will be reactive to?

Weighting

• To what parameters, even on demographics?

• Weighting on party is controversial; Best practices” says this is NOT a good idea • But what’s a pollster to do if faced with: 75

Party ID

Democrat Independent Republican Totals

The Special Case of Party ID

Phone

33% 35% 30% 98%

IVR

36% 29% 34% 99%

Internet

46% 15% 36% 97% * Mark Blumenthal @ Huffington Post August 24, 2012 76

Why Election Polls May Vary by a Few Percentage Points

• • • • • • • • • Sampling error Length of field period Live interviewers vs. IVR Type of sample used Mode of administration Respondent selection Likely voter indices Question wording & ordering Weighting

Actually, it’s a wonder they are as close as they are!

Know Your “Rights”

AAPOR Minimum Disclosure Standards • • • • • Name of the survey sponsor Name that conducted the survey of organization Exact wording of questions being released Definition of population under study Description of sampling frame used to represent population under study • • • • • • • Explanation of respondent selection Total sample size Method or mode of data collection Dates and location of data collection Estimates of sampling error (if appropriate) Description of data weighting Use of sub-groups disclosed 78

We’re

(not from the government)

Here to HELP

•

AAPOR

–

3 Presidents

• • • Paul Lavrakas [email protected]

President Scott Keeter [email protected]

Rob Santos [email protected]

Past President President-elect –

Rapid Response (Election Polling) Team

• • • • Quin Monson [email protected]

Mike Traugott [email protected]

Rob Daves Cliff Zukin [email protected]

[email protected]