The Benefits and Risks of Using Statistics

Thought Questions

1. A recent newspaper article concluded that smoking marijuana at least three times a week resulted in lower grades in college. How do you think the researchers came to this conclusion?

Do you believe it? Is there a more reasonable conclusion?

2.

Theory: On average, men have lower resting pulse rates than women do.

How could you go about trying to prove or disprove that? Would it be sufficient to measure the pulse rates of one member of each sex? Two members of each sex?

The Benefits and Risks of Using Statistics

MMR and Autism Link – Excerpts from Bad Science, Ben Goldacre

• In 1998 Dr. Andrew Wakefield published his paper in the Lancet. The study and the press conference were actually covered in a fairly metered fashion, and also quite sparsely. • This was not unreasonable. The study itself was fairly trivial, a “case series report” of 12 people essentially a collection of 12 clinical anecdotes. • In 2001 and 2002 the scare began to gain momentum. Wakefield published a review paper in an obscure journal, questioning the safety of the immunization program, although with no new evidence. These received blanket media coverage. • The media repeatedly reported the concerns of this one man, generally without giving methodological details of the research, either because they found it too complicated, inexplicably, or because to do so would have undermined their story. • Newspapers and celebrities began to use the vaccine as an opportunity to attack the government and the health service, and of course it was the perfect story, with a charismatic maverick fighting against the system. •

There were elements of risk, of awful personal tragedy, and of course, the question of blame: whose fault was autism?

The Benefits and Risks of Using Statistics

MMR row doctor Andrew Wakefield struck off register – The Guardian, Monday 24 th , 2010

• Andrew, Wakefield, the doctor at the centre of the MMR scare, has been struck off the medical register after being found guilty of serious professional misconduct.

• The General Medical Council said he acted in a way that was dishonest, misleading and irresponsible while carrying out research into a possible link between the measles, mumps and rubella (MMR) vaccine, bowel disease and autism.

• He and the other doctors published a paper in the Lancet medical journal in February 1998 suggesting the measles virus might be linked to inflammatory bowel disease and play a role in autism.

• The paper, based on just eight case studies, conceded that no definite link had been found but

Wakefield, at a press conference, said he believed that instead of the triple MMR, children should be

given doses in single jabs, preferably a year apart.

• In February 1998, the same month the Lancet paper was published, he applied for ethical permission to run a trial of a new potential measles vaccine and set up a company called Immunospecifics Biotechnologies which would produce and sell it.

• Wakefield tried the new vaccine on the child without mentioning it in medical notes or telling the child's GP.

The Benefits and Risks of Using Statistics

The MMR vaccine controversy – winners, losers, impact and challenges

• In the 1940s notifications of measles cases in England and Wales exceeded 800,000 per year, with 1,145 deaths in 1941.

• The introduction of a measles vaccination in 1968 reduced the number of deaths to double figures but notifications remained high, between 50,000 and 100,000 cases per year, as a result of low vaccine uptake.

• In 1988 a triple vaccine called MMR, containing antigens to measles, mumps and rubella, was introduced into the childhood immunisation programme in UK.

• In 1996 uptake of MMR vaccine in England and Wales was over 95%, confirmed numbers of measles cases dropped to 112, and no deaths had been attributed to acute measles since 1992.

• However, by 2004 the epidemiological picture had changed. Uptake of MMR vaccine in 2005 was 80% in England and only 58% in London.

• In just the first quarter of 2006, confirmed cases of measles exceeded 180 and a teenager died of measles, becoming the first measles death in England and Wales for 14 years.

The Benefits and Risks of Using Statistics

The MMR vaccine controversy – winners, losers, impact and challenges

• The concepts of epidemiology, statistics, cause and effect, as defined in research are difficult to

convey to a non-scientific audience.

• The level of evidence from the Wakefield team’s case study series ranked very low in the medical research community’s acknowledged hierarchy of evidence. • Smith and Pell (2003) provide a wonderful illustration of the value of observational studies and the difficulty of attempting to apply the highest rigour of investigation to a problem. • They reported the lack of systematic randomised controlled trials to prove that parachutes prevent death and major trauma. Everyone can observe that parachutes are effective, since to leap out of plane without one, causes death or major trauma. There is a direct cause and effect relationship. • The parents who believe their children were damaged by MMR are convinced of the cause and effect relationship .

• The confounding factors that have to be considered in the cause and effect debate regarding MMR and autism are complex, and in many cases unknown.

The Benefits and Risks of Using Statistics

To conduct a statistical analysis properly, one must

1. Define the population of interest 2. Decide whether it is a sample survey, observational study or randomized experiment 3. Collect quality data - Get a representative and large enough sample

The Benefits and Risks of Using Statistics

1. Define the population of interest

• In conducting a statistical analysis, we select a sample from a population of interest, in order to make inferences about the population of interest. It is very important to clearly define the

population of interest. Why?

• To know the meaning of a statistic for certain, you must find out how a general term (i.e., ”crime”, ”poverty”, ”executive compensation”, “unemployment rate” ) is defined. • No matter how much effort we invest in collecting the data, the resulting conclusion is based on a statistical definition, and that definition must be explicitly stated.

For example, what does “unemployment rate” mean? Bureau of Labor Statistics definition of unemployed:

“if they do not have a job, have actively looked for work in the prior 4 weeks, and are currently available for work.” Unemployment Rate = # meeting above definition # in the labor force

The Benefits and Risks of Using Statistics

2. Decide whether it is a sample survey, observational study or randomized experiment

•

Sample Survey

A subgroup of a large population is questioned on a set of topics.

• No intervention or manipulation of the respondents, simply asked to answer some questions.

• •

Randomized Experiment

Measures the effect of manipulating the environment in some way.

Manipulation is assigned to participants on a random basis.

Explanatory variable :

the feature being manipulated. Example

:

Drug for Cholesterol Response variable = outcome of interest. Example

:

Cholesterol Randomization helps to make the groups approximately equal in all respects except for the explanatory variable .

The Benefits and Risks of Using Statistics

2. Decide whether it is a sample survey, observational study or randomized experiment Observational Study

Manipulation occurs naturally, not imposed.

Can’t assume the explanatory variable is the only one responsible for any observed differences in the response variable .

Example: Headline: Study: Smoking may lower kids’ IQ • Headline implication: a causal connection.

• Problem: results based on an observational study. Why not set up a randomized experiment?

• Moral:

Can’t make causal connections from observational studies .

The Benefits and Risks of Using Statistics

3. Collect quality data - Get a representative and large enough sample Health Studies Cited for Transplant Cuts Put Under the Knife, Wall Street Journal, Dec 18 th , 2010

Cash-strapped Arizona has drawn national scrutiny for its decision to drop Medicaid coverage for some organ transplants as the state tries to plug a $1 billion gap in its health-care budget for next year . • For lungs, a crux of the state's position was a 1995 study of 49 patients at the University of Washington, 25 of whom received transplants; the rest were waiting at the time of the study.

• The study concluded that transplant recipients would live half a year longer than those who didn't get a new lung, but the difference wasn't statistically significant— in part because the sample size was so small. • "I'm slightly horrified that they used a 15-year-old study to make this decision," says Scott D. Ramsey, professor at Washington's medical school and co-author of the 1995 paper. • "It's sort of like making a decision on whether to pay for 2010 AIDS drugs based on data from 1995 AIDS drugs." article

The Benefits and Risks of Using Statistics

Case Study 1.2: Does Aspirin Prevent Heart Attacks? Physicians’ Health Study (1988)

22,071 male physicians (40 to 84 years old).

• • Group 1: took ordinary aspirin tablet every other day.

Group 2: took placebo (looked like aspirin but no active ingredients).

Subjects did not know which group they were assigned.

Condition Heart Attack No Heart Attack Attacks per 1000

Aspirin Placebo 104 189 10,933 10,845 9.42 17.13

What is the population of interest?

Is the study a sample survey, observational study or randomized experiment?

What should we ask about how the data was collected?

The Benefits and Risks of Using Statistics

Examples of Misuses of Statistics 51% of Women Are Now Living Without Spouse – NY Times, January 2007

• For what experts say is probably the first time, more American women are living without a husband than with one, according to a New York Times analysis of census results.

•

In 2005 (Census Bureau Survey), 51 percent of women said they were living without a spouse, up from 35 percent in 1950 and 49 percent in 2000.

“

This is yet another of the inexorable signs that there is no going back to a world where we

can assume that marriage is the main institution that organizes people’s lives,” said Prof. Stephanie Coontz, director of public education for the Council on Contemporary Families, a nonprofit research group. “ -- article

Correction: February 14, 2007

• A front-page article and chart on Jan. 16 about the rising number of women in the United States living without spouses referred imprecisely to ages of the women included in the Census Bureau survey that was the basis of the finding. The women were 15 and older, not over 15. •

Had 15-year-olds not been counted, a majority of women would have been classified as living with a spouse.

The Benefits and Risks of Using Statistics

Examples of Misuses of Statistics When Polling Numbers Don’t Look Random - Wall Street Journal, July 1, 2010

• A prominent left-wing political website(Daily Kos) is renouncing polling it had commissioned and published for the past year and a half and has sued its former pollster (Research 2000) for allegedly not performing its contractual duties by fabricating some of the polling numbers it produced.

• What they found, they say, are numbers that wouldn’t result from a survey of randomly selected participants, which they claim could mean the numbers were fabricated. • Three of their findings were published in the report on Daily Kos. The first was that, when Research 2000 broke down a poll result by gender, the percentage was almost always either even or odd for

both men and women.

• For instance, earlier this month, 22% of men and 52% of women (both even) gave Nancy Pelosi a favorable rating; 43% of men and 59% of women (both odd) gave the same rating to Barack Obama. • Using widely accepted statistical theory, the authors write, the chance of such an anomaly happening 776 out of 778 times by random chance is less than one in one followed by 228 zeros.

The Benefits and Risks of Using Statistics

Textbook Questions – Pg 12-13

Q4. Explain what problems arise in trying to make conclusions based on a survey mailed to the subscribers of a specialty magazine. Q 12. Suppose the officials in the city or town where you live would like to ask questions of a representative sample of the adult population. Explain some of the characteristics the sample should have. For example, would it be sufficient to include only homeowners? How would you go about getting a representative sample?