Review Part 3 Science

Final Exam

• • • • • • • 17 December 2013 (Tuesday) 18:30-20:30 In the Gymnasium 20 Questions All short answer 5 marks each Worth 20% of the course grade

Deadlines

Participation due: 10 th of December.

Causation ≠ Correlation

Causation does not imply correlation. If A and B are correlated there are several possibilities: • • • • A causes B B causes A C causes A and C causes B A and B are only accidentally correlated

Common Cause!

From the Daily Mail

Lede: “[new] findings, published in the latest online edition of the journal Appetite, show the way we perceive tasty treats like chocolate cake is just as important as the calorie count when it comes to expanding waistlines.”

From the Daily Mail

“They recruited almost 300 volunteers, aged from 18 to 86, and quizzed them on their eating habits and whether they were trying to lose weight. They also asked them if eating chocolate cake made them feel happy or guilty.

The results showed 27 per cent associated it with guilt and 73 per cent with celebration. When the researchers looked at weight control 18 months later, they found those riddled with guilt had gained significantly more.”

Probably Common Cause

Maybe people who eat unhealthily feel more guilty about eating chocolate. After all, they can see the harm they’re doing to themselves.

And maybe people who eat very well don’t feel guilty having chocolate.

Coincidence

In 1979, two researchers, Nancy Wertheimer and Ed Leeper, published an article alleging that the incidence of childhood leukemia was higher in Denver neighborhoods that were near electric power lines.

The “Texas Sharp Shooter”

Suppose I stand in front of a barn. I have a machine gun with me, and I am blindfolded. I shoot wildly at the barn for several minutes.

Afterward, I walk up to the barn. I find a spot where three bullets are very close together, and I paint a target around them. “Look!” I say, “at what an excellent marksman I am!”

Power Lines and Cancer

The power lines study is just like this. The researchers found places near power lines and looked at all the health problems anyone had in the area.

Of all the health problems what’s the chance that one is accidentally correlated with power lines?

Coincidence

High enough!

Later studies showed there was no relationship.

Example Study

“A randomized controlled clinical trial of auricular acupuncture in smoking cessation.” Wu TP, Chen FP, Liu JY, Lin MH, Hwang SJ.

Example Study

“A randomized controlled clinical trial of auricular acupuncture in smoking cessation.” Wu TP , Chen FP, Liu JY, Lin MH, Hwang SJ.

Person who runs the lab, Had the idea for the paper.

Example Study

“A randomized controlled clinical trial of auricular acupuncture in smoking cessation.” Wu TP, Chen FP , Liu JY , Lin MH , Hwang SJ .

People who did the work (Most work first).

RCT

“We conducted a prospective, randomized, controlled trial using auricular acupuncture for smoking cessation in 131 adults who wanted to stop smoking. Thirteen subjects withdrew from the study and 118 subjects were included in the final analyses (mean age, 53.7 +/- 16.8 years; 100 males, 18 females).”

RCT

We conducted a prospective, randomized, controlled trial using auricular acupuncture for smoking cessation in 131 adults who wanted to stop smoking. Thirteen subjects withdrew from the study and 118 subjects were included in the final analyses (mean age, 53.7 +/- 16.8 years; 100 males, 18 females).

The Importance of Randomization

Improper randomization procedures on average exaggerated effects by 41%. This is an average result, so improper randomization often leads to exaggerations that are even larger than 41%.

“Garbage In, Garbage Out”

This means meta-analyses conducted on poor quality trials should not be trusted!

RCT

“The treatment group (n = 59) received auricular acupuncture in Shen Men, Sympathetic, Mouth and Lung points for 8 weeks. The control group (n = 59) received sham acupuncture in non smoking-cessation-related auricular acupoints (Knee, Elbow, Shoulder and Eye points). The enrolled subjects were then followed monthly for 6 months after stopping the acupuncture treatment.”

Auricular Acupuncture

Sham Acupuncture

LOTR Meme

RCT

“Between both groups before acupuncture treatment, there was no significant difference with regard to gender, mean age, education level, and mean values for the age at which smoking started, smoking duration, daily number of cigarettes smoked and nicotine dependent score.”

Internet Polls

Internet polls are not trustworthy. They are biased toward people who have the internet, people who visit the site that the poll is on, and people who care enough to vote on a useless internet poll.

Representative Samples

The opposite of a biased sample is a representative sample. A perfectly representative sample is one where if n% of the population is X, then n% of the sample is X, for every X.

For example, if 10% of the population smokes, 10% of the sample smokes.

Random Sampling

One way to get a representative sample is to randomly select people from the population, so that each has a fair and equal chance of ending up in the sample.

Results

“At the end of treatment, cigarette consumption had significantly decreased in both groups, but only the treatment group showed a significant decrease in the nicotine withdrawal symptom score.”

Fagerstrom Test

https://outcometracker.org/library/FTND.pdf

Statistically Significant

P-Values

One way to characterize the significance of an observed correlation is with a p-value.

The p-value is the probability that we would observe our data on the assumption that the null hypothesis is true. p = P(observations/ null hypothesis = true)

P-Values

Obviously lower p-values are better, that means your observed correlation is more likely to be true.

In science we have an arbitrary cut-off point, 5%. We say that an experimental result with p < .05 is statistically significant.

Statistical Significance

What does p < .05 mean?

It means that the probability that our experimental results would happen if the null hypothesis is true is less than 5%.

According to the null hypothesis, there is less than a 1 in 20 chance that we would obtain these results.

Only 5% chance it’s one of these.

Results

“Smoking cessation rate showed no significant difference between the treatment group (27.1%) and the control group (20.3%) at the end of treatment. There was also no significant difference in the smoking cessation rate between the treatment group (16.6%) and the control group (12.1%) at the end of follow-up.”

Statistically Significant

Results

The decrease was real, the difference was not!

Conclusion

“Our results showed that auricular acupuncture did not have a better efficacy in smoking cessation compared to sham acupuncture.” *

*P-Value Fallacy

Actually, the researchers found that if there was no relation between acupuncture and smoking cessation, then the results they observed were typical (95% likely).

The Fallacy Fallacy

If you show that an argument is fallacious, you have not shown that it’s conclusion is false.

If you show that a relationship is not statistically significant in your experiment, you have not shown that it does not exist!!!

Effect Size

One NAEP analysis of 100,000 American students found that science test scores for men were higher than the test scores for women, and this effect was statistically significant These results are unlikely if the null hypothesis, that gender plays no role in science scores, were true.

Effect Size

However, the average difference between men and women on the test was just 4 points out of 300, or 1.3% of the total score.

Yes, there was a real (statistically significant) difference. It was just a very, very small difference.

Effect Size

One way to put the point might be: “p-values tell you when to reject the null hypothesis. But they do not tell you when to care about the results.”

The Evidential Heirarchy

1. Systematic reviews and meta-analyses 2. Randomized controlled trials 3. Cohort studies 4. Case-control studies 5. Cross sectional surveys 6. Anecdotes

Meta-Analysis

A meta-analysis is an analysis of analyses. In clearer terms, it is a study that looks at lots of different experiments that have been conducted on the same problem, and tries to “put together” all of the findings.

Blobbogram

Jadad Scores

1. Do the researchers say that the study is randomized? Yes: 1 point, No: 0 points.

If yes, was the method described and is it appropriate? Yes: 1 point, No: -1 point.

2. Do they say it’s double-blind? Y: +1, N: 0.

If yes, described and appropriate? Y: +1, N: -1.

3. Did it describe the people who dropped out of the study? Y: +1, N: 0.

Churnalism

“19% of newspaper stories and 17% of broadcast stories were verifiably derived mainly or wholly from PR material, while less than half the stories we looked at appeared to be entirely independent of traceable PR.” – Lewis et. Al (on the course website)

Sadly…

Most science journalists don’t know anything about science (they have journalism degrees), and cannot tell an odds ratio from a risk ratio or a real scientific journal from a joke.

You are better critical thinkers than the people who write the news!

Journalistic Embargoes

Even scientists are not above manipulating newspapers for undeserved fame and sometimes even money.

There is a practice whereby a scientist(s) will release a forthcoming article to the press, but not allow them to talk about it with anyone before it is published, and there is a press conference. This is called an “embargo.”

The Point of Embargoes

The idea is that when the press conference happens, the story will be big news: everyone will want to publish newspaper articles about it. So before the press conference, journalists will use the academic paper or study to write an article. But they can’t get quotes or other information from other experts, because they can’t talk about it. They can only present one side of the story, the researcher’s side.

The Link

One famous case involved researchers who had a fossil they wanted to present as “the Missing Link,” even though that makes no sense in evolutionary theory. They had a book deal and a TV special all lined up!

Not a Link

Of course, that sounds really exciting: “They discovered the missing link!!!” But when other scientists actually had a chance to read the academic article, they found that the evidence that the fossil was a direct ancestor of humans not supported at all.

Self-Censorship

Another way reporters get the news wrong is that they don’t report it. Sometimes news angers those in authority, and journalists bow to their wishes.