Prediction Markets and the Wisdom of Crowds

David Pennock Yahoo! Research NYC

Research Outline

•

Prediction Markets Survey

• • •

What is a prediction market?

Examples Some research findings

•

The Wisdom of Crowds: A Story

Research Bet = Credible Opinion

Hillary Clinton will win the election “I bet $100 Hillary will win at 1 to 2 odds” • •

Which is more believable?

More Informative?

Betting intermediaries

• • •

Las Vegas, Wall Street, Betfair, Intrade,...

Prices: stable consensus of a large number of quantitative, credible opinions Excellent empirical track record

Research Example: March Madness

Research More Socially Redeemable Example

QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.

http://intrade.com

Screen capture 2007/05/18 QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.

Research A Prediction Market

•

Take a random variable, e.g.

Bird Flu Outbreak US 2007?

(Y/N) •

Turn it into a financial instrument payoff = realized value of variable

I am entitled to: $1 if Bird Flu US ’07 $0 if Bird Flu US ’07

Research Why?

• • Get information

price



probability of uncertain event (in theory, in the lab, in the field, ...more later) Is there some future event you’d like to forecast?

A prediction market can probably help

Research A Prediction Market

•

Take a random variable, e.g.

2008 CA Earthquake?

US’08Pres = Dem?

•

Turn it into a financial instrument payoff = realized value of variable

I am entitled to: $1 if = 6 $0 if  6 = 6 ?

Research

• • •

Aside: Terminology

Key aspect: payout is uncertain Called variously: asset, security, contingent claim, derivative (future, option), stock, prediction market, information market, gamble, bet, wager, lottery Historically mixed reputation

• •

Esp. gambling aspect A time when options were frowned upon

•

But when regulated serve important social roles...

Research Getting Information

• Non-market approach:

ask an expert How much would you pay for this?

$1 if = 6 $0 if  6 •

A: $5/36

 •

$0.1389

caveat: expert is knowledgeable

• • •

caveat: expert is truthful caveat: expert is risk neutral, or ~ RN for $1 caveat: expert has no significant outside stakes

Research Getting Information

• Non-market approach: • pay an expert

Ask the expert for his report r of the probability

• • • • • •

Offer to pay the expert

• $100 + log r if = 6 “logarithmic scoring rule”, a “proper” scoring rule • $100 + log (1-r) if  6

It so happens that the expert maximizes expected profit by reporting r truthfully caveat: expert is knowledgeable caveat: expert is truthful caveat: expert is risk neutral, or ~ RN caveat: expert has no significant outside stakes

Research Getting Information

• Market approach:

“ask” the public—experts & non experts alike —by opening a market:

I am entitled to: $1 if = 6 $0 if  6 • • • Let any person i submit a bid order:

an offer to buy q i units at price p i

Let any person j submit an ask order:

an offer to sell q j units at price p j

= 6

Match up agreeable trades (many poss. mechs...)

Non-Market Alternatives vs. Markets

 Opinion poll  Sampling  No incentive to be truthful  Equally weighted information  Hard to be real-time  Ask Experts  Identifying experts can be hard  Incentives  Combining opinions can be difficult  Prediction Markets  Self-selection  Monetary incentive and more  Money-weighted information  Real-time  Self-organizing

Non-Market Alternatives vs. Markets

 Machine learning/Statistics  Historical data  Past and future are related  Hard to incorporate recent new information  Prediction Markets  No need for data  No assumption on past and future  Immediately incorporate new information

Function of Markets

Risk Management 2:

 If is bad for me, I buy a bunch of $1 if $0 otherwise  If my house is struck by lightening, I am compensated.

Risk Management Examples

 Insurance  I buy car insurance to hedge the risk of accident  Futures  Farmers sell soybean futures to hedge the risk of price drop  Options  Investors buy options to hedge the risk of stock price changes

Financial Markets vs. Prediction Markets

Primary Secondary Financial Markets Social welfare (trade) Hedging risk Information aggregation Prediction Markets Information aggregation Social welfare (trade) Hedging risk

Giving/Getting Information

• What you can say/learn % chance that – Hillary wins – GOP wins Texas – YHOO stock > 30 – Duke wins tourney – Oil prices fall – Heat index rises – Hurricane hits Florida – Rains at place/time • Where – IEM, Intrade.com

– Intrade.com

– Stock options market – Las Vegas, Betfair – Futures market – Weather derivatives – Insurance company – Weatherbill.com

QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.

http://intrade.com

http://tradesports.com

Screen capture 2007/05/18 QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.

QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.

QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.

QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.

QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.

Research Intrade Election Coverage

QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.

QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.

Research

http://www.biz.uiowa.edu/iem

QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.

QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.

QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.

QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.

QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.

http://www.hedgestreet.com/

Screen capture 2007/05/18 QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.

QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.

QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.

QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.

http://www.wsex.com/

Screen capture 2007/05/18 QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.

Play money; Real predictions

http://www.hsx.com/

http://www.ideosphere.com

Cancer cured by 2010

http://us.newsfutures.com/

Machine Go champion by 2020

Research Yahoo!/O’Reilly Tech Buzz Game

http://buzz.research.yahoo.com/

More Prediction Market Games

• BizPredict.com

• CasualObserver.net

• FTPredict.com

• InklingMarkets.com

• ProTrade.com

• StorageMarkets.com

• TheSimExchange.com

• TheWSX.com

• • Alexadex, Celebdaq, Cenimar, BetBubble, Betocracy, CrowdIQ, MediaMammon,Owise, PublicGyan, RIMDEX, Smarkets, Trendio, TwoCrowds http://www.chrisfmasse.com/3/3/markets/#Play-Money_Prediction_Markets

Catalysts

• Markets have long history of predictive accuracy: why catching on now as tool?

• No press is bad press: Policy Analysis Market (“terror futures”) • Surowiecki's “Wisdom of Crowds” • Companies: – Google, Microsoft, Yahoo!; CrowdIQ, HSX, InklingMarkets, NewsFutures • Press: BusinessWeek, CBS News, Economist, NYTimes, Time, WSJ, ...

http://us.newsfutures.com/home/articles.html

Does it work?

 Yes, evidence from real markets, laboratory experiments, and theory  Racetrack odds beat track experts [Figlewski 1979]  Orange Juice futures improve weather forecast [Roll 1984]  I.E.M. beat political polls 451/596 [Forsythe 1992, 1999][Oliven 1995][Rietz 1998][Berg 2001][Pennock 2002]  HP market beat sales forecast 6/8 [Plott 2000]  Sports betting markets provide accurate forecasts of game outcomes [Gandar 1998][Thaler 1988][Debnath EC’03][Schmidt 2002]  Market games work [Servan-Schreiber 2004][Pennock 2001]  Laboratory experiments confirm information aggregation [Plott 1982;1988;1997][Forsythe 1990][Chen, EC’01]  Theory: “rational expectations” [Grossman 1981][Lucas 1972]  More later …

Example: IEM 1992

Example: IEM

Example: IEM

[Source: Berg, DARPA Workshop, 2002]

Example: IEM

AAAI’04 July 2004 MP1-35

[Source: Berg, DARPA Workshop, 2002]

Example: IEM

AAAI’04 July 2004 MP1-36

[Source: Wolfers 2004]

Speed: TradeSports

Contract: Pays $100 if Cubs win game 6 (NLCS) Price of contract (=Probability that Cubs win) Cubs are winning 3-0 top of the 8 th 1 out.

Fan reaches over and spoils Alou’s catch. Still 1 out.

The Marlins proceed to hit 8 runs in the 8 th inning Time (in Ireland) MP1-37 AAAI’04 July 2004

Research Does money matter? Play vs real, head to head

• •

Experiment 2003 NFL Season ProbabilitySports.com Online football forecasting competition

• • • • •

Contestants assess probabilities for each game Quadratic scoring rule ~2,000 “experts”, plus: NewsFutures (play $) Tradesports (real $)

• Used “last trade” prices • •

Results: Play money and real money performed similarly

•

6 th and 8 th respectively Markets beat most of the ~2,000 contestants

•

Average of experts came 39 th (caveat)

Electronic Markets

, Emile Servan Schreiber, Justin Wolfers, David Pennock and Brian Galebach

100 90 30 20 10 0 80 70 60 50 40 Prediction Accuracy

Research

100 Market Forecast Winning Probability and Actual Winning Probability TradeSports: Correlation=0.96

NewsFutures: Correlation=0.94

0 10 20 30 40 50 60 Trading Price Prior to Game 70 80 90 100 Data are grouped so that prices are rounded to the nearest ten percentage points; n=416 teams in 208 games 75 50 25 Prices: TradeSports and NewsFutures Fitted Value: Linear regression 45 degree line 0 0 20 40 60 NewsFutures Prices n=416 over 208 NFL games.

Correlation between TradeSports and NewsFutures prices = 0.97

80

Prediction Performance of Markets Relative to Individual Experts

0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Week into the NFL season

NewsFutures Tradesports 100

Research Does money matter? Play vs real, head to head

Probability Football Avg TradeSports (real-money) NewsFutures (play-money) Difference TS - NF Mean Absolute Error

= lose_price

[lower is better]

Root Mean Squared Error

= ?Average( lose_price 2 )

[lower is better] 0.443 (0.012) 0.476 (0.025) 0.439 (0.011) 0.468 (0.023)

0.436

(0.012)

0.467

(0.024) 0.003 (0.016) 0.001 (0.033)

Average Quadratic Score

= 100 - 400*( lose_price 2 )

[higher is better]

Average Logarithmic Score

= Log(win_price)

[higher (less negative) is better] 9.323 (4.75) -0.649 (0.027) 12.410 (4.37)

-0.631

(0.024)

12.427

(4.57)

-0.631

(0.025) -0.017 (6.32) 0.000 (0.035) Statistically: TS ~ NF NF >> Avg TS > Avg

average log score arbitrage closure AAAI’04 July 2004

Real markets vs. market games

HSX IEM MP1-41

Real markets vs. market games

HSX FX, F1P6 probabilistic forecasts expected value forecasts 100 50 20 10 5 489 movies 2 1 actual AAAI’04 July 2004 1 2 5 10 20 forecast source

F1P6 linear scoring F1P6 F1-style scoring

betting odds F1P6 flat scoring F1P6 winner scoring 50 estimate 100 MP1-42 avg log score

-1.84

-1.82

-1.86

-2.03

-2.32

Research A Wisdom of Crowds Story

• • •

ProbabilitySports.com

Thousands of probability judgments for sporting events

• • •

Alice: Jets 67% chance to beat Patriots Bob: Jets 48% chance to beat Patriots Carol, Don, Ellen, Frank, ...

Reward: Quadratic scoring rule: Best probability judgments maximize expected score 1

/7 Story Survey Research Opinion

Individuals Research

QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.

• •

Most individuals are poor predictors 2005 NFL Season

• • •

Best: 3747 points Average: -944 Median: -275 1,298 out of 2,231 scored below zero (takes work!)

Research Individuals

•

Poorly calibrated (too extreme)

•

Teams given < 20% chance actually won 30% of the time

•

Teams given > 80% chance actually won 60% of the time

Research The Crowd

• •

Create a crowd predictor by simply averaging everyone’s probabilities

• •

Crowd = 1/n(Alice + Bob + Carol + ... ) 2005: Crowd scored 3371 points (7th out of 2231) !

Wisdom of fools: Create a predictor by averaging everyone who scored below zero

• •

2717 points (62nd place) !

(the best “fool” finished in 934th place)

Research The Crowd: How Big?

More: http://blog.oddhead.com/2007/01/04/the-wisdom-of-the-probabilitysports-crowd/ http://www.overcomingbias.com/2007/02/how_and_when_to.html

Research

• •

Can We Do Better?: ML/Stats

[Dani et al. UAI 2006]

Maybe Not

• •

CS “experts algorithms” Other expert weights

• •

Calibrated experts Other averaging fn’s (geo mean, RMS, power means, mean of odds, ...)

•

Machine learning (NB, SVM, LR, DT, ...) Maybe So

• •

Bayesian modeling + EM Nearest neighbor (multi-year)

Research Can we do better?: Markets

Prediction Performance of Markets Relative to Individual Experts

0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Week into the NFL season

NewsFutures Tradesports