Transcript Movie Recommendations and Netflix Prize

Netflix Prize and
Heritage Health Prize
Philip Chan
Cash Prizes to Stimulate Research
 Ansari X Prize for Private Spaceflight (2004)
[$10M]

100 km above earth twice within 2 weeks
 DAPRA Grand Challenge (2005) [$2M]

autonomous vehicle: 131 miles in 10 hours
 Archon X Prize for Genomics (2006) [$10M]

map 100 human genomes in 10 days
Cash Prizes to Stimulate Research
 Netflix Prize (2006) [$1M]

Recommend movies with 10% improvement
 Heritage Health Prize (2011) [$3M]

Days in hospital next year with 0.4 error
Netflix Prize
netflixprize.com
Netflix Prize
 Task
 Given customer ratings on some movies
 Predict customer ratings on other movies
 If John rates
 “Mission Impossible” a 5
 “Over the Hedge” a 3, and
 “Back to the Future” a 4,
 how would he rate “Harry Porter”, … ?
 Performance
 Error rate (accuracy)
Cash Award
 Grand Prize



$1M
10% improvement
by 2011 (in 5 years)
 Progress Prize


$50K per year
1% improvement
Intellectual Property
 Netflix has a non-exclusive license to the
algorithm
 Authors tell the world what the algorithm is
Participation
 51K contestants
 41K teams
 186 countries
Leader Board
 Started on Oct 2, 2006
 Improvement by the top algorithm






after a week: ~0.9%
after two weeks: ~4.5%
after a month: ~5%
after a year: ~8.4%
after two years: ~9.4%
July 26, 2009 (less than 3 years): 10%
Winner
 BellKor’s Pragmatic Chaos


7 members
Merger of 3 teams

BellKor
 AT&T Labs, USA & Yahoo! Research, Israel

PragmaticTheory
 telecommunications, Canada

BigChaos
 started a company, Austria

A combination of different algorithms
Runner-up
 The Ensemble


~30 members
“last-minute” merger



teams had 30 days to beat the first team that
crossed the 10% threshold
same accuracy
behind by 20 minutes!
Heritage Health Prize
heritagehealthprize.com
Health Care
 71M individuals admitted to US hospitals
each year
 Unnecessary admissions cost $30B
Heritage Provider Network
 Has a network of doctors in California
 Can we identify earlier those most at risk and
ensure they get the treatment they need?
 Can we reduce unnecessary
hospitalizations?
Heritage Health Prize
 Launch
 http://www.youtube.com/watch?v=GuZ8nkpygAs
 Given patient data

Predict how many days a patient will spend in
a hospital in the next year
 The prediction helps develop strategies to
reduce emergencies and hence
hospitalizations
Grand Prize
 $3M
 At most 0.4 in error (~0.5 day)
 By Apr 4, 2013 [2 years]
 $500K Consolation Prize

not below 0.4 error
Milestone Prizes
 top 2 performers at each milestone
 Aug 31, 2011

$30K, $20K
 Feb 13, 2012

$50K, $30K

http://www.youtube.com/watch?v=pkmkNnGyihY
 Sep 4, 2012

$60K, $40K
Performance of Algorithms
 Prediction Error Rate (RMSLE)
n
 (real  prediction)
i
2
i
i
n
 where
 real = log ( actual # of days + 1 )
 prediction = log ( predicted # of days + 1 )
 Prediction error threshold = 0.4 (~0.5 day)
Intellectual Property
 Exclusive license to Sponsor

and participant’s own use
 Algorithms not previously published
 Use of data sets is for the competition only

written consent for other purposes
Data Sets
 Training and validation data sets

For participants to design algorithms
 Feedback data set

For calculating standings on Leaderboard
 Scoring data set

For determining winners for prizes
 http://www.heritagehealthprize.com/c/hhp/Data
Data (in CSV format)
 Members Data (113K members)
 Claims Data (2.7M claims)
 Drug Count Data (818K prescriptions)
 Lab Count Data (361K labs)
 Outcome Data (76K in Y2, 71K in Y3)
 Target (71K in Y4 for prediction)
 Total ~264 MB (including other files)
Members Data
 MemberID
 AgeAtFirstClaim
 Sex
Claims Data
 MemberID
 ProviderID
 Vendor ID
 PCP (Primary care physician) ID
 Year
 Specialty (of physician/vendor?)
 PlaceSvc (place of service)

office, outpatient hospital, inpatient hospital, …
 PayDelay (between service and payment)
Claims Data [continued]
 LengthOfStay (in hospital)
 DSFS (days since first claim)
 PrimaryConditionGroup (diagnostic
categories)
 CharlsonIndex (affect of diseases on illness)
 ProcedureGroup (intervention categories)
 SupLOS (supplement to LengthOfStay)
 1 if LenghtOfStay is NULL because of deidentificaiton
Drug Count Data
 MemberID
 Year
 DSFS (Days since first service)
 DrugCount (unique prescription drugs)
Lab Count Data
 Member Id
 Year
 DSFS (Days since first service)
 LabCount (unique lab or pathology tests)
Outcome Data
 MemberID
 DaysInHospital_Y2 (claims in Y1)

ie, Predict Y2 based on Y1
 DaysInHospital_Y3 (claims in Y2)
 ClaimedTruncated

1 if members with “truncated” claims
Using Other Data?
 Yes


Freely available to anyone (public source)
URL needs to be published to the forum
 Except for

demographic, socioeconomic or clinical
information about the members
Naive Algorithms
 For predicting the number of Days in Hospital
in the next year
 Posted as “benchmarks” on the Leaderboard
Always Predict 15 (max)
 Everyone goes to the hospital for at least 15
days
Always Predict 15 (max)
 Everyone goes to the hospital for at least 15
days
 RMSLE = 2.628062
 550+% over threshold
Always Predict Zero
 no one goes to the hospital
Always Predict Zero
 no one goes to the hospital
 RMSLE = 0.522226
 31% over threshold
Predict Random Values
 between 0 and 15
Predict Random Values
 between 0 and 15
 RMSLE = 0.752297
 88% over threshold
Always Predict Average
 Average ~= 0.209179
Always Predict Average
 Average ~= 0.209179
 RMSLE = 0.486459
 22% over threshold
Leader Board
 Competition started on Apr 4, 2011 with
partial data
 All data were released on June 4, 2011
 Sep 9, 2011
Leader Board
 Competition started on Apr 4, 2011 with
partial data
 All data were released on June 4, 2011
 Sep 9, 2011


RMSLE: 0.456384
~14.1% over threshold
 Aug 29, 2012


RMSLE: 0.450426
~12.6% over threshold
Teams
 Sep 9, 2011


914 teams
6021 entries
 Aug 29, 2012

1292 teams
Considerations
 Accurate Prediction

algorithms
 Efficiency


time
space
Teams
 Form your own teams

www.heritagehealthprize.com
 Join my team


CSE 4403 Independent Study
CSE 5801 Independent Research
THANK YOU
www.heritagehealthprize.com