Transcript Title of Presentation

Exploring Predictions with a
Powerful Research Tool
Research Showcase
Waley Liang
Analytic Science, Lead Scientist
FICO
© 2014 Fair Isaac Corporation. Confidential.
This presentation is provided for the recipient only and cannot be reproduced or shared without Fair Isaac Corporation’s express consent.
Why Should You be Interested in an Internal Tool?
► Direct
Benefits: functionalities migrate to products that you can use,
e.g., FICO® Model Builder, FICO® Analytic Modeler Scorecard
► Indirect
Benefits: FICO data analysts are using this tool to deliver improved
analytic solutions and services for our customers
2
© 2014 Fair Isaac Corporation. Confidential.
Agenda
►Prediction
Exploration Platform (PEP)
► Motivation
► Key Analytic
► Key
►Use
Capabilities
Benefits
Cases
►Demonstration
►Discussion
3
© 2014 Fair Isaac Corporation. Confidential.
Motivation
Universe of
Machine
Learning
Algorithms
FICO Data
Scientists
PEP
FICO Products
4
© 2014 Fair Isaac Corporation. Confidential.
Key Analytic Capabilities of PEP
Predictive Algorithms
►
Random Forest (RF)
►
Stochastic Gradient Boosting (SGB)
►
Fuzzy Segmented SGB (FSSGB)
►
Sparse Logistic Regression (SLR)
Support for Experiments
►
Algorithm/methods comparisons
►
Hyper-parameter grid searches
Model Training Acceleration
►
5
Black-box Buster Reports
►
Understanding variable importance and
interactions
►
Visualization of complex variable behaviors
Via parallel/distributed computing
© 2014 Fair Isaac Corporation. Confidential.
Key Benefits of PEP
Improves Prediction
►
ML algorithms match or beat traditional models
Importance of 100’s of variables for prediction quickly assessed
► Complex relations can be visualized
► Inform Scorecard segmentation decisions
►
Generates insights
Boosts productivity
Extensible for the future
6
© 2014 Fair Isaac Corporation. Confidential.
►
Fast from data to powerful predictions, insights
► GUI based, no need for coding
►
User feedback guides addition of new algorithms
Use Cases
© 2014 Fair Isaac Corporation. Confidential.
Auto Insurance Claim Frauds
Experiment
►
Developed several TEM and
a baseline Logistic
Regression model in PEP
►
Compared models on a holdout dataset
Boost fraud detection rate at
1% review rate
Challenge
8
►
© 2014 Fair Isaac Corporation. Confidential.
►
Boosted fraud detection rate
from 7% up to 14%
►
Sped up modeling with
hyperparameter search
Results/Benefit
s
Credit Card Behavior Scoring
Experiment
►
►
Assess the benefit of a
custom model versus a
pooled model.
Challenge
9
© 2014 Fair Isaac Corporation. Confidential.
Trained 2 models using PEP
►
►
►
Model 1 on Client data
Model 2 on Pooled data
Compared performance
(AUC, KS) on the client’s outof-time data
►
Custom model KS = 0.713
► Pooled model KS = 0.709
► Confirmed the pooled
segmentation is effective
► Used ~1 week with PEP vs.
~9 weeks with traditional
approaches
Results/Benefit
s
Credit Card Attrition
Experiment
►
►
Understand the effect of
temporary inactivity on future
attrition risk.
Trained 3 models in PEP
based on recency and
frequency w/ and w/o
interaction
►
►
►
►
Challenge
10
© 2014 Fair Isaac Corporation. Confidential.
2 predictors (additive)
2 predictors (interaction)
1000’s predictors (interaction)
Compared lift and profits at
top 5% of population
►
Higher lift (6  6.8  7.5)
and profits due to interaction
and additional predictors
►
Gained understanding of
interactions via visualizations
Results/Benefit
s
Demonstration
© 2014 Fair Isaac Corporation. Confidential.
We’d Love to Hear Your Feedback
► Are
► Do
you using these or similar ML algorithms today?
you see a role for increase in ML usage in your organization in the next 2 years?
► Which
is the most important problem to tackle with ML?
►
Credit scoring
► Fraud
► Marketing
► Other (what?)
► Are
you interested in using ML as an exploratory tool only?
► Are
you interested in implementing the analytic solution from ML in a production
environment?
► Would
►
12
you prefer to carry out ML using a GUI?
With coding?
© 2014 Fair Isaac Corporation. Confidential.
Thank You!
Waley Liang
[email protected]
© 2014 Fair Isaac Corporation. Confidential.
This presentation is provided for the recipient only and cannot be reproduced or shared without Fair Isaac Corporation’s express consent.
Learn More at FICO World
Related Sessions
►Improvements in Recommendation Systems
►Addressing Attrition: Ultra-Dynamic Multi-Dimensional Attrition Analytics with Tree Ensemble Models
Products in Solution Center
►Analytic Capabilities: Modeling Techniques and Innovations
►FICO® Model Builder
Experts at FICO World
►Gerald Fahner
►Michael Cohen
Blogs
►http://ficolabsblog.fico.com/
14
© 2014 Fair Isaac Corporation. Confidential.
Please rate this session online!
Waley Liang
[email protected]
15
© 2014 Fair Isaac Corporation. Confidential.