Transcript Presentation

The Business of
Predictive Modeling
December 17, 2013
Christine Hofbeck, FSA, MAAA
Centroid Analytics, LLC
AGENDA
PART I -- INTRODUCTION
PART II – MODELING 101 (Basic Steps)
PART III – “GOLDEN QUESTION”
PART IV – OPERATIONAL CONSIDERATIONS
2
INTRODUCTION
Predictive modelling [sic] is the process by which a
model is created or chosen to try to best predict the
probability of an outcome. -- Wikipedia
VISUALIZE
In practice:
Identify
patterns/
segment
risks
3
Customers
Most Profitable Lines
Develop
business
rules
Improved
decision
OTHER
making
you are only limited by
your creativity
Potential Applications
OPTIMIZE
VISUALIZE
Operational Efficiency
Distribution Channels
Claims Management
Pricing / Reserves
Customers
Most Profitable Lines
or Products
Target Marketing
DATA
MINIMIZE
Risk
Fraud
4
OTHER
you are only limited by
your creativity
Potential Applications (Life)
1. Triage UW decisions; implement STP for (more) applicants
2. Decrease purchase of traditional UW requirements by
determining when they may not be necessary
3. Identify & target customers more likely to buy
4. Identify customers more likely to lapse – intervene if
profitable, allow unhealthies to lapse
5. Inforce book management
6. Identify most desirable agents
7. Smart customer handling
5
Predictors
A predictive model is made up of a number of predictors
(“independent variables”), which are data elements likely
to influence future behavior or results (“dependent
variable”).
DON’T USE ONE VARIABLE
the mean predicts the future
but doesn’t tell us why…
(“underfit”)
6
DON’T USE ALL VARIABLES
exactly replicates the past…
cannot predict the future
(“overfit”)
SEEK PARSIMONY
BASIC STEPS (Modeling 101)
Define &
Scope
Review &
Refine
Data Prep
Model
Build
Implementation
7
Model
Validation
Define & Scope
Exactly what
are we trying
to predict?
For whom/what
are we trying to
predict this? (“unit
of exposure”)
Do we have
the systems
capacity to
implement?
8
How will
the results
be used?
What is the
budget?
Consider IT,
staff, data
purchase,
training, etc.
How long do
we have to
build? To
implement?
Insource or
outsource?
Data Prep
* sometimes the most time intensive step of modeling
INTERNAL DATA
o
o
o
o
# years
accuracy
ability to access
primary key
EXTERNAL DATA
o
o
o
o
match rate
cost – to model
cost – to use
frequency of update
MODELING DATASET
9
Consider both expected & unexpected relationships –
creativity in data exploration can be the key
to your competitive edge!
Data Prep (cont’d)
1. COMBINE various data sources
2. CONVERT to desired exposure unit or format
3. CORRECT inaccurate data
4. INSPECT to remove variables:
- Too many blank values that cannot be imputed
-
All/most values the same
Data cannot be relied upon
Data will not be captured going forward
Legal advice not to use
5. BUCKET (“bin”) values
10
Model Build (cont’d)
UNIVARIATE ANALYSIS – test each variable one by
one to see which ones may be predictive.
MULTIVARIATE ANALYSIS – examine multiple
variables in different groups to obtain the
best, USABLE results – remember parsimony!
INTERACTIONS – which variables can be
combined into a “mega variable” to improve
results (i.e., does 1+1 = 1.5? does 1+1 = 3?)
11
Complicate the model (add variables, interactions)
and simplify the model (remove variables, bin)
to find the preferred combination.
Model Build (cont’d)
Various tests can be used to determine variable inclusion:
12
STATISTICAL
CONSISTENCY
JUDGMENT
P-values
Cramer’s V
Confidence
intervals
Type III tests
Of patterns -
Apply business
knowledge to
assess whether
suggested
relationships
make sense
Over time
Over random
parts of a
dataset
Model Validation
ACTUAL vs. EXPECTED-- how close did we get?
Generally, a subset of the data is withheld during the
modeling process for validation:
OUT OF TIME
OUT OF SAMPLE
withhold most
recent data
withhold randomly
generated % of records
Model validation graphs are useful for communicating
model performance to non-technical audiences.
13
Model Validation – Sample Chart
0.9
0.8
Outcome
0.7
0.6
Actual
0.5
Expected
0.4
0.3
0.2
1
2
3
4
5
Decile
14
6
7
8
9
10
Implementation
BUSINESS RULES
What decisions will be made based on the prediction?
May vary by location, business, rate group, etc.
SYSTEM BUILD
Scoring engine (collects data & calculates predictions)
Decision tool (executes business rules)
User interface
TRAINING
Anyone who will interact with the model must
understand what it does and why
15
Review & Refine
REPORTING
How close did we get to the goal?
How far did we exceed it?
Multiple reporting packages required for varied audiences,
for example:
 Executives – highlights in aggregate by zone, business
unit, product
 Actuaries – detailed results by variable, state, rate group
 Marketing – by broker/agent, location
 Underwriting – by underwriter as a performance measure
Frequency of update – weekly, monthly, quarterly, yearly?
Method of calculation – automated? ad hoc?
Review & Refine (cont’d)
MODEL UPDATES
WHY?
 As target customer is attained, characteristics of inforce
book will change
 Business goals/strategies may change
 New data may become available
 Tolerance for certain characteristics may change
HOW?
 Update current variable relativities (“recalibrate”)
 Start over - search for more predictive variables (“recast”)
HOW OFTEN?
17
Advantages of Modeling Over
Traditional Approaches
1. Many additional and often unconventional variables may
be examined
2. Modeling a particular variable controls for the effects of
other included variables – we don’t risk double counting or
attributing effects to the wrong variables
3. Traditional approaches segment data into smaller
categories which impact credibility
4. Interactions are introduced
18
The above advantages can lead to improved accuracy,
enhanced business and strategic benefits, more reliable
assumptions, improved risk mitigation, etc.
THE “GOLDEN QUESTION”
Through brainstorming, feedback loops, and data
review, determine what single characteristic
(“golden question”) will define your target
19
OPERATIONAL CONSIDERATIONS
1. Executive & cross-functional support
2. Time/cost versus depth of investigation
3. Strategic modeling process
4. Cross-functional involvement throughout build
5. Thorough training
20
Executive & Cross-Functional Support
If target users don’t support the model,
they will resist using it.
Gaining complete support can be difficult:
1. Resistance to change
2. Concern that model results will highlight
current deficiencies
3. Lack of understanding of predictive models
21
Support (cont’d)
My position will be
eliminated if a model
is now used to select
risks. My expertise
must not be important
to the company.
The model will suggest
that my current method
is incorrect, which will
reflect poorly on my
performance/reputation.
I will
have to take
on additional work
associated with
new processes. My
workflow will
double (triple).
22
I already have an
established plan. I
know who our target
customer is.
We’ve always
done it this way,
and it’s worked.
I don’t see a
reason to change
anything.
I don’t know how
to explain this to
a broker/agent
so I don’t want
to use it.
I found one
outlier so the
model must be
wrong.
Time/Cost vs. Depth of Investigation
The process of building and implementing a model can
typically be quite lengthy – longer than most expect
•
•
•
•
•
•
Simpler Study (3-12 months)
Results more conservative
Perhaps internal data only
Generous binning
Limited interactions
May be appropriate if goal is
a general sense of direction
23
OR
•
•
•
•
More thorough investigation
Additional time
Additional development cost
Possible greater payoff
through enhanced
segmentation and data
exploration
Remember that a simple model does not
necessarily indicate a simple study!
Strategic Modeling Process
TARGET PREDICTION/USE
o Ensure target is appropriate for the intended use
o While many ideas are interesting, you may wish to focus
on those which are actionable
STATISTICAL SIGNIFICANCE vs. ULTIMATE IMPACT
o The most statistically significant model may not be the
most impactful
o Consider ease of implementation, repeatability, updates
o Identify when “less is more”!
FLEXIBILITY
o Allow for unexpected insights which could lead to
unanticipated changes in business strategy or process
o Sometimes the insights gained from the journey will prove
more important than the planned goal
Cross-Functional Involvement
Data, product & IT experts, legal advisors, and model
users must remain engaged throughout the model build
Insight from functions
Insight to functions
Eases training and implementation
Keep modelers apprised of changes in strategy
25
Legal considerations around certain variables
Thorough Training
The model isn’t done when it’s done.
Who will provide the training?
Who is most appropriate to provide training?
Modeling team
General training team
Functional experts
Consulting team*
Other
No clear answer – but this must be thoughtfully
considered and appropriately executed to reap the full
benefits of the model which was built
26
*Consider what information may be shared (non-proprietary)
Discussion/Q&A
Remember…
Modeling is a complete business strategy
NOT just a mathematical process
So how will YOU use predictive modeling
to improve your business?
27
Christine Hofbeck, FSA, MAAA
Centroid Analytics, LLC
[email protected]
908.884-4103 (c)
908.574-5351 (w)
www.centroidanalytics.com