Transcript Title of Presentation

Unified Scorecard Performance Monitoring
The Design, Implementation, and Use of a Standardized
Monitoring Program
Jerrod Appenzeller
Vice President, Credit Risk Analytics Consultant
Wells Fargo
© 2014 Fair Isaac Corporation. Confidential.
This presentation is provided for the recipient only and cannot be reproduced or shared without Fair Isaac Corporation’s express consent.
Creating a standardized scorecard
reporting package to efficiently distribute
results, proactively manage risk, and meet
the heightened demands of reviewers
2
© 2014 Fair Isaac Corporation. Confidential.
Agenda
►Project
Overview
►Monitoring
Reports
►Standardization
►Monitoring
►Factors
►Use
Considerations
Practices
Impacting Results
in Driving Decisions
►Summary
3
© 2014 Fair Isaac Corporation. Confidential.
Project Overview
4
© 2014 Fair Isaac Corporation. Confidential.
Project Overview
Goals
► Meet
heightened model risk monitoring
expectations
► Create
efficiencies in generating and
distributing results
► Expedite
model reviews and use changes
► Increase
speed and efficacy of model and
policy decisions
5
© 2014 Fair Isaac Corporation. Confidential.
Actions
► Centralized
model monitoring function
► Inventoried,
aligned, and documented
assessment methodologies
► Clarified
the expectations of all reviewing
parties
► Created
a streamlined, uniform, and
flexible monitoring package
Monitoring Reports
► Report
Types
► Common
6
Metrics
© 2014 Fair Isaac Corporation. Confidential.
Monitoring Reports
► Inventorying
of types of reports and documentation of metrics
► Clearly
defining (unifying definitions where needed) and reducing the number of charts
and metrics produced helped expedite model reports and discussions
► Educational slides for each report, chart, and metric were appended to each quarterly
report deck for business users who may only see credit scoring jargon once each quarter
7
© 2014 Fair Isaac Corporation. Confidential.
Example Score
► All
charts, metrics, etc. are generated using simulated scorecard output
Example Score Valid Values
10
Highest Risk
8
© 2014 Fair Isaac Corporation. Confidential.
20
30
40
50
60
70
80
90
100
Lowest Risk
Report Types
Front-End Reports
► Track
distributional changes of the scored population along with credit decisions
made using the score
9
Report
Brief Description
Insight Provided
Population Stability
Shows scorecard-level distributional chances in
the scored population
How is the applicant or account credit quality
changing over time
Characteristic Analysis
Shows scorecard characteristic-level
distributional changes in the scored population
The source of the changes in the scored
population
Final Score
Displays credit decisions made by score
Decisioning and over-ride process
Credit Chronology
A written log of changes in credit policy or
scorecard uses
A timeline of changes that may impact
monitoring results
© 2014 Fair Isaac Corporation. Confidential.
Report Types
Back-End Reports
► Track
ability of scorecard to distinguish between outcomes once actual
performance is known
► Origination
10
scores may also track distributional changes of booked population
Report
Brief Description
Insight Provided
Booked Distribution
Origination scores: distributional changes in the
booked population
How booked credit quality (portfolio risk) is
changing over time
Early Performance
Monitors scorecard lift using earlier
performance target and window than targeted
An early indication that a scorecard may be
losing ability to distinguish outcomes
Vintage Performance
Monitors scorecard lift by booking vintage
How scorecard's ability to distinguish outcomes
has changed over time
Log Odds Analysis
Displays changes in shape of the log odds by
score/score bin
Changes in overall odds (shifts) and odds
between score levels (rotation)
Target Performance
Graphical representations and quantification of
score's ability to distinguish outcomes
Initial score's (non-refreshed) ability to
distringuish actual targeted performance
© 2014 Fair Isaac Corporation. Confidential.
Common Metrics
Population Stability Index (PSI) and Information Value (IV)
► Quantifies
difference between two distributions—the higher the more different
► Often
referred to as PSI when monitoring distributional changes and IV when
used to quantify the differences in outcome distributions by score or characteristic
11
© 2014 Fair Isaac Corporation. Confidential.
Common Metrics
PSI and IV Calculation
Difference in
empirical densities
In IV formula, this component is
often referred to as the Weight of
Evidence (WOE).
Log of ratio of
empirical densities
𝑛
𝑷𝑺𝑰 =
𝑖=1
𝐶𝑖 /𝐶
𝐶𝑖 /𝐶 − 𝐵𝑖 /𝐵 ∙ 𝑙𝑛
𝐵𝑖 /𝐵
𝑛
𝑰𝑽 =
𝑖=1
𝐶 = 𝐶𝑢𝑟𝑟𝑒𝑛𝑡
𝐵 = 𝐵𝑎𝑠𝑒𝑙𝑖𝑛𝑒
𝐺𝑖 /𝐺 − 𝐵𝑖 /𝐵 ∙ 𝑙𝑛
𝑛 = 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑏𝑖𝑛𝑠
𝐺𝑖 /𝐺
𝐵𝑖 /𝐵
𝐺 = 𝐺𝑜𝑜𝑑𝑠
𝐵 = 𝐵𝑎𝑑𝑠
𝐺𝑖
Note WOE is not a log odds ratio as G and B are from
different probability distributions. Log odds would be: 𝑙𝑛 𝐵𝑖
© 2014 Fair Isaac Corporation. Confidential.
Common Metrics
Kolmogorov-Smirnov (K-S)
► The
maximum difference between cumulative outcome distributions by score or
characteristic
KS
𝑲𝑺 𝑺𝒕𝒂𝒕𝒊𝒔𝒕𝒊𝒄 = 𝑀𝑎𝑥(𝐵𝑖 − 𝐺𝑖 )
𝐵𝑖 = 𝑟𝑖𝑠𝑘 𝑑𝑒𝑠𝑐𝑒𝑛𝑑𝑖𝑛𝑔 𝑐𝑢𝑚𝑢𝑙𝑎𝑡𝑖𝑣𝑒 𝑏𝑎𝑑𝑠 𝑖𝑛 𝑠𝑐𝑜𝑟𝑒 𝑏𝑖𝑛 𝑖
𝐺𝑖 = 𝑟𝑖𝑠𝑘 𝑑𝑒𝑠𝑐𝑒𝑛𝑑𝑖𝑛𝑔 𝑐𝑢𝑚𝑢𝑙𝑎𝑡𝑖𝑣𝑒 𝑔𝑜𝑜𝑑𝑠 𝑖𝑛 𝑠𝑐𝑜𝑟𝑒 𝑏𝑖𝑛 𝑖
13
© 2014 Fair Isaac Corporation. Confidential.
Common Metrics
Receiver Operating Characteristics (ROC) Curve
► The
ROC curve plots the ecdf* of goods to the ecdf of bads at each score or
characteristic level
*ecdf: empirical cumulative distribution function
14
© 2014 Fair Isaac Corporation. Confidential.
Common Metrics
Gini Coefficient Calculation Using ROC
𝑮𝒊𝒏𝒊 𝑪𝒐𝒆𝒇𝒇𝒊𝒄𝒊𝒆𝒏𝒕 =
𝐴𝑈𝐶 − 0.5
= 2(𝐴𝑈𝐶 − 0.5)
0.5
𝑛
𝐴𝑈𝐶 =
𝐺𝑖+1 − 𝐺𝑖
𝑖=0
𝐵𝑖 + 𝐵𝑖+1
2
𝐹𝑖𝑟𝑠𝑡 𝑝𝑜𝑖𝑛𝑡 𝑖𝑠 𝑡ℎ𝑒 𝑜𝑟𝑖𝑔𝑖𝑛 0, 0 : 𝐺0 = 𝐵0 = 0
𝐴𝑈𝐶 = 𝐴𝑟𝑒𝑎 𝑈𝑛𝑑𝑒𝑟 𝑡ℎ𝑒 𝐶𝑢𝑟𝑣𝑒
𝐺 = 𝑅𝑖𝑠𝑘 𝐷𝑒𝑠𝑐𝑒𝑛𝑑𝑖𝑛𝑔 𝐶𝑢𝑚𝑢𝑙𝑎𝑡𝑖𝑣𝑒 𝐺𝑜𝑜𝑑𝑠
𝐵 = 𝑅𝑖𝑠𝑘 𝐷𝑒𝑠𝑐𝑒𝑛𝑑𝑖𝑛𝑔 𝐶𝑢𝑚𝑢𝑙𝑎𝑡𝑖𝑣𝑒 𝐵𝑎𝑑𝑠
𝑛 = 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑠𝑐𝑜𝑟𝑒 𝑏𝑖𝑛𝑠
© 2014 Fair Isaac Corporation. Confidential.
AUC
Common Metrics
Cumulative Accuracy Profile (CAP)
► The
CAP plots the ecdf* of the total to the ecdf of bads at each score or
characteristic level
16
© 2014 Fair Isaac Corporation. Confidential.
Perfect classification
point (BadRate, 1)
Common Metrics
Accuracy Ratio (AR) Calculation Using CAP
𝑨𝒄𝒄𝒖𝒓𝒂𝒄𝒚 𝑹𝒂𝒕𝒊𝒐 =
𝐴𝑈𝐶 − 0.5
𝐴𝑈𝐶 − 0.5
=
𝐴𝑈𝑃 − 0.5 0.5 − (0.5 ∗ 𝐵𝑎𝑑𝑅𝑎𝑡𝑒)
𝑛
𝐴𝑈𝐶 =
𝑇𝑖+1 − 𝑇𝑖
𝑖=0
𝐵𝑖 + 𝐵𝑖+1
2
𝐹𝑖𝑟𝑠𝑡 𝑝𝑜𝑖𝑛𝑡 𝑖𝑠 𝑡ℎ𝑒 𝑜𝑟𝑖𝑔𝑖𝑛 0, 0 : 𝐺0 = 𝐵0 = 0
𝐴𝑈𝐶 = 𝐴𝑟𝑒𝑎 𝑈𝑛𝑑𝑒𝑟 𝑡ℎ𝑒 𝐶𝑢𝑟𝑣𝑒
𝐴𝑈𝑃 = 𝐴𝑟𝑒𝑎 𝑈𝑛𝑑𝑒𝑟 𝑡ℎ𝑒 𝑃𝑒𝑟𝑓𝑒𝑐𝑡 𝐶𝑙𝑎𝑠𝑠𝑖𝑓𝑖𝑐𝑎𝑡𝑖𝑜𝑛 𝐶𝑢𝑟𝑣𝑒
𝑇 = 𝑅𝑖𝑠𝑘 𝐷𝑒𝑠𝑐𝑒𝑛𝑑𝑖𝑛𝑔 𝐶𝑢𝑚𝑢𝑙𝑎𝑡𝑖𝑣𝑒 𝑇𝑜𝑡𝑎𝑙
𝐵 = 𝑅𝑖𝑠𝑘 𝐷𝑒𝑠𝑐𝑒𝑛𝑑𝑖𝑛𝑔 𝐶𝑢𝑚𝑢𝑙𝑎𝑡𝑖𝑣𝑒 𝐵𝑎𝑑𝑠
𝑛 = 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑠𝑐𝑜𝑟𝑒 𝑏𝑖𝑛𝑠
© 2014 Fair Isaac Corporation. Confidential.
Standardization Considerations
► Summarizing
► Baseline
► Metric
18
Data
and Benchmark Selection
Variants
© 2014 Fair Isaac Corporation. Confidential.
Summarizing Data
Binning Approaches and Considerations
► Binning Approaches
► Equal
depth based on volume quantiles
► Equal width based on equal score or characteristic ranges
► Algorithmically derived bins are generated based on maximizing distinctiveness of a
performance variable (often subject to volume constraints)
► Metrics
may be calculated using binned data based on the volume of data present
at each valid value of a score or characteristic
► Gini, AR,
KS, and other metrics utilizing cumulative data may be closely
approximated using binned data, but simple code or software can generate these
exactly
19
© 2014 Fair Isaac Corporation. Confidential.
Summarizing Data
Equal Depth Versus Equal Width Binning
Example: summarizing the volume of a score ranging from 0 to 500 with 10 bins
Equal depth breaks at each decile of volume*
*Ties may make bins not perfectly even in depth
20
© 2014 Fair Isaac Corporation. Confidential.
Equal width binning breaks every 50 points
Baseline and Benchmark Selection
Baseline Selection
► Baselines
are utilized to determine shifts/stability of scorecard and
characteristics
► Baseline
selection
► fixed
baseline
► rolling baseline
► Potential
reasons to change a fixed baseline
► redevelopment,
re-binning, and/or re-weighting
► new adjuster to model output
► enduring shift due to change in score use, credit policy, marketing, economic
conditions, etc.
21
© 2014 Fair Isaac Corporation. Confidential.
Baseline and Benchmark Selection
Benchmark Selection
► Scorecard
performance is often not tracked in isolation, but relative to a
benchmark
► Potential
benchmarks:
► The
same score’s performance on prior/development data (check for deterioration)
► An alternate score’s performance on the same population (champion-challenger
approach)
► Both
22
© 2014 Fair Isaac Corporation. Confidential.
Metric Variants
Lorenz Curve
► A Lorenz
curve has been defined as either a ROC or CAP chart
► This
also may lead to differences in the calculation of Gini in the case where the AUC was
calculated using a CAP curve but still divided by 0.5 (part Gini, part AR)
► Competing
definitions are understandable, as various articles contain varying definitions*
► Additionally,
Gini and AR have been said to be equivalent in the past by the Basel
Committee in 2005 and Standard and Poor’s in 2010, but are now widely accepted to be
distinct ratios^
► We
decided to use the more broadly-accepted labels of CAP (total by bads) and ROC
(goods by bads)
*one example of a stark contradiction can be found between page 5 here: http://marlenemueller.de/publications/PaperKKM.pdf
and page 5 here: http://is.muni.cz/do/econ/soubory/konference/vasicek/20667044/Rezac.pdf—these articles differ on both Lorenz
and ROC
^based on our prior definitions: AR = Gini / (1—BadRate)
23
© 2014 Fair Isaac Corporation. Confidential.
Metric Variants
Alternate Gini
► Some
decision tree algorithms auto-grow using a Gini metric that has the same
meaning, but the lower the better (inverted)
► Know
and understand how metrics are calculated when using third-party modeling
software to avoid misinterpretation
► In
addition to assisting non-technical audiences, alternate definitions were another
reason for definitional/educational slides being appended to reporting decks
24
© 2014 Fair Isaac Corporation. Confidential.
Monitoring Practices
► Population
Stability
► Characteristic Analysis
► Target
25
Performance
© 2014 Fair Isaac Corporation. Confidential.
Population Stability
► Binned
to equal width for some scores to track policy impacts
► Binned
to equal depth (of the baseline) for others to systematically ensure sufficient volume
► Baseline
fixed based on development data time period, time period after a use change, or
time period after most recent economic change (vendor score) depending on the score
► PSI
thresholds are based on common subjective definitions
PSI
0.00 -< 0.03
0.03 -< 0.10
0.10 -< 0.25
0.25 -< 0.50
0.50 -< inf
26
© 2014 Fair Isaac Corporation. Confidential.
Implication
No significance
Weak shift
Moderate shift
Strong shift
Extremely strong shift
Characteristic Analysis
Summary characteristic stability heat-map
Score 1 Seg A
Score 1 Seg B
Score 2 Seg A
Score 2 Seg B
Score 2 Seg C
Char 1
0.05
0.12
0.01
0.51
0.09
Char 2
0.07
0.02
0.02
0.11
0.09
Char 3
0.03
0.00
0.01
0.01
0.00
Char 4
0.03
0.08
0.12
0.26
0.21
Char 5
0.00
0.01
0.02
0.00
0.00
Weighted-Average point shift with bars in cells or off to the side
Score 1 Seg A
Score 1 Seg B
Score 2 Seg A
Score 2 Seg B
Score 2 Seg C
27
© 2014 Fair Isaac Corporation. Confidential.
Char 1
-1.51
-2.53
-2.82
-5.13
-0.20
Char 2
1.77
0.87
0.82
-0.45
1.21
Char 3
-0.30
-0.06
-0.06
-0.42
0.29
Char 4
-1.45
-1.33
-1.93
-2.20
-0.37
Char 5
0.75
0.23
0.19
0.07
0.32
Scorecard Target Performance
► Chose
ROC over CAP (Gini over AR) due to ease in interpretation under varying bad rates
► Examples
below are the same score over varying bad rates with the same AR and Gini
Validation Bad Rate:
Development Bad Rate:
28
© 2014 Fair Isaac Corporation. Confidential.
4.1%
81.1%
Validation and Development Gini
and AR are the same
Scorecard Target Performance
► Track,
► Both
at a minimum, scorecard-level Gini and KS for performance metrics
can be visually represented by ROC chart
KS
29
© 2014 Fair Isaac Corporation. Confidential.
Factors Impacting Results
30
© 2014 Fair Isaac Corporation. Confidential.
Factors Impacting Results
► Metric
contradictions may necessitate multiple or hierarchical
predetermined performance thresholds
Score 1
Gini = 0.77
KS = 0.60
Score 2
Gini = 0.72
KS = 0.62
31
© 2014 Fair Isaac Corporation. Confidential.
Factors Impacting Results
► Policy
impacts (cherry-picking)
Acquisition scores:
performance assessment
suffers a bias due to policy
overlays
Policy
All scores 30 and under must
meet some other criteria
Volume Impacts
30:
-3% of goods
-25% of bads
20:
-5% of goods
-55% of bads
10:
-10% of goods
-85% of bads
Overall: -0.9%
Performance Impact
KS:
-9.7%
Gini:
-9.3%
32
© 2014 Fair Isaac Corporation. Confidential.
Factors Impacting Results
► Strict
cutoff ROC animation
Acquisition scores:
performance assessment
suffers a bias due to the
conditional nature of the scored
population available for
assessment
33
© 2014 Fair Isaac Corporation. Confidential.
Cutoff
10
20
30
40
50
60
70
80
90
Percent Change
Volume
Gini
KS
0%
-2%
-3%
-2%
-8%
-9%
-6%
-17%
-21%
-14%
-30%
-35%
-24%
-45%
-45%
-38%
-56%
-75%
-90%
-52%
-62%
-68%
-100%
-56%
-60%
-60%
-100%
Factors Impacting Results
► Strict
cutoff ROC animation
Acquisition scores:
performance assessment
suffers a bias due to the
conditional nature of the scored
population available for
assessment
34
© 2014 Fair Isaac Corporation. Confidential.
Cutoff
10
20
30
40
50
60
70
80
90
Percent Change
Volume
Gini
KS
0%
-2%
-3%
-2%
-8%
-9%
-6%
-17%
-21%
-14%
-30%
-35%
-24%
-45%
-45%
-38%
-56%
-75%
-90%
-52%
-62%
-68%
-100%
-56%
-60%
-60%
-100%
Factors Impacting Results
► Strict
cutoff ROC animation
Acquisition scores:
performance assessment
suffers a bias due to the
conditional nature of the scored
population available for
assessment
35
© 2014 Fair Isaac Corporation. Confidential.
Cutoff
10
20
30
40
50
60
70
80
90
Percent Change
Volume
Gini
KS
0%
-2%
-3%
-2%
-8%
-9%
-6%
-17%
-21%
-14%
-30%
-35%
-24%
-45%
-45%
-38%
-56%
-75%
-90%
-52%
-62%
-68%
-100%
-56%
-60%
-60%
-100%
Factors Impacting Results
► Strict
cutoff ROC animation
Acquisition scores:
performance assessment
suffers a bias due to the
conditional nature of the scored
population available for
assessment
36
© 2014 Fair Isaac Corporation. Confidential.
Cutoff
10
20
30
40
50
60
70
80
90
Percent Change
Volume
Gini
KS
0%
-2%
-3%
-2%
-8%
-9%
-6%
-17%
-21%
-14%
-30%
-35%
-24%
-45%
-45%
-38%
-56%
-75%
-90%
-52%
-62%
-68%
-100%
-56%
-60%
-60%
-100%
Factors Impacting Results
► Strict
cutoff ROC animation
Acquisition scores:
performance assessment
suffers a bias due to the
conditional nature of the scored
population available for
assessment
37
© 2014 Fair Isaac Corporation. Confidential.
Cutoff
10
20
30
40
50
60
70
80
90
Percent Change
Volume
Gini
KS
0%
-2%
-3%
-2%
-8%
-9%
-6%
-17%
-21%
-14%
-30%
-35%
-24%
-45%
-45%
-38%
-56%
-75%
-90%
-52%
-62%
-68%
-100%
-56%
-60%
-60%
-100%
Factors Impacting Results
► Strict
cutoff ROC animation
Acquisition scores:
performance assessment
suffers a bias due to the
conditional nature of the scored
population available for
assessment
38
© 2014 Fair Isaac Corporation. Confidential.
Cutoff
10
20
30
40
50
60
70
80
90
Percent Change
Volume
Gini
KS
0%
-2%
-3%
-2%
-8%
-9%
-6%
-17%
-21%
-14%
-30%
-35%
-24%
-45%
-45%
-38%
-56%
-75%
-90%
-52%
-62%
-68%
-100%
-56%
-60%
-60%
-100%
Factors Impacting Results
► Strict
cutoff ROC animation
Acquisition scores:
performance assessment
suffers a bias due to the
conditional nature of the scored
population available for
assessment
39
© 2014 Fair Isaac Corporation. Confidential.
Cutoff
10
20
30
40
50
60
70
80
90
Percent Change
Volume
Gini
KS
0%
-2%
-3%
-2%
-8%
-9%
-6%
-17%
-21%
-14%
-30%
-35%
-24%
-45%
-45%
-38%
-56%
-75%
-90%
-52%
-62%
-68%
-100%
-56%
-60%
-60%
-100%
Factors Impacting Results
► Strict
cutoff ROC animation
Acquisition scores:
performance assessment
suffers a bias due to the
conditional nature of the scored
population available for
assessment
40
© 2014 Fair Isaac Corporation. Confidential.
Cutoff
10
20
30
40
50
60
70
80
90
Percent Change
Volume
Gini
KS
0%
-2%
-3%
-2%
-8%
-9%
-6%
-17%
-21%
-14%
-30%
-35%
-24%
-45%
-45%
-38%
-56%
-75%
-90%
-52%
-62%
-68%
-100%
-56%
-60%
-60%
-100%
Factors Impacting Results
► Strict
cutoff ROC animation
Acquisition scores:
performance assessment
suffers a bias due to the
conditional nature of the scored
population available for
assessment
41
© 2014 Fair Isaac Corporation. Confidential.
Cutoff
10
20
30
40
50
60
70
80
90
Percent Change
Volume
Gini
KS
0%
-2%
-3%
-2%
-8%
-9%
-6%
-17%
-21%
-14%
-30%
-35%
-24%
-45%
-45%
-38%
-56%
-75%
-90%
-52%
-62%
-68%
-100%
-56%
-60%
-60%
-100%
Factors Impacting Results
► Strict
cutoff ROC animation
Acquisition scores:
performance assessment
suffers a bias due to the
conditional nature of the scored
population available for
assessment
42
© 2014 Fair Isaac Corporation. Confidential.
Cutoff
10
20
30
40
50
60
70
80
90
Percent Change
Volume
Gini
KS
0%
-2%
-3%
-2%
-8%
-9%
-6%
-17%
-21%
-14%
-30%
-35%
-24%
-45%
-45%
-38%
-56%
-75%
-90%
-52%
-62%
-68%
-100%
-56%
-60%
-60%
-100%
Factors Impacting Results
► Low
bad rates for performance tracking
► Remove/expand
reporting segmentation (not development segmentation)
► Use an “ever” as opposed to “current” bad definition (may not be appropriate for curetarget model)
► Lower days-past-due (e.g. from 90 to 60 DPD)
► Expand performance/look-back window (e.g. from 24 MOB to 12-24 MOB or 2-24 MOB)
► Expand bad definition (include BK, charge-offs, modifications, etc.)
► Whenever
expanding performance definition, we should also track the target
performance definition used to develop the scorecard
43
© 2014 Fair Isaac Corporation. Confidential.
Factors Impacting Results
► Development
► Reporting
44
segmentation (segmented models)
segmentation (post-development segmentation)
© 2014 Fair Isaac Corporation. Confidential.
Factors Impacting Results
► Binary
Classification Models
“Triangular” approach measures the accuracy as implemented
Gini = 0.41
45
© 2014 Fair Isaac Corporation. Confidential.
“Derived” approach measures the accuracy the model is
capable of producing by varying thresholds
Gini = 0.68
Factors Impacting Results
► Other
Considerations
► Continuous
performance
► Secondary performance target
► Inferred performance
► On-top score adjustments
46
© 2014 Fair Isaac Corporation. Confidential.
Use in Driving Decisions
► Modeling
► Policy
47
Decisions
Decisions
© 2014 Fair Isaac Corporation. Confidential.
Use in Driving Decisions
Enumerative Versus Analytic Reporting
“How are we using our results?”
Enumerative: a study or report aimed at describing and categorizing data—quantification
Analytic: a study or report which attempts to find underlying causes and/or improve a
process (e.g. model risk management, credit risk management)—results in action
48
© 2014 Fair Isaac Corporation. Confidential.
Modeling Decisions
Model Performance
► Review
required when PSI is above a predetermined threshold
We could re-class and re-weight characteristics if we need to capture a “new normal” for a specific
characteristic
► We could nullify a variable for performance or shift (e.g. a flag where everyone now essentially receives
the same value) or data reasons (internally or externally sourced) and reweight
►
► Review
►
49
required when performance drops by a predetermined fixed level or percentage
We may redevelop if overall performance is poor
© 2014 Fair Isaac Corporation. Confidential.
Policy Decisions
Setting Score Cutoff
► Performance
check around potential cutoffs
► KS
► Distance-to-Perfect
► Confusion
► FPR
► Cash
and TPR
flow based approaches
► Assign
50
matrix to compare good/bad volume tradeoffs
dollar value to net revenue and expected credit loss
© 2014 Fair Isaac Corporation. Confidential.
Policy Decisions
Performance Around Cutoff
► We
can explore specific cutoff points by looking at KS and (Euclidean) Distance-to-Perfect
Score low-to-high risk
Score low-to-high risk
51
© 2014 Fair Isaac Corporation. Confidential.
Policy Decisions
Confusion Matrix
Policy
Accept
a
c
Reject
b
d
𝑻𝒓𝒖𝒆 𝑷𝒐𝒔𝒊𝒕𝒊𝒗𝒆 𝑹𝒂𝒕𝒆 =
𝑑
𝑐+𝑑
𝑭𝒂𝒍𝒔𝒆 𝑷𝒐𝒔𝒊𝒕𝒊𝒗𝒆 𝑹𝒂𝒕𝒆 =
𝑏
𝑎+𝑏
True Positive Rate (TPR)
Actual
Good
Bad
False Positive Rate (FPR)
© 2014 Fair Isaac Corporation. Confidential.
Policy Decisions
Cash Flow Approaches
► We
checked the model’s performance for any given cutoff level and considered volume
tradeoffs using the confusion matrix
► Now,
we may be able to assign a net value to a, b, c, and d to determine the expected
benefit of competing policies
Accepting defaulters (c)
Benefits
Costs
Policy
Actual
53
© 2014 Fair Isaac Corporation. Confidential.
Good
Bad
Accept
a
c
Reject
b
d
Policy Decisions
Expanding Cutoff
► Issue
of current interest
► Wells
Fargo expansion to 600 FICO from 640 for FHA mortgages
► http://www.bloomberg.com/news/2014-05-01/easier-homeowner-credit-compelling-wells-
fargo-mortgages.html
► http://www.nationalmortgagenews.com/dailybriefing/Wells-Fargo-Lowers-Credit-Scoresfor-FHA-Loans-1041001-1.html
54
© 2014 Fair Isaac Corporation. Confidential.
Policy Decisions
Expanding Cutoff
Do you have expansion
performance data?
Yes
No
Can you proxy data?
Correct data biases,
determine volume and
risk trade-offs, and
consider overlays
55
© 2014 Fair Isaac Corporation. Confidential.
Yes
No
Quantitative analysis with
careful understanding of
proxy limitations
Purchase data or utilize
qualitative approach
Summary
► Results
► Action
56
Items
© 2014 Fair Isaac Corporation. Confidential.
Results
► Created
one set of reporting templates for all products and scores (some
templates only applied to origination scores)
► Uniform
metrics and formatting increased readability and aesthetics
► Educational
slides regarding assessment techniques and details of the models
increased engagement during model performance presentations
► Flexibility
of the reports to new data greatly increased speed in fulfilling the
requests of reviewers, specifically for requests supporting change validations
57
© 2014 Fair Isaac Corporation. Confidential.
Action Items
► Clearly
define metrics and assessment techniques
►
Discuss and standardize as needed
► Append definitions and other informational slides to reporting decks to ensure consistency
► Inventory
all models and assessment techniques and design a reporting structure that
reduces redundancies and possibly the overall number of metrics
► Create
a list of the requirements and reporting frequency expected by all reviewing parties
to find overlaps to avoid redundant reporting
► Employ
segment/product/score heat-maps when possible to prioritize time
► Discuss
and generate pre-defined performance and shift thresholds
►
An action plan should be established with escalation procedures should a threshold be breached
► Utilize
components of the model monitoring package to pro-actively manage risk and
evaluate potential policy changes or competing strategies
58
© 2014 Fair Isaac Corporation. Confidential.
59
Thank You!
Jerrod Appenzeller
[email protected]
515-213-5109
© 2014 Fair Isaac Corporation. Confidential.
This presentation is provided for the recipient only and cannot be reproduced or shared without Fair Isaac Corporation’s express consent.
Learn More at FICO World
Related Sessions
►Managing your Model Portfolio – Wednesday 3:15-4:45
Products in Solution Center
►FICO® Model Central
►FICO® Score Rating Scale
Experts at FICO World
►Andy Flint – Analytics Product Management
Blogs
►www.fico.com/blog
60
© 2014 Fair Isaac Corporation. Confidential.
Please rate this session online!
Jerrod Appenzeller
[email protected]
61
© 2014 Fair Isaac Corporation. Confidential.