Transcript Training for Di's team - International Swaps and

THE ADVANCED IRB FORUM
Monday 23 June
Kevin Ryan
AREAS TO BE COVERED
• Issues in validating Internal Ratings systems
• How are supervisors reacting – emerging
thoughts
• The FSA Consultative Paper
– Sections on Validation, External Models & Data,
Data Quality, Assessment Horizon & PD
estimation
NOT TO BE COVERED
Detailed ‘how to do it’ to get your IRB systems
approved by the FSA
• We do not know enough at this stage
– “We recognise a lot of work remains in this
complex area with many challenges for firms and
us”
• Responsibility of firms to validate their rating
systems
– Firms ‘validate’; supervisors ‘approve’ (or certify)
STYLISED SUPERVISORS’ VIEW
OF RATING SYSTEMS
• Data and validation a challenge!
• Good work has been done, but not been part of
the mainstream activity of the firms, so
– Even in the best firms, few people understand the issues
– Knowledge and oversight by senior management and
control functions quite low
– Resources for improvements can be difficult to get, so tradeoffs made
– Limited appetite for fresh thinking or challenging
assumptions
• Appetite for short cut solutions, eg agency ratings
A MODEL IS PROVED BY ITS
PERFORMANCE ?
SMALL SAMPLE SIZE
In the common binomial test, construct a
confidence interval around estimated PD
• Suppose a PD estimate is 100bp, and you want
to be 95% confident that actual PD is between
80 and 120bp
A MODEL IS PROVED BY ITS
PERFORMANCE ?
SMALL SAMPLE SIZE
In the common binomial test, construct a
confidence interval around estimated PD
• Suppose a PD estimate is 100bp, and you want
to be 95% confident that actual PD is between
80 and 120bp
You need 9500 borrowers in that grade
A MODEL IS PROVED BY ITS
PERFORMANCE ?
SMALL SAMPLE SIZE
Using the common binomial test, what levels of
true PD are consistent with zero observed
defaults, at a 95% confidence level?
• If you have 500 borrowers in a grade, any PD up
to 1.25%
• If you have 80 in a grade, a PD as high as 6% or
7%
(Binomial test doesn’t work if PD x Borrowers below 5)
A MODEL IS PROVED BY ITS
PERFORMANCE ?
DEFAULTS ARE CORRELATED
• The distribution you can expect from defaults is
many periods with actual defaults below the
mean, and fewer periods (typically clustered
together) with actual defaults well above the
mean
IMPLICATIONS
• Confidence bounds wider than binomial
• More difficult to interpret
A MODEL IS PROVED BY ITS
PERFORMANCE ?
IMPLICATIONS
• “Regardless of the efforts used by banks, …there
will still remain some portfolios where there will
likely never be sufficient default data to calibrate
PDs with any degree of statistical significance”
Internal Ratings Validation Study
• “We’re going to need to be pragmatic for some
years to come”
WHAT CAN BE DONE?
BROADER APPROACH TO VALIDATION, TO
SUPPLEMENT OUTCOMES ANALYSIS
• Logic and conceptual soundness of the approach
• Statistical testing prior to use
• Monitoring of process – are the methods being
applied as intended
• Benchmarking – compare to relevant alternatives
David Wright, FRB, 19 June
WHAT CAN BE DONE?
MORE ON OUTCOMES ANALYSIS, INCLUDING
• More exploration and use of tolerance levels by
firms, which requires
• Better understanding of distribution of expected
defaults
• Supervisory interest in information content of
transition matrices
• What can be learnt from other industries?
– Insurance experience of modeling rare events?
FSA APPROACH – FSA CP
KEY POINTS
• Accountability
– Firms responsibility to validate & submit
documentation with IRB application
– Application to be signed by chief executive
• We expect he will have similar questions to supervisors
• Independence
– Approved by senior committee
– Independent staff to participate
FSA APPROACH – FSA CP
KEY POINTS
• Scope
– To cover all portfolios, but depth will vary with
significance
• Materiality considerations
– Methods may vary, especially by portfolio
• If more reliance can be placed on back-testing we should
need less additional evidence
• If less reliance on back-testing, more additional evidence
is needed
FSA APPROACH – FSA CP
KEY POINTS
• Scope
– Must assess the accuracy of the overall output of
the system
• Not just the inputs
• Will need to take account of overrides and judgmental
adjustments to any underlying statistical models
– Based on statistical analysis of rating system,
related internal data, and third party and publicly
available information
FSA APPROACH – FSA CP
• Coverage
– Full documentation
– Clarity on what rating system aiming to predict, and
expected distributions
– Take full account of adjustments between unbiased
estimates and those used in regulatory capital
calculation
• Conservatism
• Cyclical effects
• Double default effects
FSA APPROACH – FSA CP
• Coverage
– Clearly set out standards of control
– Clearly set out limitations of approach
• Assume that there will be candour
– Include work to demonstrate both ability to rank
into grades (discriminate) and estimate a PD etc
(calibrate)
– Include steps to be taken in event quantitative tests
for power and accuracy are breached
FSA APPROACH – FSA CP
• Discrimination/ranking
– Firm must justify that system shows a ‘high degree
of power in line with industry norm for portfolios of
that nature’
– We do not specify what test should be used, eg
Gini or other, or its level
• But we consult on such a test
– Expect firms to strive for best model they can,
other things being equal
FSA APPROACH – FSA CP
• Discrimination/ranking - issues
– Some see discrimination as key to model building
– Some research that high discrimination needed to
achieve accurate PD measures
– But levels dependent on features of portfolio,
number of defaults
– High is good, but may be over-fitting
– Firms must use targets, but reluctant to admit to
them
– Importance of expertise to interpret
FSA APPROACH – FSA CP
• Calibration
– Standardised ‘scorecard’ proposed
– Firm to justify its estimate against own historic
experience and external sources
– Take account of factors which may expect to lead
to differences; eg conservatism, cycle effects
– We do not specify tests for assessing accuracy
• But we consult on such a test
– If actuals not consistent with estimates, firms must
justify differences and/or take steps to improve
FSA APPROACH – FSA CP
• Data quality
– “Data quality was not seen as the major obstacle to
validation in the long term”
Internal Ratings Validation Study
– Some evidence that cleaning the data produces more
powerful models, while greater accuracy self evident
– Missing defaults?
– “We recognise that the data accuracy challenge for
IRB is significant” FSA CP
– “We propose quantifiable targets to cover
completeness and accuracy that will rise over time”
FSA APPROACH – FSA CP
• External models and data
– In principle supportive as can supplement internal
data and models; also external vendors may
operate to higher standards than firms
– But external models carry explicit risk of black box,
with limitations not known and application to
inappropriate portfolios
– Limit to how much vendors will reveal
– We are trialing ‘vendor grid’ which would
standardise information to be provided – aimed at
reducing risks
FSA APPROACH – FSA CP
• Pooled data
– In principle supportive as, like external data, can
supplement internal experience, and without
commercial confidentiality issues of external
vendors
– “Issues and challenges facing pooling initiatives
• Consistent data definitions for all
• Legal restrictions – data protection and confidentiality
• Potential increase in systemic risk”
Internal Ratings Validation Study
FSA APPROACH – FSA CP
• Expert judgment
– “There is a feeling that not enough time has been
spent on discussing acceptable validation
techniques for these types of systems”
Internal Ratings Validation Study
– Will look to accommodate, but validation
challenges increase
– Are there some cases where it is not feasible to
produce quantified estimates?
– Or use ranking with conservative estimates of
losses
BENCHMARKING
• A complement to back-testing
• Conceptually two types, although differences
between them may be blurred in practice
– Relative benchmarks – compare estimates
between firms to identify outliers
– Absolute benchmarks - compare with an external
benchmark given some credibility
• We propose some element of benchmarking in our
proposed approaches to discrimination (industry
standard) and calibration (comparison with
external sources)
BENCHMARKING
POSSIBLE INITIATIVES
• Private sector services to benchmark ratings
and/or estimates
• Some international interest in requiring firms to
rate ‘test portfolios’
• FSA considering targeted benchmarking at obligor
level where back-testing difficult and amounts
large
– Very early stage of thinking but would welcome
feedback
– Could be run by industry, FSA or other body
SOME VALIDATION CHALLENGES
• Scale of task given possible number of models
– Materiality to firm v materiality to the market
– Importance of consistency
• Need for firms and supervisors to increase
expertise
– Even statistical models require expert judgment
– How much do supervisors need to know?
• Can we give enough guidance to allow objective
self-assessment?
• What is the scope and appetite for improved
standards?
SOME VALIDATION CHALLENGES
• What is the supervisory standard?
– Can we identify a definite pass or a definite fail?
• Benchmarks which must be beaten to qualify, or if
beaten are sufficient
• Implications for expert judgment approaches, or
methods which ‘perform’ less well?
• Other objectives may be avoidance of systemic risk,
and allowing entry to stimulate competition
– How do we incentivise firms to improve?
• Rising standards or Pillar 2 requirements
– Too much conservatism will take away incentives
• LGD and EAD?