Transcript Evaluation of Credit Guarantee Schemes (CGS): Some

EVALUATION OF CREDIT
GUARANTEE
SCHEMES (CGS): SOME
CONSIDERATIONS
Allan Riding
March 2014
Goals for today
1. Review some of the ways in which credit
guarantee schemes (CGS) have been
evaluated.
2. Provoke discussion about ways in which
credit guarantee schemes (CGS) should
be evaluated.
3. Solicit constructive feedback on most
recent attempts to evaluate CGS.
1.0
Rationales for Credit Guarantee
Schemes (CGS)
• Typical rationales for CGS
– According to economic theory intervention is not
warranted without an imperfection, and, when there
is an imperfection, the intervention should address
the imperfection.
– Information asymmetries between borrower and
lender are often cited as an imperfection that provides
a rationale for CGS.
– Likewise, scale may also be viewed as an imperfection
that supports CGS-type interventions if “good” firms
are denied financing.
• Lenders are unable to distinguish good borrowers from
bad borrowers because small scale loan balances do not
warrant costly due diligence.
1.0
On Interventions in the Credit Market
• Information asymmetries are more pronounced
for…
– Smaller firms? (Many)
– Innovative firms? (Freel 2007)
• More informationally opaque due to unknown technologies;
collateral issues
– Growth-oriented firms? More informationally opaque
due to chaos, uncertainty of growth (Ennew and Binks
1996)
– Women-owned firms? (CFIB, 1994; Fay and Williams,
1992)
– Etc.
2.0 Interventions
• Development banks
– Often with mandate to act as complementary lenders
to those in private sector
• Mixed outcomes
– Yeyati et al IIDB, 2004
– Rudolph, World Bank 2009
• Credit Guarantee Schemes
– The perception that SMEs require interventions such
as CGSs is pervasive:
• Green (2003) reports more than 2,250 CGSs across almost
100 countries.
• OECD (2013) the instrument of choice in insulating SMEs
from the financial crisis
3.0 Evaluating Interventions
• Given the prevalence of CGS, evaluations of
cost/benefit balance are necessary, especially
when governments are guarantors
– Many governments face high debt and are operating
at deficits so value of initiatives must be understood
– Financial crisis has prompted commercial lenders to
be more conservative.
• Provides incentive for increased demand for CGS as business
borrowers become increasingly likely to resort to CGS
– Basel III allows lenders to use risk rating of guarantor
as replacement for risk rating of borrower
• Provides incentive for increased demand for CGS from
commercial lenders
3.0 Evaluating Interventions
• Starting point
– Need to define precisely what are the goals of
the intervention
– Facilitate additional access to financial capital among
firms likely to generate-save for their lack of access to
credit-economic welfare?
– UK: to educate lenders with respect to SME lending;
– Canada: to facilitate SME growth;
– Japan: to rescue firms in disaster situations.
– Other?
– Essential that evaluation measures progress
towards the stated goals
3.0 Evaluating Interventions
• How to conduct evaluation?
– Canada and UK experience (mostly on
Canada)
• “Evaluation” has many definitions
– Useful summary (source of next slides): David
Storey (1998) Evaluating the Impact of Public
Policies to Support Small Businesses in
Developed Economies, University of Warwick
Working Paper
3.1 Evaluating CGS, UK example
• UK:
– Collected questionnaire data from borrowers
who had received loans under the terms of the
(then) DTI Loan Guarantee Scheme (LGS)
– Conducted interviews with primary owners of
CGS borrower firms AND with matching
lender loan account managers
– Analysis of internal data on frequency of
defaults among LGS borrower firms (Cowling
et al.)
3.1 Evaluating CGS: On Survey Data
• Survey data issues:
– Key respondent problem
• Was the responder to the survey the individual within
the firm with best knowledge of issues
– Selection bias
• Sampling frame may affect findings
– Non-response bias
• Low response rates can compromise results, especially
if non-response was systematic
– Recollected data
• Event-related data (e.g., pertaining to date of loan
disbursement) get lost in mists of time.
3.1 Six Steps
• Six categories of evaluations frequently
respect to government programs
Main concern is
to document
expenditure,
make clear that
expenditure is
observed
with
compatible
with program
goals.
An
accountin
g and legal
function,
but no
economic
role.
Example:
BDC
(Canada)
study
3.1 Six Steps
• Six categories of evaluations frequently
respect to government programs
Compares the
outcomes of
firms assisted
by the policy
with those of a
control
group
observed
with
of firms that
had not been
assisted
Matching based
on factors
thought to
influence
sought-for
outcomes
Can be
difficult to
identify basis
for matching
3.2 Key resource: Data
• Canada:
– Collected questionnaire data from borrowers who
had received loans under the terms of the (then)
Industry Canada CSBF (Canada Small Business
Financing program)
– Conducted interviews with primary owners of
CGS borrower firms AND with matching lender
loan account managers
– Used external data on frequency of defaults
among CSBF borrower firms and “comparable”
non-CSBF borrowers
3.2 Key resource: Data
• Canada Key Resource
– Surveys on Financing of Small and Medium
Enterprises
• Initial Survey conducted in 2000 (>10,000 responses)
– Follow-up survey in 2001 (~4,000 responses)
•
•
•
•
2004 (>12,000 responses)
2007 (>12,000 responses)
2010 (>12,000 responses)
2013 (only for firms with >10 employees)
– All are stratified and weighted as to size, region,
sector and age
3.2 Key resource: Data
• Canada Key Resource
– Surveys on Financing of Small and Medium
Enterprises
• Initial Survey conducted in 2000 (>10,000 responses)
– Follow-up survey in 2001 (~4,000 responses)
•
•
•
•
2004 (>12,000 responses)
Linked to tax data for t=1 to t=4 using unique
2007 (>12,000 responses)
business identifier
2010 (>12,000 responses)
2013 (only for firms with >10 employees)
– All are stratified and weighted as to size, region,
sector and age
3.3 Dealing with the counter-factual
• Key task of evaluation is assessing the
counter-factual
– What would have happened to businesses in
the absence of policy (e.g., CGS)?
• the so-called ‘counter-factual’.
– The effect of policy is defined as the difference
between what actually happened and what
would have happened in the absence of policy.
Not
directly
observable
• Split data into two groups:
(1) firms that had applied
for non-guaranteed loans;
(2) firms that had applied
for and received
guaranteed loans.
• Used sample (1) to ‘reverse
engineer’ credit scoring
algorithm.
• Applied credit scoring
model to sample (2) finding
that approximately 75%
would have been rejected
by credit scoring model.
3.4 Evaluation of Benefits
• Compares the outcomes of firms assisted
by the policy with outcomes for firms
which had not been assisted
– inference is that differences in outcomes can
be attributed to the impact of the policy
– Problem is that control group firms may not
be directly comparable
• For example, if control group simply comprises
other firms in the economy, these may differ
systemically from firms that sought loans
(arguably less growth oriented) (Recall BDC study)
•
•
•
•
Used 2004 Survey of Financing of
SMEs.
Used statistical techniques that
seek to explicitly take account of
sample selection bias (based on
Heckman, 1977).
• Developed statistical model of
the selection process and
then, taking account of the
factors that drive the selection
process, estimated a second
equation to explain the
outcomes.
Compared job creation by CSBFassisted SMEs with that of other
benchmark samples
Found that CSBF added
significantly to job and value
creation
• Essentially, that the CSBF
more than pays for itself.
3.5 Evaluation of costs.
• Two components
– Costs of program administration
• Depends on level of government involvement
– e.g., in Canada government is passive: keeps records;
compensates commercial lenders to honor defaults;
receives fees on loans from lenders
– e.g., in some jurisdictions, government approves loan
guarantees and second tier re-insurance is the norm
• Estimated from internal data.
– Costs of honoring defaults
• Timing matters!!
4.0 Evaluation of costs.Early claims (a) are costly and
(b) may reflect lenders’ poor
loan adjudication practices.
• Costs of honoring defaults
• Timing matters!!
Figure 1: Chronology of Claim Receipts
Propor on of All Claims
Received Each Quarter
10%
8%
6%
4%
2%
0%
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36
Time from Disbursement to Default (Quarters)
Later claims may
not matter much;
also borrowers had
paid fees over life
of loans
Example: Costs of honoring defaults
Frequency, by Fiscal Year of Disbursement, of
Claims Received within 15 Months of Disbursement
3.5 Evaluation of costs.
• Costs of honoring defaults
– Difficult to attribute costs to particular cohorts
• Tendency to use logistic/probit regression models to
attribute costs to firm attributes/program
parameters/cohort.
• BUT…
– Right-censoring of data (need to wait years before knowing
whether or not a given loan will default).
– Impacts of attributes etc. may vary over time
» example: age of borrower firm at time of disbursement
may be a key factor immediately following
disbursement (if young firms are riskier than old firms)
but arguably less of a factor ten years later
Logistic Regression Example
Logistic Regression Model:
Claims within 15 Months of Disbursement.
N
81,342
81,342
% of cases
100.0%
100.0%
Partnership
Incorporated
4,485
64,915
5.5%
79.8%
New Firms
1 to 3 years
42,206
11,233
51.9%
13.8%
Other bank or trust
Caisse Populaire
Credit Union
2,128
19,523
6,271
2.6%
24.0%
7.7%
Primary sectors
Construction
Wholesale
Retail
Transportation & Warehousing
Professional, Scientific & Technical Services
Accommodation etc.
Info, cultural, finance, insurance, real estate
Other services
Franchise category
Franchise, Retail
Franchise, Accommodation etc.
5,485
4,523
1,346
12,156
10,231
2,119
16,893
1,282
21,821
6.7%
5.6%
1.7%
14.9%
12.6%
2.6%
20.8%
1.6%
26.8%
2,086
Firm size
and 6,730
2.6%
8.3%
Number of employees (LN)
Ratio of revenues to loan size
Borrower Type (Legal Form)
Age of business
Category of lender
Broad Sectoral Groupings
p-value
0.401
0.000
0.000
0.000
0.008
0.000
0.000
0.000
0.000
0.000
0.086
0.000
0.000
0.000
0.000
0.000
0.185
0.530
0.000
0.814
0.643
0.577
0.001
0.000
0.004
Exp(b)
0.97
0.82
0.52
1.53
3.73
2.49
0.69
0.42
0.47
0.24
0.36
1.37
1.09
0.31
0.95
0.94
0.86
0.66
0.56
0.53
3.5 Evaluation of costs.
• Costs of honoring defaults
• Event history methods (survival analysis)
– For example, proportional hazards model (aka Cox
Regression)
– Probability of default =
a) A baseline pattern that depends on time since
disbursement
MODIFIED BY
a) A function that models how attributes, program
parameters etc. shift the baseline pattern
– Used widely in medicine (how does dosage of medicine
influence survival time?), quality control (how does
attribute of production item influence useful life?).
Example: Proportion Hazards Model
Results
Proportional Hazards Model of Claim Receipts
Number of employees (LN)
Ratio of revenues to loan size
Borrower Type (Legal Form)
Partnership
Incorporated
Age of business
New Firms
1 to 3 years
Category of lender
Other bank or trust
Caisse Populaire
Credit Union
Broad Sectoral Groupings
Primary sectors
Construction
Wholesale
Retail
Transportation & Warehousing
Professional, Scientific & Technical Services
Accommodation etc.
Info, cultural, finance, insurance, real estate
Other services
Franchise category
Franchise, Retail
Franchise, Accommodation etc.
p-value
0.123
0.649
0.013
0.144
0.107
0.000
0.000
0.000
0.000
0.643
0.000
0.000
0.000
0.026
0.858
0.000
0.000
0.447
0.004
0.000
0.427
0.000
0.000
0.010
0.000
Exp(b)
1.02
1.01
0.91
1.06
1.27
1.19
1.03
0.75
0.71
0.88
1.01
1.34
1.20
1.04
1.27
1.22
1.07
1.16
0.87
0.80
4.0 Summary
• In evaluating CGS, need (at least?) three
dimensions
– Estimation of incrementality/additionality
– Estimation of benefits (and factors that
influence benefits)
– Estimation of costs
• Administrative data
• Default data
THANK YOU
Comments, Questions, Discussion