Transcript Uplift Modeling

Uplift Analysis
with the Quadstone System
Monday, 7th January 2005
7.30am PST / 10.30am EST / 3.30pm GMT / 16.30 CET
Any trouble getting into the conference call:
contact [email protected].
How to ask questions
• Return to Meeting Manager
• Use Chat
© 2005 Quadstone
Uplift Analysis
Nicholas J. Radcliffe
Chief Technology Officer
Agenda
MOTIVATION
Demo 1: Up-sell example (binary outcome)
When is Uplift Modelling important?
Demo 2: Deep-sell example (continuous outcome)
TECHNICAL CONSIDERATIONS
Practical considerations and guidelines
Small population issues and extensions
The quality measure: Qini
TRIAL
How to get a trial copy & datasets
“We have to find a way of making the important measurable,
instead of making the measurable important”
— Robert McNamara
“I know half the money I spend on advertising is wasted,
but I can never find out which half ”
— John Wanamaker
Demo 1: Up-sell example
Binary outcome
SCENARIO
Mobile phone company
3G MMS Video phone promotion
Some mass advertising
- non-targeted customers can purchase
Direct calling campaign to drive further sales
Random 250k chosen from 10m base for trial
c. 75k actually targeted; c. 175k as control
When is Uplift Modelling Important?
Two Separate Benefits
• Not targeting people who are little affected
• Reponse: Don’t spend money targeting or offer discounts
to people who will buy anyway
• Attrition: Don’t spend money trying to save people who
will go anyway
• Targeting people with low probability but high
responsiveness
• Response: Do spend money on people who aren’t very
likely to buy if you do, but are very responsive to
offers/contact
• Attrition: Do spend money trying to save people who
aren’t at huge risk of attrition, but can be made much more
likely to stay
© 2005 Quadstone
When would a conventional model be misled?
Pre-existing knowledge
of product
High
pre-existing
purchase rate
Many
influences
coupon
© 2005 Quadstone
When are negative effects likely?
• Sometimes, our actions actually drive
customers away, especially when:
attrition risk
dissatisfied / angry customers
risqué / offensive
communications
intrusive contact mechanisms
forgotten standing charges
© 2005 Quadstone
Demo 2: Deep-sell example
Continuous outcome
SCENARIO
Grocery retailer
Direct mail campaign to increase spend
Weekly Spend measured in 12-week “pre”-period (AWS)
Also in 6-week post period (AWSPostCampaign)
Objective is difference: (PostMinusPreAWS)
Random 250k chosen from 10m base for trial
c. 75k actually targeted; c. 175k as control
Control Group Structure
• Control group
• Must be representative: technique will give
•
misleading results otherwise
In practice, this means randomly select controls
from target group
All possible
Targets
recipients
• There must be enough of them
© 2005 Quadstone
Population Size
• Population size
• “Rule of 500”: to detect a x% difference (uplift),
•
•
© 2005 Quadstone
x% of the smaller population (usually controls)
should ideally be at least 500 people
So if looking for 1% difference, control group
needs to have at least 50,000 people
So consider longitudinal controls – contact half
now, half later
Pruning and Validation
• Pruning
• Autopruning is implemented, based on qini
•
variance
In practice, fairly unaggressive, so recommend
manual pruning
• Validation
• Ordinary test-training fine if there is enough data
• If not, consider k-way crossvalidation
© 2005 Quadstone
Small Population Extensions
• Bagging (oversampling method) and k-way cross•
•
•
validation
Analysis candidate selection
• useful if there are “too many” analysis candidates
Stronger pruning (variance-based)
Stratification
• Not part of product, but potentially available as an
extension if purchased
© 2005 Quadstone
Return on Investment
• Key thing is that Campaign ROI depends on the net
effect (i.e. uplift) of action, not apparent response
• (reduction in churn) × (value of saved people) – (cost of action)
• (increase in purchase rate) × (value of purchase) – cost
• (increase in spend) – (cost of action)
etc.
• Quadstone System has many suitable ROI FDL
functions ( fx) built in (even without uplift license)
© 2005 Quadstone
So how do you
measure what’s important?
Quality Measure Considerations
• Can only estimate uplift by segment
• This is what we are used to with control groups
• One person does not have a (knowable,
measurable) uplift
• Generalizing measures like classification
error/accuracy or R2 doesn’t look promising
• Rank statistics do seem more promising
because they can sometimes be computed
on a segmented basis
© 2005 Quadstone
Can we use/modify the Gini for Uplift?
Overall uplift: x%
x%
uplift
Possibility of negative effects
0%
© 2005 Quadstone
x%
% of customers targeted
100%
Summary: When to use Uplift
• Uplift modelling is just a better way of
•
modelling the true effect of an action
Particularly relevant to:
• Retention (where it’s the number/value of people
•
•
© 2005 Quadstone
you save that’s important
Up-sell, cross-sell, deep-sell (where it’s the
incremental revenue or profit that’s important)
Risk management actions (where it’s the
reduction in risk achieved that’s important)
Where to find out more
• www.quadstone.com/system/uplift/
• For more in-depth training: our Uplift
Analysis course. Contact
[email protected]
© 2005 Quadstone
Questions and answers
© 2005 Quadstone
After the webinar
• These slides, the data and a four-week trial
•
license are available via
www.quadstone.com/training/webinars/
Any problems or questions, contact
[email protected]
© 2005 Quadstone
Uplift: Quick Reference
Building uplift models
•
•
•
•
Ensure random control
group exists
Set partition field with P
interpretation (1 for
treated, 0 for control)
Set objective (binary,
continuous/discrete)
Hit go
Using difference viewers
•
•
•
Pruning
•
•
Switch to test dataset
Hit Autoprune
Creating results field
•
© 2005 Quadstone
Use “Uplift as difference”
Crossdistribution Viewer
places partition field on “
axis” automatically
For view shown, drag
count to depth, duplicate
mean (ObjectiveField) and
drag on to height
Can configure which
population is viewed by
right-clicking on functions
Using ROI Functions
•
These are available under
fx in Table Viewer when
deriving new field.
Upcoming webinars
th
Thursday, 17 February 2005
Data Preparation in the Quadstone System Version 5
7.30am PST / 10.30am EST / 3.30pm GMT / 16.30 CET
If there’s a webinar topic you’d like to see, please let us
know via [email protected].
www.quadstone.com/training/webinars/
© 2005 Quadstone
Your feedback
Suggestions or feedback? Please enter them in the feedback
form or send them to [email protected]
© 2005 Quadstone
Modifying the Gini for Uplift?
Unaffected
by action
uplift
x%
0%
© 2005 Quadstone
x%
% of customers targeted
100%
The Shape of the Qini Curve
10%
Can’t do better than 100%
sales: if 90% of control
group purchases then
maximum uplift = 10%
?
Why is
this flat?
neutral
–ve
uplift
x%
0%
© 2005 Quadstone
Can’t do worse than 0%
sales: if 5% of control
group purchases then
maximum negative
uplift = 5%
+ve
x%
% of customers targeted
100%