Transcript Interpolation vs. Exterpolation: Evaluating the Model

Interpolation vs.
Extrapolation:
Evaluating the Model Dependency
of Counterfactuals Using R
Heather Stoll
Department of Political Science
University of California, Santa Barbara
Plan of Attack
1.
2.
3.
4.
The methodologist’s dilemma
All about R
Example: assessing counterfactual model
dependency (King and Zeng 2006a, 2006b)
Implementation via R package WhatIf (Stoll,
King and Zeng 2006)
The Methodologist’s
Dilemma
You’ve developed cutting edge
statistical techniques:
Now what?
Three Choices

Assume applied researchers will write their own
code to implement new techniques

Wait for commercial software packages to
implement them

Costner approach: “Build it and they will come”
All About R
Why R?





Open source statistical computing environment
UNIX/LINUX, Windows, and MacOS versions
Object-oriented, user-friendly programming
language “not unlike” S of SPLUS
Widely used by statisticians and methodologists
in many fields
Implements range of statistical and graphical
techniques, both commonplace and specialized
R’s Default Capabilities








Linear and nonlinear modeling
Clustering and classification analysis
Classical statistical tests
Non-parametrics
Time series analysis
Matrix and algebraic operations
Publication-quality graphics
And much, much more…
A Picture is Worth 1000 Words
© R Foundation, from
www.r-project.org
What the Methodologist Does




Write functions to implement techniques
Incorporate functions, sample data sets, help
files in a “package”, a mechanism for loading
optional code and attached documentation
Build package and make available for download
from CRAN and/or personal website
Can also simply make functions available as text
files for users to individually input (“source”)
into R
What the User Does





Install R from CRAN (http://www.r-project.org)
Launch R
Type: install.packages(“PackageName”) and
then library(PackageName)
Call desired functions
And …
R Add-on packages
aaMIMutual information for protein sequence alignments abindCombine multi-dimensional arrays accuracyTools for
testing and improving accuracy of statistical results. acepackace() and avas() for selecting regression
transformationsactuarActuarial functionsadaptadapt -- multidimensional numerical integrationade4Analysis of
Environmental Data : Exploratory and Euclidean methodadehabitatAnalysis of habitat selection by animals adliftAn
adaptive lifting scheme algorithm agceanalysis of growth curve experimentsakimaInterpolation of irregularly spaced
dataAlgDesignAlgDesign alr3Methods and data to accompany Applied Linear Regression 3rd editiamapAnother
Multidimensional Analysis Package AMOREA MORE flexible neural network packageAnalyzeFMRIFunctions for
analysis of fMRI datasets stored in the ANALYZE foraodAnalysis of Overdispersed Data apeAnalyses of Phylogenetics
and EvolutionapTreeshapeAnalyses of Phylogenetic TreeshapeArDecTime series autoregressive
decompositionarulesMining Association Rules and Frequent ItemsetsashDavid Scott's ASH routinesassistA Suite of SPlus Functions Implementing Smoothing Splines asterAster ModelsasypowCalculate Power Utilizing Asymptotic
Likelihood Ratio Methods awsAdaptive Weights SmoothingBACCOBundle of emulator, calibrator bayesmBayesian
Inference for Marketing/Micro-econometrics bayesmixBayesian Mixture Models with JAGSbayesSurvBayesian Survival
Regression with Flexible Error and Random EffecBayesTreeBayesian Methods for Tree Based ModelsbaymvbBayesian
analysis of multivariate binary data betaregBeta Regression. BhatGeneral likelihood explorationBHH2Useful Functions
for Box, Hunter and Hunter II bicreducReduction algorithm for the NPMLE for the distribution function obimBayesian
Interval Mapping Diagnostics bindataGeneration of Artificial Binary DataBiodemBiodemography functionsbioparaSelfcontained parallel system for R bitopsFunctions for Bitwise operationsbivpoisBivariate Poisson Models Using The EM
Algorithm blightyUnited Kingdom coastlinesBMABayesian Model Averaging boaBayesian Output Analysis Program
(BOA) for MCMC BolstadBolstad functions booleanBoolean logit and probitboostBoosting Methods for Real and
Simulated Data bootBootstrap R (S-Plus) Functions (Canty)bootstrapFunctions for the Book "An Introduction to the
Bootstrap" bqtlBayesian QTL mapping toolkit BradleyTerryBradley-Terry modelsbrlrBias-reduced logistic
regressionBRugsOpenBUGS and its R interface BRugs BSDABasic Statistics and Data Analysis BsMDBayes Screening
and Model Discrimination butlerUnit testing, profiling and benchmarking for RcalibrateCalibration of Biplot Axes
caMassClassProcessing & Classification of Protein Mass Spectra (SELDI) Data carCompanion to Applied Regression
catAnalysis of categorical-variable datasets with missing valuescaToolsMiscellaneous tools: I/O, moving window
statistics, etc. catspecSpecial models for categorical variables cbaClustering for Business AnalyticscclustConvex Clustering
Methods and Clustering IndexesCDNmoneyComponents of Canadian Monetary AggregatescfaAnalysis of configuration
frequencies (CFA)CGIwithRCGI Programming in RchangeLOSChange in LOSchplotAugmented Convex Hull Plots…
Recent PoliSci R Packages




MCMCpack (Quinn): Bayesian inference via
Markov chain Monte Carlo
Anchors (Wand): Analyzing survey data with
anchoring vignettes
eco (Imai and Lu): Bayesian ecological inference
in 2x2 tables
Matching (Sekhon): Multivariate and propensity
score matching software for causal inference
Counterfactual Model
Dependency
What would happen if pigs could fly?

Much social science inference is counterfactual
The first known attempt to answer this question was in 1909 by J.T.C. MooreBrabazon, who earlier the same year was the first British pilot to fly in Britain. On
the left is Moore-Brabazon in his personal French-built Voisin aero plane. On the
right is a pig in a wicker basket behind a sign that says "I am the first pig to fly."
The Problem

Counterfactuals far from
data (unrealistic) are
model dependent

But how far is too far?

Model dependence
usually studied via
sensitivity analyses but
this has many drawbacks
Example of extreme model dependence for
out-of-sample predictions
Assessing Dependence via Distance

Two procedures for assessing distance of
counterfactual from data:
1.
2.

Determine whether counterfactual involves
extrapolation or interpolation
Calculate proportion of observations “nearby”
counterfactual using Gower’s non-parametric (or
any other) distance metric
Neither requires sensitivity analyses of any sort
Interpolation vs. Extrapolation




Interpolation less model dependent (safer) than
extrapolation, assuming minimal smoothness of
conditional expectation function
Interpolation = counterfactual vector x falls in convex
hull of data, X; extrapolation = x outside of convex
hull of X
Convex hull well-known, but computationally difficult
to identify; also hard to determine membership
No existing implementations for high dimensional data
common in social science research
Convex Hulls
Source: Wikipedia
Source: http://www.ifor.math.ethz.ch/
~fukuda/polyfaq/polyfaq.html
Solution

Check whether or not x can be expressed as convex
combination of all points in X; bypass identification of
hull

Done by checking if feasible solution to standard form
linear programming problem with degenerate objective
function exists

Computationally efficient even for large n and k; makes
use of existing algorithms
Gower’s Distance




Sometimes may want to make finer distinction
Measure distance between each observation and
counterfactual using Gower’s metric
Interpretation of G2 = distance between two points as
percentage of distance across X
Summarize n values of G2 for each counterfactual by:
1.
2.
Plotting empirical CDF
Calculating percentage of observations “nearby”
counterfactual (e.g., G2 less than geometric variability of X)
Implementation: R
package WhatIf
Example: UN Peacekeeping




Doyle and Sambanis (2000): 124 post-WWII civil wars;
study contribution of UN peacekeeping operations to
peacebuilding success
Counterfactuals of interest: for civil wars with UN
involvement, how much success if UN had not gotten
involved? For civil wars without UN involvement, how
much success if UN had?
Construct counterfactual dataset from factual by
replacing dummy UN involvement variable with 1variable; other 10 covariates kept as is
Analyze using WhatIf
R : Copyright 2005, The R Foundation for Statistical Computing
Version 2.2.1 (2005-12-20 r36812)
ISBN 3-900051-07-0
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.
Natural language support but running in an English locale
R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
> library(WhatIf)
################################################
#######
##
## WhatIf (Version 1.4-2, built 2006-01-23)
## Complete documentation available from http://gking.harvard.edu/whatif
##
################################################
#######
> data(peacef)
> data(peacecf)
> my.result <- whatif(data = peacef, cfact = peacecf)
Loading required package: lpSolve
> summary(my.result)
> plot(my.result, numcf = 1, type = “b”)
Summary of Counterfactual Inference Analysis
Call: whatif(data = peacef, cfact = peacecf)
Total Number of Counterfactuals: 122
Number of Counterfactuals in Convex Hull: 0
Average Percent 'Nearby': 0.01303413
Counterfactual in Convex Hull, True or False, and Percentage of Observed
Data Points 'Nearby' Counterfactual:
Counterfactual In Hull Percent Nearby
1
FALSE 0.008196721
2
FALSE 0.008196721
3
FALSE 0.008196721
4
FALSE 0.008196721
5
FALSE 0.008196721
6
FALSE 0.008196721
7
FALSE 0.008196721
8
FALSE 0.008196721 …
Results of Analysis



All 122 counterfactuals extrapolations (outside
of convex hull)
Few counterfactuals near most observations (on
average, only 1.3% of observations “nearby” as
defined by geometric variability)
Data contain little information for answering
key causal question: forecasts based more on
model specification than evidence
Consequences




New model specification
incorporating additional interaction
term
Original and modified models make
similar in-sample predictions
Out-of-sample predictions for
counterfactuals extremely divergent
Hence: counterfactual inferences in
fact sensitive to model specification
Conclusion

Need to know extent to which data as opposed
to model drives conclusions

Use R to make new techniques like these for
analyzing counterfactual model dependency
available to applied researchers