Transcript Document
Today’s program
• Herwart / Axel: Kiva intro (the Galak et al. paper) • Follow-up questions • Non-response (and respondent list) • Multi-level models in Stata
Your follow-up questions
See Galak_etal_follow_ups.rtf
Advanced Methods and Models in Behavioral Research – 2012
Issues
• Consider doable in short vs long term • Consider doable with the same or similar data, and on the same or another site • Consider topic (it should not be Kiva) • Not just ... “this might also be interesting” Advanced Methods and Models in Behavioral Research – 2012
Typical paper follow-ups
• Paper is wrong • There is an alternative explanation for the analytical results • Paper’s conclusion might be dependent on design/measurement/analysis. Redo with different kind of design/measurement/analysis • Paper is right, but conclusions limited to a given time or place or context • Paper argues that X’s of a given kind are important check several other X’s of that kind, or several X’s of a different kind you check you • Relative importance of (kinds of) X’s • A connection X Y is given with a “theory” behind it. You check that theory behind it in more detail (typically with another kind of design/measurement/analysis).
Finding other papers on the topic
• Look at the references in the original paper • Search for Kiva related papers (either in Google scholar or directly in Web of Science / Scopus ...) • Search for more general key-words: “micro financing” & “decision-making” • Other literature: try “matching” (e.g. Literature on online dating), other sites with similar setup (e.g. eBay), persuasion and trust, charitable giving ...
New analysis/paper: conjoint experiment
• Tests importance of stimuli directly (allows comparison of importance of different stimuli + choice of stimuli) (also note the analysis on magnitude of donation in Galak et al., and their complicated analysis of the similarity argument) • • Population consists largely of non-donors No real money involved • ...
Non-response analysis
• Not all of the ones invited are going to participate • Think about selective non-response: some (kinds of) individuals might be less likely to participate.
How might that influence the results?
sample
Back to the (multi-level) statistics...
Advanced Methods and Models in Behavioral Research – 2012
MULTI – LEVEL ANALYSIS
Advanced Methods and Models in Behavioral Research – 2012
Multi-level models or ...
•Bayesian hierarchical models •mixed models (in
SPSS
•random effects models •subject specific models ) •hierarchical linear models •random coefficient models •variance component models •variance heterogeneity models dealing with clustered data.
One solution:
the variance component model
Advanced Methods and Models in Behavioral Research – 2012
Clustered data / multi-level models
• Pupils within schools (within regions within countries) • Firms within regions (or sectors) • Vignettes within persons Advanced Methods and Models in Behavioral Research – 2012
Two issues with clustered data
• Your estimates will (in all likelihood) be too precise: you find effects that do not exist in the population [further explanation on blackboard] • You will want to distinguish between effects within clusters and effects between clusters [see next two slides] Advanced Methods and Models in Behavioral Research – 2012
On individual vs aggregate data
For instance: X = introvert Y = school results X = age of McDonald’s employee Y = like the manager Advanced Methods and Models in Behavioral Research – 2012
Had we only known, that the data are clustered!
Using the school example: lines represent schools. And within schools the effect of being introvert is positive!
So the effect of an X within clusters can be different from the effect between clusters!
Advanced Methods and Models in Behavioral Research – 2012
MAIN MESSAGES
Be able to recognize clustered data and deal with it appropriately (how you do that will follow) Distinguish two kinds of effects: those at the "micro-level" (within clusters) vs those at the aggregate level (between clusters) (and ... do not test a micro-hypothesis with aggregate data) Advanced Methods and Models in Behavioral Research – 2012
A toy example – two schools, two pupils
Two schools each with two pupils. We first calculate the means.
(taken from Rasbash) 3 2 -1 -4 School 1 School 2 Overall mean(0) Overall mean= (3+2+(-1)+(-4))/4=0 Advanced Methods and Models in Behavioral Research – 2012
Now the variance
3 2 Overall mean(0) -1 -4 School 1 School 2 The total variance is the sum of the squares of the departures of the observations around mean divided by the sample size (4) = ( 9+4 + 1+16 )/4=7.5
Advanced Methods and Models in Behavioral Research – 2012
The variance of the school means around the overall mean
3 2 2.5
-1 -4 School 1 School 2 -2.5
Overall mean(0) The variance of the school means around the overall mean= ( 2.5
2 +(-2.5) 2 )/2=6.25 (total variance was 7.5) Advanced Methods and Models in Behavioral Research – 2012
The variance of the pupils scores around their school’s mean
3 2 2.5
-1 -4 -2.5
School 1 School 2 The variance of the pupils scores around their school’s mean= ((3-2.5) 2 + (2-2.5) 2 + (-1-(-2.5)) 2 + (-4-(-2.5)) 2 )/4 =1.25
Advanced Methods and Models in Behavioral Research – 2012
-> So you can partition the variance in individual level and school level
How much of the variability in pupil attainment is attributable to factors at the school and how much to factors at the pupil level?
In terms of our toy example we can now say 6.25/7.5= 82% of the total variation of pupils attainment is attributable to school level factors 1.25/7.5= 18% of the total variation of pupils attainment is attributable to pupil level factors And this is important; we want to know how to explain (in this example) school attainment, and appararently the differences are at the school level more than the pupil level Advanced Methods and Models in Behavioral Research – 2012
Standard multiple regression won't do
Y +4 -3 +2 0 +1 +2 -3 +4 … D1 -1 1 0 1 … … … … … D2 -1 1 0 0 … … … … … D3 0 1 1 -1 … … … … … D4 1 0 0 1 … … … … … D5 0 -1 -1 0 … … … … … id 1 1 2 2 3 3 4 4 … …
So you can use all the data and just run a multiple regression, but then you disregard the clustering effect, which gives uncorrect confidence intervals (and cannot distinguish between effects at the cluster vs at the school level) Possible solution (but not so good) You can aggregate within clusters, and then run a multiple regression on the aggregate data. Two problems: no individual level testing possible + you get less data points.
So what can we do?
Advanced Methods and Models in Behavioral Research – 2012
Multi-level models
The usual multiple regression model assumes ... with the subscript "i" defined at the case-level.
... and the epsilons independently distributed with covariance matrix I.
With clustered data, you know these assumptions are not met.
Advanced Methods and Models in Behavioral Research – 2012
Solution 1: add dummy-variables per cluster
• So just multiple regression, but with as many dummy variables as you have clusters (minus 1) ... where, in this example, there are j+1 clusters.
IF the clustering is (largely) due to differences in the intercept between persons, this might work.
BUT if there are only a handful of cases per person, this necessitates a huge number of extra variables Advanced Methods and Models in Behavioral Research – 2012
Solution 2: split your micro-level X-vars
Say you have: Make sure that you understand what is happening here, and why it is of use.
then create: and add both as predictors (instead of x 1 ) Advanced Methods and Models in Behavioral Research – 2012
Solution 3: the variance component model
In the variance component model, we split the randomness in a "personal part" and a "rest part" Advanced Methods and Models in Behavioral Research – 2012
Now: how do you do this in Stata?
[note to CS: use age and schooling as examples to split at restaurant level]
relevant commands
xtset and xtreg bys
convenience commands
tab , gen() order edit drop des sum Advanced Methods and Models in Behavioral Research – 2012
Up next
• How do we run the "Solution 1”, "Solution 2”, and “Solution 3” analysis and compare which works best? What about assumption checking?
• Random intercept we now saw, but how about random slopes?
Advanced Methods and Models in Behavioral Research – 2012
When you have multi-level data (2 levels)
1. If applicable: consider whether using separate dummies per group might help (use only when this does not create a lot of dummies) 2. Run an empty mixed model (i.e., just the constant included) in Stata. Look at the level on which most of the variance resides.
3. If applicable: divide micro-variables in "group mean" variables and "difference from group mean" variables.
4. Re-run your mixed model with these variables included (as you would a multiple regression analysis) 5. (and note: use regression diagnostics secretly, to find outliers and such) Advanced Methods and Models in Behavioral Research – 2012
To Do
• Make an invitation list using the Excel template that I will send to you later this week (don’t invite anyone just yet!) • Make sure you understand the multi-level concept with random intercepts (that is: c 0 varies per cluster), and know how to do it in Stata • Try the assignment on the website. Next week we will work on that data in class. Check out the practice data “motoroccasion8March2012.dta” on the website as well. It’s practice data.
Advanced Methods and Models in Behavioral Research – 2012
Data: TVSFP on influencing behavior
Advanced Methods and Models in Behavioral Research – 2012
Online already (though not visible) motoroccasion8March2012.dta
Advanced Methods and Models in Behavioral Research – 2012