Transcript Correlation and Regression

CORRELATION AND REGRESSION
CORRELATION
A measure of the degree to which 2 (generally
continuous) variables are related.
 Measure of relationship or association.





Time studying-grades: more studying, higher grades.
Wage-job and satisfaction: higher salaries, more job
satisfaction.
Anxiety-grades: higher anxiety, lower grades.
These relationships tell us nothing about
causality
CORRELATION DOES NOT EQUAL
CAUSATION!

1) Does A cause B or does B cause A?
2) Could be a third variable:
More interesting or engaging jobs might pay better.
 So, higher salaries might not produce more
satisfaction.
 Job satisfaction may result from having more
interesting jobs, which also tend to pay better.


Let’s say there is a correlation between colds and
sleep (more colds, less sleep)

Think of a 3rd variable that might be responsible for
this relationship.
SCATTERPLOTS

Useful way to look at the relationship between two
variables:
A figure in which the individual data points are plotted in
two-dimensional space
 Every individual is represented by a point in 2 dimensional
space.
 Ex. Salary (X) and Job Satisfaction (Y)

 Predictor
Variable – variable from which a
prediction is made (X axis).
 Criterion Variable – variable to be predicted
(Y axis).

We likely want to predict Job Satisfaction from
our knowledge of Salary.
NATURE OF THE RELATIONSHIP
Positive relationship: As X increases, Y increases.
 Negative relationship: As X increases, Y
decreases.
 No relationship: As X increases, Y neither
increases or decreases.


If we draw a line through the points in a scatterplot
that best fits all the points, we can get an idea about
the nature of the relationship.
160
Slide 6
Appreciation of Dimmu Borgir
140
120
100
80
60
40
20
0
-20
10
20
30
40
50
Age
60
70
80
90
90
Slide 7
Appreciation of Dimmu Borgir
80
70
60
50
40
30
20
10
10
20
30
40
50
Age
60
70
80
90
100
Slide 8
Appreciation of Dimmu Borgir
80
60
40
20
0
-20
10
20
30
40
50
Age
60
70
80
90
MEASURING RELATIONSHIPS
Correlation coefficient
 The most common is Pearson’s r, or just r for
short.
 This measures the relationship between 2
continuous variables.
 Range from 1 to –1


Positive slope (r = + .01 to +1) - As one variable
increases.

In other words, the variables are varying in the same
direction.
If there is no relationship between the variables, the
correlation would be 0.0.
 Negative slope – (r = -.01 to -1.00). As one variable
increases, the other decreases.

R
We look at the sign of the correlation to
determine its direction
 We look at the absolute value to determine its
magnitude. The closer the correlation is to the
absolute value of 1, the stronger the relationship
between the two variables.

TYPES OF RELATIONSHIP
Correlation coefficients are measuring the degree
of linear relationship between two variables.
 Of course, 2 variables can be related in other
ways.

For example, we could have curvilinear relationships
(U or inverted U for example.
 I like to call this the beer-fun curve.


If you do not obtain a big r, this just means you
do not have a linear relationship between the 2
variables.
The two variables might still be related in some other
way.
 This is why Scatterplots are handy…you can get a
feel for the data just by looking at it sometimes.

PEARSON PRODUCT-MOMENT
CORRELATION COEFFICIENT
Based on covariance
 Degree to which 2 variables vary together
 Covariance (negative to positive infinity)

High pos. cov.: Very + scores on X paired with very +
scores on Y
 Small pos. cov.: Very + scores on X paired with
somewhat + scores on Y
 High neg. cov.: Very + scores on X paired with very –
scores on Y.
 Small neg. cov.: Very + scores on X paired with
somewhat – scores on Y.
 No Cov: High + scores on X paired with both + and scores on Y

COVARIANCE




For each person, we look at how much each score
deviates from the mean.
If both variables deviate from the mean by the same
amount, they are likely related.
Variance tells us how much scores on one variable
deviate from the mean for that variable.
Covariance is very similar.
 It
tells is by how much scores on two variables differ from their
respective means.
COVARIANCE
•
•
•
•
•
Calculate the error between the mean and each
subject’s score for the first variable (x).
Calculate the error between the mean and their
score for the second variable (y).
Multiply these error values.
Add these values and you get the cross product
deviations.
The covariance is the average cross-product
deviations.
Cov( x, y) 
  xi  x  yi  y 
N 1
Note the similarity between these equations
VarianceX 
  xi  x  xi  x 
N 1
Covxy 
x y

 xy 
N 1
n
Note the similarity between these equations
( x )
x  n

n 1
2
S
2
x
( y )
y  n

n 1
2
2
S
2
y
2
COVARIANCE, WHY NOT STOP THERE?
•
It depends upon the units of measurement.
–
•
Solution: standardization
–
•
•
E.g. The Covariance of two variables measured in
Miles might be 4.25, but if the same scores are
converted to Km, the Covariance is 11.
Divide by the standard deviations of both variables.
The standardised version of Covariance is known
as the....
Correlation coefficient!!!!!!
PEARSON PRODUCT MOMENT
CORRELATION COEFFICIENT:
r
Covxy
sx s y
COMPUTE R, TO WHAT END?
What can we make of a correlation = .09?
 Is that still positive? Is that meaningful?

Just like we will always get a difference between two
means, we will always get some sort of correlation
between two variables just due to random variability
in our sample.
 The question is whether our obtained correlation is
due to error or whether it represents some real
relationship?


What do we need?
Hypothesis testing !!!!!!!
HYPOTHESES

ρ or Rho, is the POPULATION correlation.
Ho: ρ = 0
 H1: ρ ne 0

So, the null hypothesis is saying that there is no
relationship between our two variables, or that the
population correlation is zero.
 The alternative hypothesis is saying that there IS a
relationship between our two variables, or that the
population correlation is NOT zero.

This is a non-directional example, but we can have
directional predictions too.
 Ho: ρ >= 0OR
ρ <= 0
 H1: ρ < 0 OR
ρ>0

SET UP YOUR CRITERION
Need the df = n-2 and α
 Rcrit tells you: If your calculated correlation
exceeds this critical correlation, you can conclude
there is a relationship between your two
variables in the population, such that…
 Otherwise you retain the null and conclude that
there is no relationship between your two
variables.

AN EXAMPLE!
Is there a relationship between individuals’ OSpan (X) and their need for cognition (Y)?
 Measure both variables on continuous scales.
 ƩX = 36
X
X2
Y
Y2
 ƩX2 = 218
6
36
7
49
 ƩY = 37
4
16
5
25
2
 ƩY = 225
8
64
8
64
 ƩXY = 219
8
64
7
49

Calculation
 Time!!!

XY
42
20
64
56
5
25
5
25
25
2
4
3
9
6
3
9
2
4
6
DECISION CRITERIA
1 or 2 tailed prediction?
 Let’s go 2 tailed, so

Ho: ρ = 0
 H1: ρ ne 0

α = .05
 df = n-2 = 7-2 = 5


rcrit = .754
Covxy 
S 2x
S2y
x y

 xy 
N 1
n
219  (36 * 37) / 7

 4.7857
7 1
2
(
x
)

2
x


218  (36 * 36) / 7
n


 5.4762
n 1
7 1
2
(
y
)
 y 2  n
225  (37 * 37) / 7


 4.9048
n 1
7 1
(36) 2
218 
7  2.34
Sx 
6
(37) 2
225 
7  2.2147
Sy 
6
r
Cov xy
sx s y

4.7857
2.34*2.2147
 .923
OK, what do we conclude about the Null?
R2
The variability in y that can be accounted for by x
 R2 = Just square our correlation
 R2 = .9232 = .8519
 Interpretation?

A proportion of .8519 of the variability in need for
cognition scores is accounted for by ospan.
 OR, 85.19% of the variability in need for cognition
scores is accounted for by ospan
 OR, Ospan accounts for 85.19 % of the variability in
need for cognition scores.

IS ONE R DIFFERENT FROM ANOTHER R
We can also test the significance of r by
converting it to a z (which is normally
distributed).
 We can also take the zs for 2 correlations and
compare them using a t-test.

Field explains how to do this.
 There is a lot of calculations, it is mechanical, and I
am not going to spend time on this
 You can also do this online quickly:


http://faculty.vassar.edu/lowry/rdiff.html
FACTORS THAT INFLUENCE CORRELATION

1) Range restrictions

Range over which x or y varies is restricted



Ex. S.A.T. and G.P.A.
With range restriction, the correlation could go up, it could go
down, but usually it decreases.
2) Nonlinearity of relationship
Usually get a weaker relationship.
Mathmatically, correlation meant to measure linear
relationships
 With range restrictions, could go up if you eliminate the
curvilinear aspect of relationship



3) The effect of heterogeneous samples

Sample observations could be subdivided into 2 distinct sets
on the basis of some other variable


Ex. Movies and Dexterity with male and female subgroups (draw)
Really there is no relationship between movies and dexterity,
however, because females score higher than males on both
variables, there is a positive correlation
TYPES OF CORRELATIONS

1) Pearson's r:


Both variables are continuous
2) Spearman's correlation coefficient for ranked data
(rs)

Use the same formula as you do for pearson correlation
coefficient


3) Point Biserial correlation (rpb)

Correlation coefficient when one of your two variables
measured is dichotomous, the other is continuous.


1st in graduating class, 2nd, 3rd, etc. by IQ.
MF and liking for romance movies.
4) Phi

Correlation coefficient when both of the variables are
dichotomous.

MF and whether they have traveled out of the US.
PARTIAL AND SEMI-PARTIAL
CORRELATIONS
Partial

correlation:
Measures the relationship between two variables,
controlling for the effect that a third variable has on
them both.
Semi-partial

correlation:
Measures the relationship between two variables
controlling for the effect that a third variable has on
only one of the others.
25-Apr-20
Partial Correlation
Andy Field
Semi-Partial
Correlation
REGRESSION

(Simple) Linear Regression is a model to predict
the value of one variable from another.
 Used
to predict values of an outcome from one predictor.
 Predict NC with Ospan

Multiple Regression is an extension:
Used to predict values of an outcome from several
predictors.
 Predict NC with Ospan and IQ
 It is a hypothetical model of the relationship between
several variables.

We can use regression to predict specific values of our
outcome variable given specific values of our
predictor variables.
 Multiple Regression is an extension:

HOW DO WE MAKE OUR PREDICTIONS?

With a regression line (simple in this case)
Yi  b0  b1X i   i


^
Yi (aka Y ) = Predicted value of Y


Value we will estimate w/regression equation
b1 = Slope of the regression line.



The amount of difference in y associated with 1-unit
difference in x
Regression coefficient for the predictor
Direction/Strength of Relationship
bo = Intercept (the predicted value of y when x = 0;
where the line intercepts y axis)
 x = Value of a predictor (O-span) variable.

WHAT DOES THIS LINE DO?


Method of least Squares
When drawing our line, we want the line the best goes through
our data points.
 Well we do this by minimizing error.
^
 Line is Yi (orY
) = predicted values of Y
 Data points are Y
 Lets draw this out for the Ospan and NC data
(Y- Yi) = Errors in prediction: residual
 Regression equation minimizes squared error, or
variance around the line.


Error =  (Y  Y )2
CALCULATING THE REGRESSION LINE
b
Cov XY
sx
2
r
sy
sx
....................if .s y  s x , r  b
Y  b X

a  Y bX 
N
2.2147
b  4.7857 / 5.4762  .923
 .874
2.34
37  (.874)(36)
a  5.286  (.874)(5.143) 
 .791
7
^
Y  .874 X  .791
THE LINE IS A MODEL

This line will fit the data as best as possible, but
that does not mean it fits the data well.
As we discussed, we could have non-linear
relationships.
 How do we test the fit of the model to the data?
 By assessing the degree to which the line captures
variability(or minimizes error).
 How do we get there?


With our old friends, SUM OF SQUARES!!
WHAT ARE WE TRYING TO PREDICT?
Y!
The mean of Y is one type of model that predicts Y.








If we were to guess someone’s Y, we would use the mean of
Y.
If X is not related to Y at all (i.e., if X does not predict Y at
all), the line predicting Y with X would be what?
Parallel with the X-axis (i.e., slope = 0!)
What would the Y-intercept of that line be?
The mean of Y!
So, we are going to start by measuring the total
variability in Y.
How do we do this, look at the sum of the squared
deviations of Ys from the mean (Y predicted from the
“mean” model).
 That is, calculate SSTOTAL for Y!!!
 You can do it!

THE ALTERNATIVE MODEL

Another way to predict Y is by using X. If X IS related to Y
in a linear fashion, the slope of that line should NOT = 0.


We can create a line that predicts Y with X.
THIS line, or regression equation, minimizes squared error,
or variance around the line.


What is another word for error?


Chance.
What do we do with chance when computing test statistics?


Of course 999,999,999 times out of a billion, this regression line will not
perfectly predict Y. We will have…error.
That’s right, we stick it in the denominator of whatever ration we
compute.
So, we need to know how much variability there is between
the model line (regression equation) and the Y values.
That is, we need a measure of variability around the model
line.
 Each Y deviates a little around the line, right? So, what
should we call the SS that measures these deviations from the
line?


SSRESIDUAL !!! 
THE ALTERNATIVE MODEL CONTINUED

We calculate a regression line that minimizes error.


We can measure that error in terms of variability.
Is this model line different from the mean line?
That is, does the model line account for more error (or
significantly reduce the amount of unexplained variability
relative to the mean model)?
 That is, does the slope of the model line differ from the
slope of the mean model (0)?


To get at this, we measure the variability of the model
line from the mean model.
At each value of X, how different is the Y predicted by the
model line and the mean model?
 Square all those deviations, add them up and what do you
have?


SSMODEL !!! 
SUMS OF SQUARES
SUMMARY OF SS
 SST:

Total variability
OR variability between Y values and the Y
mean.
 SSR:

Residual/Error variability
OR, variability between the regression model
and the Y values).
 SSM
: Model variability
 OR difference in variability between the
regression model and the Y mean.
 OR variability between the model and the Y
mean.
TESTING THE MODEL: ANOVA
SST
Total Variance In The Y Data
SSM
Improvement Due to the Model
 If
SSR
Error in Model
the model results in better
prediction than using the mean, SSM
should be greater than SSR
TESTING THE MODEL: ANOVA

Mean Squareds
Sums of Squares / respective df (as with ANOVA)
 This givens us Average variability, VARIANCE, or
Mean Squares (MS).


Mean Squared Terms can be used to calculate F!
F
MSM
MSR
HOW TO CALCULATE SS


SSTotal is what we know.
SSResidual

(Y  Y

=
hat
)2
2
TOT

N TOT
conceptually
Standard Error of the estimate (average deviation of
Y from the line, or:
s y  yhat 

Y
( YTOT ) 2
2
(
Y

Y
)

hat
N 2
Calculation wise, St. Error of the Estimate =
S y  yhat  S y (1  r 2 )(
N 1
)
N 2
Square that and you have MSResidual
 Multiply by DF (N-2) and you have SSresidual

SS CONTINUED

SSModel?

How to we find that?
SSmodel = SStotal – SSresidual
 Using our O-span and NC data, we can calculate
a regression line and test whether it is
“significant.”
 Sstotal(in Y) = 29.429
 St. Error of
=
N Estimate
1
7 1

S y  yhat  S y (1  r 2 )(
N 2
)  2.2147 (1  .9232 )(
72
)  2.2147 .1777  .934
MSresidual = .867; SSresidual = .867 x (7-2) = 4.335
 SSmodel = 29.429 – 4.335 = 25.0294

F
MSmodel = 25.0294/1
 F = SSmodel = 25.0294/.867 = 28.945

Interpretation?
 The line predicting NC with Ospan accounts for
significantly more variance in Y than the a line
with no slope intersecting the y-axis at the mean
for Y.
 The line predicts more variance in Y than would
be expected from chance alone.


How MUCH more?
TESTING THE MODEL: R2

R2
The proportion of variance accounted for by the
regression model.
 The Pearson Correlation Coefficient Squared

2
R 
SS M
SS T
TESTING REGRESSION COEFFICIENTS
Is the slope of our regression line significantly
different from 0?
 We have one predictor, and our ANOVA is
significant, so in this case we know the answer is
yes.
 How can we test the coefficient anyway?
 With a t-test.

T-TEST FOR COEFFICIENT
Numerator: The coefficient – 0
 Denominator: The standard error for the
coefficient in is the denominator. 1

MSresidual


 ( X  Xbar )
2
We just conduct a one-sample t-test to see if the
slope is different than 0.
SPSS Time!!!
MULTIPLE REGRESSION
Predicting one outcome variable with more than
one predictor variable.
 Same basic thing, just a longer line equation.

y  b0  b1 X1 b2 X 2    bn X n   i
B0
•
•
•
b0 is the intercept.
The intercept is the value of the
Y variable when all Xs = 0.
This is the point at which
the regression plane crosses
the Y-axis (vertical).
BETA VALUES
•
•
•
b1 is the regression coefficient for
variable 1.
b2 is the regression coefficient for
variable 2.
bn is the regression coefficient for
nth variable.
THE MODEL WITH TWO PREDICTORS
bAdverts
b0
bairplay
METHODS OF REGRESSION

Hierarchical:


Forced Entry:


Experimenter decides the order in which variables
are entered into the model.
All predictors are entered simultaneously.
Stepwise:

Predictors are selected using their semi-partial
correlation with the outcome.
HIERARCHICAL REGRESSION
Known
predictors (based on past
research) are entered into the
regression model first.
New predictors are then entered
in a separate step/block.
Experimenter makes the
decisions.
HIERARCHICAL REGRESSION
It
is the best method:
 Based
on theory testing.
 You can see the unique predictive
influence of a new variable on the
outcome because known predictors
are held constant in the model.
Bad
Point:
 Relies
on the experimenter
knowing what they’re doing!
FORCED ENTRY REGRESSION
All
variables are entered into the
model simultaneously.
The results obtained depend on
the variables entered into the
model.
 It
is important, therefore, to have
good theoretical reasons for
including a particular variable.
STEPWISE REGRESSION I
Variables are entered into the model based on
mathematical criteria.
 Computer selects variables in steps.
 Step 1


SPSS looks for the predictor that can explain the
most variance in the outcome variable.
STEPWISE REGRESSION II
Step
2:
 Having
selected the 1st
predictor, a second one is
chosen from the remaining
predictors.
 The semi-partial correlation is
used as a criterion for
selection.
SEMI-PARTIAL CORRELATION

Partial correlation:


measures the relationship between two variables,
controlling for the effect that a third variable has on
them both.
A semi-partial correlation:

Measures the relationship between two variables
controlling for the effect that a third variable has on
only one of the others.
SEMI-PARTIAL CORRELATION IN
REGRESSION
The
semi-partial correlation
 Measures
the relationship between
a predictor and the outcome,
controlling for the relationship
between that predictor and any
others already in the model.
 It measures the unique contribution
of a predictor to explaining the
variance of the outcome.
PROBLEMS WITH STEPWISE
METHODS

Rely on a mathematical criterion.
Variable selection may depend upon only slight
differences in the Semi-partial correlation.
 These slight numerical differences can lead to major
theoretical differences.


Slide 64
Should be used only for exploration
ANALYSIS OF VARIANCE:
ANOVA
The
F-test
 looks
at whether the variance
explained by the model (SSM) is
significantly greater than the
error within the model (SSR).
 It tells us whether using the
regression model is significantly
better at predicting values of the
outcome than using the mean.
HOW TO INTERPRET BETA
VALUES
Beta
values:
 the
change in the outcome
associated with a unit change in
the predictor.
Standardised
 tell
beta values:
us the same but expressed as
standard deviations.
HOW WELL DOES THE MODEL FIT THE
DATA?
There are two ways to assess the accuracy of
the model in the sample:
 Residual Statistics



Standardized Residuals
Influential cases

Cook’s distance
STANDARDIZED RESIDUALS
In an average sample, 95% of standardized
residuals should lie between  2.
 99% of standardized residuals should lie
between  2.5.
 Outliers


Any case for which the absolute value of the
standardized residual is 3 or more, is likely to be
an outlier.
SPSS TIME!
COOK’S DISTANCE
Measures the influence of a single case on the
model as a whole.
 Weisberg (1982):


Absolute values greater than 1 may be cause for
concern.
ASSUMPTIONS 1
 Variable


Type:
Outcome must be continuous
Predictors can be continuous or dichotomous.
 Non-Zero

Variance:
Predictors must not have zero variance.
 Linearity:

The relationship we model is, in reality, linear.
 Independence:

All values of the outcome should come from a
different person.
ASSUMPTIONS 2
 No

Multicollinearity:
Predictors must not be highly correlated.
 Homoscedasticity:

For each value of the predictors the variance of
the error term should be constant.
 Independent

Errors:
For any pair of observations, the error terms
should be uncorrelated.
 Normally-distributed
Errors
MULTICOLLINEARITY
 Multicollinearity
exists if predictors
are highly correlated.
 This assumption can be checked
with collinearity diagnostics.
Tolerance should be more than 0.2
(Menard, 1995)
• VIF should be less than 10 (Myers, 1990)
•
•
Inverse of tolerance
CHECKING ASSUMPTIONS ABOUT
ERRORS
Homoscedacity/Independence of
Errors:
 Plot ZRESID against ZPRED.
Normality
of Errors:
 Normal probability plot.
HOMOSCEDASTICITY:
ZRESID VS. ZPRED
Good
Bad
NORMALITY OF ERRORS: HISTOGRAMS
Good
Bad
NORMALITY OF ERRORS: NORMAL
PROBABILITY PLOT
Normal P-P Plot of Regression
Standardized Residual
Dependent Variable: Outcome
1.00
Expected Cum Prob
.75
.50
.25
0.00
0.00
.25
.50
Observed Cum Prob
Good
Bad
.75
1.00
WHAT ABOUT INTERACTIONS???
Thus far we have really just discussed the
relative predictability of multiple variables.
 But, predictor variables CAN of course interact,
and sometimes we expect them to interact.
 SPSS does not automatically include interaction
terms, not does it calculated them for us .

Fortunately, interaction terms are easy to calculate
and include in regression analyses.
 How on Earth can we do this??
 Multiply the predictor variables together.

2-way? AxB
 3-way? AxBxC

INTERACTION EXAMPLE WITH CONTINUOUS
PREDICTORS
Earlier, we predicted need for cognition with
Ospan and IQ.
 But, what if we want to assess the
productiveness of the interaction between Ospan
and IQ on Need for cognition?

Just multiply those two variables together.
 E.g., compute INT_OxIQ = Ospan*IQ
 Execute.
 To assess the influence of the interaction, use a
hierarchical regression analysis.

Enter Ospan and IQ first
 Enter the interaction second.

REGRESSION WITH ONLY CATEGORICAL
PREDICTORS




Our old friend, ANOVA
We CAN have just categorical predictors (IVs) and a
continuous DV and analyze that with a regression.
To do this, we must dummy code the categorical
variables (0 vs. 1).
Example: 2 Expectation (none vs. Analgesia) x 2 Sex
(F vs. M) with pain as the DV.

Expectation



None = 0
Analgesia = 1
Sex


Male = 0
Female = 1
Expectation x Sex interaction (multiply these two
variables).
 SPSS TIME!!!!!!!

REGRESSION WITH CATEGORICAL AND
CONTINUOUS PREDICTORS
If at least one of our predictors is continuous, use
regression.
 There are more steps now.




1) Center the continuous predictor variable(s). That
is, subtract the mean of Y from each value of Y.
2) Compute the interaction(s) between the categorical
(dummy coded) and the centered continuous
predictors. Just multiply these variables together to
create the interaction, as before.
Use a hierarchical regression, entering the
predictors first, and the interaction second.

For more than 2 predictors, enter predictors first, 2way interactions second, 3-way interactions third,
and so on.
REGRESSION WITH CATEGORICAL AND
CONTINUOUS PREDICTORS CONTINUED

Example: 2 Expectation (none vs. Analgesia) x 2
Goal (continuous) with pain as the DV.

Expectation
None = 0
 Analgesia = 1

Goal (1 = no accuracy goal to 9 strong accuracy goal).
 Expectation x Sex interaction (multiply these two
variables).
 SPSS TIME!!!!!!!

WHAT IS AN ANCOVA?
This is really just an ANOVA in which you
account for the variability of a continuous
variable in your analysis.
 This variable should theoretically be related to
the DV.
 An ANCOVA is a regression analysis in which
the continuous predictor is entered in the first
step of a hierarchical regression analysis.
 Why would we do this??
 Account for (remove) extraneous variance.

DOING AN ANCOVA AS A REGRESSION
AND ANOVA
Example: 2 (Expectation) x 2 (Sex) controlling for
pain tolerance to predict Pain. This is an
ANCOVA.
 In SPSS, lets to this as an ANOVA, then look at
it again as a hierarchical regression.
 In regression:
 1) enter the covariate (tolerance), expectation,
and sex.
 2) enter the ExS interaction.
 The results will be identical.
