- [Instructor] Hello, and welcome to the video lecture
on multiple regression analysis and estimation.
So we're gonna be looking at
how to do a linear regression
with more than one regressor
and look at some of the properties of the model,
what makes a good model, some of the assumptions
that make for an unbiased estimator,
much like we did last time, but now adding more regressors.
And we're also gonna think about
how to understand better
what makes for a more efficient estimator.
We'll start with the K=2, the 2 regressor,
and then look at the general form
where there's K regressors, look at the residuals,
how we derive the OLS estimator,
look at the ceteris paribus assumption,
goodness of fit, the degrees of freedoms,
and then again, looking at those assumptions,
which, if they all hold,
that we can say that OLS is the best unbiased estimator,
and then last, looking at
how do we know what the right model is?
What happens if we add a irrelevant regressor,
and what happens if we miss a relevant regressor?
So here is the homework.
So I want you to think about what is multicollinearity,
what's the consequence and what's perfect collinearity,
what happens if you omit a variable,
and when doesn't it matter?
Three, I want you to unpack the three factors
of the OLS estimator's variance.
What drives the variance in this estimator?
And then there are also a couple
of computer homework problems.
Last time, we looked at the one regressor model.
Now we're gonna look at the two.
So now we're looking at a model
where on the left-hand side is wage,
so what drives whether or not somebody
has a low or a high wage.
And we're looking at two regressors,
education and experience.
So we're really interested mostly in education.
So basically does education pay for itself?
What's the returns to education?
And may also be interested in experience,
but we know that if we omit experience
that it goes into the error term.
And it is very likely that if we asked a survey
that education and experience
may be related,
that they are correlated.
For example, the more education that you have,
maybe you're spending more,
spent more of your life on education,
so you have less experience.
That may be one way of thinking about it,
but you can think they're almost certainly correlated.
So if they are, and we forget to put in experience,
then experience, we know, is in the error term,
and then, almost certainly,
our assumption of the error term
and the regressors not being correlated is violated.
And thus, we will almost certainly have a biased estimator.
As another example,
we might wanna look at a model where the test score,
so what is the test score of some school
based on how much that school district
spends on schools, their expenditure,
and the average household income.
So if we only, so we're really interested,
again, in expenditure.
Does higher expenditure lead to higher test scores?
That was, I would say, what we would think,
and maybe even what we would hope,
but if we forget to put in average income,
almost certainly,
expenditure and income are correlated.
So again, if we forget to put it into the model,
then we're almost certainly going to get a biased estimate
of the effect on expenditure.
So in each case, the beta-1,
the one that we're most interested in,
is a measure of
if we increase X1 by one unit,
how much does the Y increase?
So basically we want to include all the relevant regressors,
so we can account for them,
so they don't end up in the error term,
so we don't have a biased estimator.
We want to account for everything,
so we can make a good case
that the estimator for our beta-1 is unbiased.
Remember back that an absolutely essential assumption
is that our error term is uncorrelated
with any of our regressors.
So now that we have two regressors,
it must be uncorrelated with both.
Thinking ahead to have an unbiased estimator,
that the error term must be uncorrelated
with both regressors.
So that would be
that in the population
that the average residual
for any individual, so the expected value,
would be zero.
And that would hold true for any value of X1 or X2.
So no matter what the respondents say on the survey,
no matter what their X1 and X2 is,
the expected value of the residual is zero.
So, and remember that, that that's important
because if that is not true,
that if dU/dX1
does not equal zero,
then we can't tell as we change X1
whether the change in the observed Y
is due to a change in Y-hat or a change in the error term.
Much the same holds when we look at the general case,
so K regressors.
So in many cases, we're going to have more than one,
more than two, but a good number of regressors.
So to think about the original example,
that wages are probably affected,
not just by education experience,
by training, by ability, by all kinds of other things.
Test scores, in the same way,
have many other factors that would affect them.
When we think of the example of local food expenditures,
it wouldn't just be income, but many other factors,
preferences in where you live and household size
and all kinds of things like that.
So you can probably think of other examples
of where many Xs might affect our Y.
This equation at the top is the general form,
where we have K regressors.
And so when we run it through our software,
there's K plus one parameters,
beta-0, beta-1, through beta-k.
Again, beta-0 is the intercept.
And remember that we sort of slide it up and down
so that the expected value of u equals zero.
And in some cases, it could be interpreted
as the value of X, or the expected value of X,
if all the other, if X,
if all the Xs equals zero,
or if all the betas equals zero.
And in many cases, we think of beta-1 to beta-k
are called the slope parameters
'cause they measure the sort of change,
the slope of the line for that regressor,
and u is the disturbance term, as always.
So once we get some data,
and we run it through a software package,
like SPSS, we get a number of things back.
One is we get estimates, these beta-hats.
So we get K plus 1 beta-hats.
And then if we take everybody's X and plug it in
and multiply each of these,
each of their Xs times the beta-hat
and get them all, add them all up,
you'll get the Y-hat for that individual.
So the forecast of what we would say
or what would be our best guess,
that that individual who answered X in that way,
this is what the predicted value of their Y is.
And note that when we put hats on things,
that that's always the estimate.
That's just how we sort of denote it.
And if there are N observations,
so we do a survey,
and we get N observations,
again, what OLS does
is it minimizes the sum of squared residuals.
So everybody has a ui-hat,
and that's the difference between their Y-hat
and their, what they actually said Y on the survey.
So we square those and add them up.
And that is
how we get these estimates.
We can have a sample regression line.
So that is everybody's Y-hat,
which we get by taking everybody's beta-hat,
or everybody's X, and multiplying it by the beta-hat,
as we see.
Note that everybody's Y-hat,
every Yi-hat is on the regression line.
Why is that?
So think about that, and we'll discuss it in class.
So we interpret these beta-hats as the partial effect
of a one-unit change of that Xi on Y.
So as we change that X by a unit,
that beta-hat denotes the expected change in Y.
So, if we go and change every X
by some amount of units
and multiply them by the beta-hat,
it gets the expected change in Y-hat.
Or you can have,
hold all Xs constant except one
and only change one X, and then beta-hat for that X
would be the change in Y just for changing that one.
And this is sort of the magic of regression,
that it allows us to really isolate the effect
of changing one X while holding all else the same.
So we don't have to go
and collect data where,
for the first many Xs,
everybody answers it the same way.
And then only on, say, the fourth or fifth question,
do they change it, that we can collect
where lots and lots of, there are a lot of answers.
Basically, everybody answers it slightly differently,
but the magic of a regression
is we can still isolate, holding all else equal,
what is the change by just changing one X
and everything else stays the same?
We can also measure the effect of changing a lot of Xs
or even all of them.
So all we have to do is sort of plug it,
plug these change in X, into all of these equations,
into the equation, multiply by the various hats,
add them up, and there you go.
Or, you could change by one unit,
or you can change by basically any unit.
If you do change every X by one unit,
the change in Y-hat will just be the sum
of the various beta-hats.
So hopefully, mathematically, all of that makes sense.
Remember that when we run a regression
that every individual
has a Y-hat, the predicted value,
and a u-hat, the value of their residual.
So where do they fall on the regression line?
And then that u-hat measures the difference
between what they actually said
and what we would've predicted they said.
And again, the i is for each individual in the sample.
So you get the Y-hat for each individual
by plugging their Xs in to the model, again,
multiplying them by the beta-hats
and coming up with the Y-hat.
And then we also, everybody has a u-hat.
And we will learn down the road
that we can save them both in SPSS.
So when we go into SPSS, I will show you how to do that.
Here are a number of mathematical
and statistical properties of the residual.
The sample average of ui-hat equals zero
because the mean is zero.
And so the mean of Y, the observed values,
equals the mean of Y-hat.
That Y-bar-hat equals Y-bar.
the sample covariance between each Xk and u
is zero.
And the point of the mean observation,
so the mean of Y, the mean of X1,
all the way through Xk, always lies on the regression line.
An important thing to note is that adding regressors
almost always changes the value of your beta-hats.
So when you go from beta,
from one regressor to two, so from k=1 to k=2,
the value of your beta-1-hat will change.
In the first model, you only have X1,
and then in another model, you add another X, X2,
there will only be two cases
where the value of beta-1-hat does not change.
First is that if the beta-2-hat equals zero,
so X2 has no effect on Y.
And the other is if X1 and X2 are uncorrelated.
So if X2 is uncorrelated
with Y or with X1,
those are the only two cases where adding X2
will not change the value of our beta-1-hat.
So think about it in this way.
So we run this Y
with two Xs, and then we run it again
with only one X.
Now, suppose that this A1,
which is the coefficient
in our second model, when we didn't include X2,
we could write it as beta-1 plus beta-2 times D,
where D is the slope coefficient
of if you had
regressed X2 on X1.
They will be the same.
So this A1 or A2 will be the same,
only if one of these two things is true,
either B2 here is zero,
so X2 has no effect on Y,
or if X1 and X2 are uncorrelated,
so that if this D is zero.
And I am going to show you in class what this looks like,
kind of drawing a Venn diagram.
And in the general case of K regressors,
that when you add regressors,
usually it's going to change the value of your beta-1.
So the only time that this would not be true
is if all the betas equaled zero,
so none of our regressors have any effect on Y,
or if X1 is uncorrelated with every other X,
with X2,...,Xk.
Both of these would be very rare instances, almost always.
Y and X2 are gonna have at least some correlation,
or X1 will have some correlation
with one or more, probably all of our other Xs,
X2 through Xk.
So the bottom line here is adding or subtracting regressors
almost always changes the value of every beta.
And that's why it's so important to include the right ones.
Thinking again about the concept of R squared,
how well does our model fit the data?
How much of the variation in Y
is explained by the variation in the Xs?
We can, again, decompose it as SST,
the total sum of squares, the variation in Y,
the explained sum of squares, the variation in Y-hat,
and the sum of squared residuals, SSR.
So note that, again, OLS
makes SSR as small as possible,
as before.
So in this drawing here,
they're writing SST as TSS,
but it's the same thing.
It's the total sum of squares.
So SST
is the sum of each Y
minus the mean of Y squared.
SSE is the sum of each Y-hat minus Y-bar.
And SSR is the sum of squared residuals.
So here's the formula.
It's one that you've seen before.
So again, SSD equals SSR.
SSE plus SSR, and we do a bit of math,
and we get R squared,
which is defined as one minus SSR
divided by SST.
So it is that,
that part of the variation in Y,
which is explained by the Xs.
R squared is always a number between zero and one.
Hardly ever is it zero.
Hardly ever is it one.
In fact, in any regression,
you'll basically never see this.
What does it mean if SSR equals zero?
That's something that you could ponder and think about.
Here are some properties of R squared.
So it never decreases, and it almost always increases
when you add a regressor.
So even if you add a total nonsense regressor,
like your shoe size
or how many letters are in your dog's name
or anything like that
that has nothing to do with your model,
it's still going to increase R squared.
And therefore, it's a poor criterion
of whether to add a regressor.
You almost always, since it always goes up,
it really, it's not going to tell you anything.
There is a way that you can calculate R squared
to see which sort of compensates
for the decrease of degrees of freedom.
So it sort of looks at, is the model,
is the R squared better, given that we know
that we lost some degrees of freedom
and sort of compensates it for that.
And that's a better way.
Now we're gonna look at the same kind of assumptions
that we looked at last time.
So these are the things that we assume are true
or that must be true for an OLS model
in order for it to be the best unbiased estimator.
And these are the same as you've seen before,
that it has to be linear in parameters, random sampling,
non-stochastic Xs
that are not perfectly colinear,
and that the residual has to have zero conditional mean.
So I'm gonna walk through each one.
So it has to be a linear model
that you can actually write the population model
in these terms, as a function of Y and Xs, as you see here.
Again, the betas are the unknown parameters,
and the u is the disturbance term.
And what this means is that the betas
cannot have any exponent other than one
for it to be a linear function.
Note that the Xs can have exponents,
so it could be squares or logs
or square roots or all kinds of other things.
And I think squared is probably the most common one
that you're gonna encounter.
If you think that the relationship has a curve in it,
that it's not aligned, you can often add a square.
Next is random sampling.
So again, we draw a sample from a population,
and it's a random sample.
So there's no clear selection bias.
So we don't only choose old folks or men
or high income or larger household
or married people or anything like that.
It's representative of the population.
Third is no perfect collinearity,
that none of the, each regressor,
well, one way to think of it,
is must add some new information,
that there's no perfect linear relationship
among any of the regressors.
And we need this for it to work mathematically,
that their betas will not be defined if you can't,
if there is perfect collinearity.
Note that they can be correlated,
that they almost always are and will be,
just not perfectly so.
So if you are a matrix algebra nerd,
know that our X matrix,
the matrix is a singular matrix.
It'll have a zero determinant, and it cannot be inverted.
So much like we saw before,
it's sorta like dividing by zero.
It's just undefined.
It's certainly okay
for them to be a non-linear relationship.
So you could include both income and income squared,
again, if you think the relationship has a curve in it,
like increasing, but at a decreasing rate.
And that's fine.
Here are some examples.
So one might be
the expenditures in Canadian and US dollars,
where US dollars equals a times Canadian dollars,
a is the exchange rate.
We often find it too in dummy variables.
So if you code
whether you live in Vermont as a yes,
one equals yes, zero equals no,
and have another variable, non-Vermont, coded the other way,
the sum of these two is always one.
So you can include both.
You only include one of these two in your model.
The intuition is how can we measure the effect of US dollars
by holding Canadian dollars constant?
Or how can you, if you're thinking about plant growth,
how do you account for temperature
in degrees Fahrenheit
holding degrees Celsius constant?
So that's the intuition.
So, also, it adds no new information,
that if you know degrees Fahrenheit,
then you automatically know degrees Celsius.
And the degrees Celsius, one would say,
adds no new information at all.
We must also have a positive degree of freedom,
so N must be strictly greater than K+1.
So the number of observations must be strictly greater
than the number of regressors plus one.
Otherwise, you have more unknowns than equations,
and you have either no or infinite solution.
And the more degrees of freedom that you have,
the lower the variance of beta.
And here's a YouTube that explains that.
There are many benefits to having a big N,
to having a large sample size,
and one of those is it increases your degrees of freedom.
And as you see here in this table,
the more degrees of freedom that you have,
the lower the test stat has to be
to be significant.
And next week when we talk about hypothesis tests,
I think this is gonna make even more sense.
But basically, the higher degree of freedom,
the bigger N leads to a more efficient estimator.
And it's a theme that we're gonna revisit
over and over again, more information
leads to lower variance,
or more information leads to
a more efficient estimator, or a lower variance estimator.
And having bigger N, getting information from more people,
is one way that you can get more information.
The next assumption, again,
this is one we should be familiar with by now,
that the expected value of the error term,
no matter the value of X, is zero.
And that no matter the value of X,
the expected value is the same.
It's this idea that the error term
is uncorrelated with the regressors,
and this is needed to have an unbiased estimator.
And that is why, as we'll see,
that omitting an important variable
will result in bias.
When this zero conditional mean holds,
we say that our regressors are explanatory.
The variables are exogenous.
That's what we want, exogenous is good.
When they are correlated with the error term,
they are said to be endogenous.
And most, a lot of the topics
that we'll be covering toward the end of class
will deal with how to detect
if they're endogenous and what to do about it.
So, if we have these four assumptions holding,
if all four are true,
then every beta-hat is unbiased.
And know that what this means is the procedure is unbiased.
The model is unbiased.
It doesn't mean that every single beta-hat
will fall exactly on the true value.
It means that there's no systematic reason
why we should think it's too big or too small,
and if we did this over and over again
that the value would converge to its true value.
And that's what unbiased means.
We're gonna talk about two cases now under specification.
What regressors should you include in your model?
And we're gonna talk first about overspecifying,
which is including irrelevant ones,
like your shoe size
or the number of letters in your dog's name
or something like that.
And sort of probably more seriously,
what happens when you omit ones that you should include?
And we'll see those omitted variable bias.
First, we'll deal with the issue of overspecifying,
which is including a variable that is irrelevant.
It could be nonsense
or just has nothing to do with the model.
So suppose we specify this model with three regressors,
and assumptions 1 through 4 are met,
and everything is cool, but X3 has no effect.
So X3 has no effect on Y,
that in the true parameter in the population is zero.
The slope is zero.
Changing X3 has absolutely no effect on Y.
What happens?
Well, there's good news and bad news.
The good news is since, recall a few slides ago,
that because B3 or beta-3 equals zero,
it won't create bias.
It won't effect any bias of beta-1 or beta-2.
However, it will inflate the variance of the other betas.
So it will increase the variance of beta-1 or beta-2.
So, and there's a few ways that you can think about this.
One, it takes away a degree of freedom for no reason.
And two, it takes away some of the explanatory power
of the other Xs.
So especially if X3
has any overlap with X2 and X1,
it will take away some of the information
that is in those variables
and therefore take away their explanatory power.
And we're gonna talk about this a bit more
when we think about the variance of our beta-hats,
which is sort of the end of this topic.
So underspecifying is, in a sense, more serious
because it creates bias.
But sometimes we can know what the direction
and maybe even size of the bias is.
So assume that the true model is this,
that only X2 and X1
are the relevant regressors,
and it's all well-behaved,
assumptions 1 through 4 hold.
We wanna know, what is beta-1?
What is the effect of X1 on Y?
However, we forget, for some reason, when we exclude X2,
we don't know enough to include it,
there is no data available, something like that,
and so we run, instead, this regression
with just a single regressor, X1.
And I'm putting the A instead of B
to sort of set this apart, so that it's clear, hopefully.
So here's an example from the Wolters book.
Again, we're looking at wages,
and we're really interested in the returns to education.
And we have these two regressors in the true model,
education and ability.
So you come with some innate ability,
and you get education,
and that's what drives your wage.
Again, there's probably more,
but just to make a simpler model.
But we don't have a variable measuring ability.
So we run just a single regressor model,
education, and we get A1.
And that's what we see.
And here, the error term, which is v,
which we're calling v here,
is beta-2 times ability plus the error,
since we forgot about adding ability.
And almost certainly, ability and education are correlated,
that I would assume that those with more ability
probably seek more education.
That might be one hypothesis.
But regardless of the direction,
I think common sense says that your innate ability
and how much education that you get
would be somewhat correlated.
So we can think about what's the magnitude
and the direction of A1?
So A1 is then beta-1
plus beta-2 times d,
where d is the slope of regressing X2 on X1.
So it's how correlated are X1 and X2?
What is the effect of it?
And beta-hat-2 is the slope from the real model,
had we been able to run this.
So here, the expected value of A1
is the expected value of beta-1-hat
plus beta-2-hat times d.
So the bias is this second term,
beta-2-hat times d.
Now, if
beta-2-hat equals zero,
so if ability has no effect on wages,
or if d equals zero,
that is, ability and education are uncorrelated,
then A1 is unbiased.
Then we're fine.
And we sorta talked about that a few slides ago.
So again, if d equals zero,
then X1 and X2 are uncorrelated.
And that would mean
that the expected value of X2 given X1
is just X2.
And then the error term,
then X2 does not,
is not correlated with X1,
and X2 being in the error term
does not violate our assumption 4, and everything is cool.
But again, that's going to be rather rare.
And in our example of omitting ability,
I think, hope you can see,
that that would not be very good reasoning.
So we can also think about what is the direction of it
and maybe even the magnitude.
We can use our intuition.
So if d is greater than zero,
X2 and X1 are positively correlated,
folks that have more ability seek education,
that would be my guess, but maybe that isn't true,
if it's less than zero,
people with more ability get less education.
So you need to look at the effect of X2 on Y
as well as the effect of X2 on X1,
and we can sort of intuit this.
So again, this is our real model,
where education and ability are both included.
So if beta-2 is greater than zero,
which means more ability leads to higher wage,
that would be my guess,
and d is also greater than zero,
more ability leads to more education,
and that means that the bias is positive,
that the effect of education would be smaller.
A1 is greater than A2,
so we overstate the effect of education.
When we have a k variable equation,
it's gonna depend again on how correlated
the omitted variable is with the ones that we include,
but know that every B may be biased,
not just those correlated with the omitted regressor.
Here is how I approach it in general.
I tend to use a big tent approach
and include a lot of Xs, at least in the early model.
So you always wanna base it on these three things,
previous research, theory,
and just sort of common sense and introspection.
So you think about
what does previous research suggest we include?
What does theory suggest we improve?
And what does common sense suggest that we improve?
And we will learn down the road with hypothesis test
how to pair down and how to sort of come to the right model,
to start with a lot of Xs,
and there are tests to see,
well, if we take these out, which is the best model?
We're going to talk about the variance
of the OLS estimators.
What are the variance of beta-hat?
We've spent a bunch of time
looking at the assumptions under which they are unbiased.
And now we wanna think about how efficient are they
'cause we want the best unbiased estimator.
So beta-hat
has variance because it has the error term in them
because it has the observed Y in them.
So each time you,
so even if you've specified the model right,
each time you draw a sample and run it,
you're going to get a slightly different beta-hat
because there's a new set of error terms that are drawn.
We're going to assume now that there is homoscedasticity,
which has that the variance of the error term is a constant.
We're gonna deal with that later
of what do we do when that's not true,
how to test for it and how to account for it.
But now we're going to assume
that the variance of u, given any value of X,
equals a constant, which we call sigma squared.
That is, it's the same, regardless of any value of X,
so no value of X will change the variance
of u, so sort of the shape of the bell curve.
We know it's centered over zero,
but it could be a very tall, skinny bell curve.
It could be a very short, fat bell curve.
The key here, and that's basically
what we're trying to measure, but the assumption here
is that no matter what the value of X is,
that the shape of that bell curve is the same.
Here's the formula for the variance of beta-hat-j.
So you take one particular regressor, Xj.
What is the variance of its beta-hat?
And here it is.
So it has three parts, sigma squared,
SSTj, and one minus R squared j.
So we're gonna think about sigma squared,
that that is, that's the variance of u,
which we already saw, and we're gonna see in a bit
how to measure that.
There's also SSTj,
which is the total sample variation of X.
So it is the sum of every X,
so what everybody said on the survey
times the mean of that X,
so the mean value from our sample,
subtract every individual's Xj from the mean of Xj,
square it, and sum it up from one to N.
So for each of our N respondents,
it's the total sum of squared, so squared X.
And R squared j is the R squared
of if you were to take Xj
and regress it on all the other Xs.
So if we're looking at X1,
it would be the R squared.
If we took X1 and put it on the left-hand side
and regressed X2, X3, X4,...,Xk,
and look at that R squared, that is what this is.
So it's basically a measure of how correlated
is this Xj with all of the others?
A high R squared implies that they're highly correlated.
A low R squared j means that they are not.
Why does this matter?
Why do we care about variance?
Well, if we have a high variance,
so if you can think about it
as kind of a short, fat bell curve,
we have a less precise estimator.
There's a need for larger confidence intervals.
And thus, we're less likely to find significance.
So if you run a regression,
and you don't find anything significant,
it's kind of deflating.
It's like, er, this is,
it's not very interesting.
So even from a really practical matter,
you wanna find things, if they are significant,
you want to find that 'cause that's sort of
what's interesting to talk about.
So I will go over each of the three components of variance.
First is sigma squared.
So a higher, note that in the formula,
higher sigma squared means higher variance.
It means that the error terms are all over the place.
It means that we have a short, fat,
very spread out bell curve.
Another way of thinking about it
is more noise in the equation
makes it harder to predict partial effects.
And know that it is a population measure.
It's independent of the sample size.
And it is unknown, but there is a way to estimate it,
which we'll see in a minute.
SSTj, again,
is the total variation in X.
And the more variation in X,
the smaller the variance.
It's in the denominator, so you want a big SSTj.
So that means that you want Xs
to have some variation.
So if Xj here is age,
you want your sample,
you always want your sample to be random,
but if you're drawing from,
folks from across the age spectrum
from, say, 18 to 99,
the Xs are going to be more spread out.
And the partial effect of age
is going to be a lot easier to predict.
So you can think about it that
if you're trying to eyeball the slope of a line,
and all of the dots are all sort of converged
around a single X, so you only have folks
that are 31, 30, 31, 32, 31, 30,
it's gonna be hard to sort of,
what is the slope of this line?
This is another example
where more info leads to lower variance.
And increasing sample size unambiguously
increases the variation in X
because we're not dividing by N here.
It's just summing them up.
So the more Xs that you have,
unless every single one is on the mean,
adding N will make SST get bigger
and will decrease your variance.
R squared j is the R squared that if you would take Xj
and put it on the left side and run a regression
with all the other Xs on the right side,
what's the R squared?
So if R squared j is one,
we have a perfect linear combination.
And you'll see that we're dividing by zero.
And again, it makes this blow up too.
And in this case, Xj adds no new information.
So you want Xj
to say something that the other Xs don't say.
And the more it says something that the other Xs don't say,
the more new information that you're getting here,
and the lower the variance.
There's a way, and I will show you,
to calculate the variance inflation factor,
or the VIF.
And in SPSS, you'll find it under collinearity diagnostics.
And the VIF is one divided by one minus R squared j.
If it's less than four,
this is just sort of a rule of thumb I've learned,
if your VIF is less than four, it really isn't a problem.
If it's less than 10, it isn't a major problem.
It's more than 10, you may have a problem.
Here, I will show you
that if you exclude a variable,
it introduces bias, but it also decreases the variance.
So there's kind of a trade-off here.
So think back of our k=2 example,
where the real model, say, includes X1.
I mean, it includes X2,
but we run another model where we exclude it.
So beta-1-hat is where we include it,
and it's the formula that we know well,
the bottom box here,
the variance of A1 now no longer has this term.
Since R squared 1
is always a number between zero and one,
one minus that number is also a number between zero and one.
And by including it
in the third box down,
it's going to have,
you're dividing by a number less than one,
which is like multiplying by a number greater than one.
So the variance of A1 is going to be less
than the variance of beta-1-hat.
So that's the trade-off, and that's why it happens.
So here's more about that.
And it depends on how much new information does X2 add?
The more new information,
the smaller this R squared is,
and the less effect that it has on the variance.
So you can play with the math
and see if you can see what is going on here.
So the variance of A1 is always smaller
than the variance beta-1-hat,
unless X2 is uncorrelated with X1,
and then it would be the same.
And if X1 and X2 are uncorrelated,
I mean, are correlated,
the bias trade-off depends on whether B2
is zero or not.
But the variance of beta
is always going to be greater.
The bottom line is that adding an irrelevant variable
exacerbates multicollinearity and increases variance.
And adding observations can decrease variance,
but it doesn't address bias.
To calculate the variance,
we need to estimate sigma squared.
So we want an unbiased estimator of that,
which is sigma squared hat.
And we do this by using the residuals,
which is, for every person,
ui-hat equals Yi minus Y-i-hat,
so what they actually said versus the predicted value
based on what their Xs are.
Sigma squared hat is sum
of the ui-hat squared
normalized by degrees of freedom.
So it's the sum of squared residuals
divided by n minus k minus one.
And note that as N goes to infinity,
this is going to get smaller and smaller and smaller.
So again, a bigger N
will make for a smaller sigma squared hat,
and which makes the overall variance
of beta-1-hat smaller and smaller.
The standard error of the regression
is the positive square root of this estimate,
of sigma-hat squared, square root of.
And this estimate is unbiased,
only if the assumption 5,
homoscedasticity, holds.
So now you could, in theory,
calculate each of the components
and calculate the variance for a beta-1-hat.
So here is our final problem set for this,
and want you to think about collinearity,
about omitted variable bias,
and the variance of the beta-hat,
and why, as each one increases or decreases,
what happens to the variance and why?
This is what we did.
Thanks for watching this,
and have a good day.