- [Instructor] Hello
and welcome to the lecture on classical linear regression.
So we're going to look at the case where we have 1x
in the linear model.
It's the simplest case, and we'll build on it as we go.
So this is what we're gonna do:
we're gonna look at the single regressor example.
Again, there's really not going to be a lot
of real life examples where you can only have 1x,
but by understanding the properties
of this most simple model,
provides the building blocks that we can use to build
and understand as we add more regressors.
So we'll start with that.
We're gonna look at the betas,
at the disturbances for the error term,
look at the OLS estimator.
So how does ordinarily squares
derive the value of the beta based on the data
that we have?
And the five assumptions,
which lead us to the
OLS estimator being
the best unbiased estimator that we can get
when these five things hold.
So here is
problem set 2.
So what's the relationship between beta naught
and the expected value of the disturbance term?
I want you to think about
the single regressor example,
and what does beta 1 hat measure?
What does it mean?
And give an example as needed;
3: Describe the intuition of how beta 1 hat is derived.
3: What is the major implication of the linearity
of this model?
And last, what are the five assumptions?
What is the major implication if they will hold?
And what is an example of if they do not hold,
well, what's an example of a violation?
And then I would also like you to do two computer problems
in SPSS, C2.1 and C2.2
in the Wooldridge text.
I will provide the data and the questions.
So our simplest regression is when k equals 1,
when we have one regressor,
and you can see, here's our model,
y equals beta nought plus beta 1 x 1 plus u.
So y is known as the dependent variable,
most commonly, that's what we're gonna call it.
You may also see the explain, the response,
the predicted, et cetera, where X is the independent,
or sometimes called the regressor,
and then u is the error term or the disturbance.
So this model has two betas.
So beta naught is the intercept or the constant term.
So you can think about it as if x equals zero,
what's the value of Y?
Sometimes this would have
sort of a sensible interpretation, sometimes not.
It really depends on
the nature of the model.
And beta 1 is the slope parameter.
Usually, this is the figure that we're most interested in
in econometrics.
This is the primary relationship.
And that is because we know
that y will change
by beta units
if we change x by 1.
And as long as our error term does not change,
that we can express it as delta y equals beta 1 delta x.
So if x would change by five units,
the predicted value of y will change by five times beta.
And an important thing about this is that
this linear assumption assumes that no matter the value
of x, so if you have a big x or a small x,
you change x by a little or a lot,
it doesn't matter where you start.
It doesn't matter if x is big or small.
The slope of the line doesn't change that.
One unit change in x will have the same effect on y,
no matter where you start.
So
one of the fundamental assumptions
is that the expected value of our error term is zero.
So we assume that it will have a mean of zero,
that if we draw over and over and over again,
that the value of the mean is going to converge to zero.
And mathematically, this is pretty easy to make happen,
that we can sort of slide the intercept up and down
to make sure that this holds true.
So in a sense, we sort of normalize everything else,
especially the beta naught or the box on the right here,
the alpha term, so that it's true.
It's a very fundamental part of econometrics,
that the expected value of the error term
is always zero.
Not only is the expected value
of the error term always zero.
The error term is always uncorrelated with the x,
and this will be true in this model when we have one x
or as many x's as we have,
the error term is uncorrelated with every x.
Another way of saying that is
the error term is mean independent of x.
That no matter what value of x that you may have,
so no matter what x someone might answer on a survey,
the expected value of the error term is always the same.
That knowing the value of x does not tell you
any information about the value of the error term,
that they are independent mathematically,
or that you could also say that they are orthogonal,
that they sort of don't have anything to do with each other.
They have no correlation.
So knowing something about the x does not tell you anything
about the error term.
So putting these two things together, the last two slides,
it means that the expected value of the error term,
given x always equals zero.
And this is called the conditional mean assumption.
Again, no matter what value of x.
So no matter what the person says on the survey
for their value of x,
say, the expected value of the error term
for that person will always be zero.
And therefore the expected value of y given x
is always beta naught plus beta 1x
since the expected value of the error term is zero.
And this actually sort of forms the regression line,
which we'll see in a few minutes.
And again,
that a 1 unit change in x changes the expected value of y
by beta 1 units.
So what this means is, on average,
for the average person or for the expected value,
that if you change x by 1 unit,
it will change y by exactly beta 1,
but for the expected value, sort of, on average,
if we were to forecast what the effect is,
that's what it would be.
And it doesn't matter if our x is large or small
because it is a linear relationship.
So the lack of correlation between u and X
is extremely important in econometrics.
That might be the single most important thing
that we learn here.
It's the fundamental assumption.
And that is
because,
let's assume that that is not true.
Let's assume that u changes as X changes
and therefore du/dx, or delta u/delta x does not equal zero.
What we wanna do in econometrics is,
what is the change in the expected value, the y hat,
as x changes?
How does a change in the value of x change
the predicted or the expected value of y?
But we don't actually know the expected value of y.
All we know is y.
And so if we change x and we change y,
so when, you know,
by looking at different data points,
the change in y that we see may be due to the change
in the expected value, but it must also be,
or may also be, as a result in the change in the error term.
And we can't tell and we can't pull those things apart.
We can't see, oh, this part is due to the expected value
and this is due to the error term,
because all that we observe is y.
So we really wanna know beta, which is d y-hat/dx.
But if du/dx does not equal zero,
we cannot make that forecast.
And this is
a concept that we're gonna revisit a lot,
but it's a very important one.
And so the interpretation then
is that beta 1 is the change in y for the average person.
It's the forecasted, it's the expected,
but it may not be true in every case.
So if I, you know, if X is income and Y is expenditure,
if you put an extra dollar in my pocket,
I will spend beta 1 of that on the good
that we are wondering about.
Expected, that's the expected value.
I may or may not spend exactly that much,
but that is how we interpret this.
So now that we know some of the uses of our estimates
of beta, let's think about how we derive it.
So suppose that we draw,
we have our two question survey,
we ask X, we have Y, and we have this linear model.
So beta hat is the sum of,
on the numerator,
so look over here at this sort of in the box,
the second equation,
the sum of
each x minus the mean of x
times each y times the mean of y, all summed up.
And in the denominator,
each x minus the mean of x, squared.
So another way of saying is
it's the covariance of x and y
divided by the variance of x.
One of the ways that I liked to think of it is
the mean slope.
And this is just an intuition.
Just one way that it helps me to think of it,
that the slope of a line
is the change in y
over the change in x.
So you can kind of in that second equation there,
sort of divide through change of y over change in x.
So it's sort of the average slope
of how does x change as y changes.
Another way of thinking about it,
the covariance of x and y,
the intuition is,
to what extent do these two variables
sort of march in lockstep or not?
If one gets bigger, does one get smaller,
or the other way around?
Whether they both get bigger or both gets smaller,
or do they have no relationship because
of the strong or weak, et cetera?
And then sort of divided by,
or normalized by the overall variation in x.
So how do these two things change together,
normalized by how does x itself change?
So that's sort of a few ways of thinking about
both the intuition and the math of the OLS estimator,
this beta hat.
So what the OLS estimator does is,
it minimizes the sum of squared residuals.
So it takes
everybody's observation and it chooses beta
so that the distance between their actual answer,
what they actually said, y,
and their predicted value squared,
is as small as possible.
That it minimizes the square of the distance
between y and y hat.
Another way of thinking about it,
or, specifically, it minimizes the sum of the u i squares.
you take everybody's u i, square it,
add them up and
beta OLS makes that number as small as it can.
So it's just like an optimization problem
that you might have had in microeconomics.
You're choosing the value of beta
so that the value of u i squared is as small as possible.
So you're minimizing it.
And again, where y hat is the estimated value of y
and y i is the actual value.
So once you know beta,
your beta naught and beta 1,
you can draw a regression line.
So you take the actual value of x
and plug it into that equation and get y hat.
So you take every individual on the survey, your answer,
and this person and this person and this person,
put in their x, and then every individual then has a y hat.
And if you graph that and it makes a line,
that is the regression line.
And another way of thinking about it is,
you sort of have this graph,
you look at the value of x,
you go up to the regression line,
you go over and see what is the y-value on that line,
and that is y hat.
And again, in OLS,
the program chooses the beta hat,
so beta 1 hat and beta naught hat,
to minimize the sum of squared residuals.
So if you're into matrix algebra, if you're nerdy,
I'm not quite this nerdy,
although I guess I am a bit nerdy,
here is the optimization problem.
And you can see
that beta equals x prime x inverse x prime y.
And do you see that it sort of looks like that formula
that I showed you,
that mean slope or the covariance of x and y divided by
the variance of x?
And if you wanna know more, here's a proof.
So basically you minimize by choosing beta u prime u,
or the sum of squared residuals.
And again, this is the sample regression line.
So for every individual,
you can input an x and calculate what their y hat is,
or even for any value of x,
even if someone didn't actually answer that on a survey,
you can say, well, if someone had given that value of x,
what would we predict they would have said for y?
And that would be their y hat
and you can use this formula.
The reason we spend so much time on OLS is to think back
of the criteria
for estimators,
that when a number of assumptions hold, OLS is blue,
it's the best linear unbiased estimator.
So "best" means the most efficient,
the lowest variance.
Unbiased, we learned about estimate,
and that this is a linear function.
So we're gonna see these five assumptions
in the next few slides.
And when these hold true,
that the first four are needed
for the beta to be unbiased,
and if the fifth one holds, it's also the most efficient.
So here are the five assumptions,
and I'm gonna go through each one
in each of the next few slides.
But here they are.
First, it's that it is linear in parameters.
So this is a linear model,
that you can model the phenomenon
that you're interested in by a linear equation;
Two is it's random sampling.
So the way that you draw your sample is random.
So you're not sort of privileging one group over another,
or just sort of taking whoever comes along
or only taking the survey of older people,
or lower income or only men or anything like that.
That it's a random sample that resembles the population.
Two is that X is a non-stochastic constant.
So your X's do not change over time,
or at least during the timeframe of the sampling.
So if you go and take another sample,
if you ask the same person again,
that their X's will be the same.
And another thing is,
not everyone says the same thing for X,
that X has to have some variation.
Because if X is income and Y is expenditure,
if everybody has the exact same X,
how do you know how Y changes?
How would you think about,
like graphing it, how would you know the slope of a line
if everybody says the exact same X,
as you need to have something with variation, X 2.
The fourth one is this conditional mean,
that the expected value of everybody's error term
given their X equals zero for every individual.
So X has no effect on the expected value of the error term.
And finally, the fifth one is needed
for it to be the most efficient estimator.
And that is the variant of the error terms is a constant
and the covariance of your error term and my error term,
the error term of two individuals are zero.
So your error term has nothing to do with my error term.
So thinking of our bingo ball example,
the bingo ball that you choose has no effect
on what bingo ball that I choose.
So we call this heteroskedasticity and uncorrelated errors.
And another way of thinking about this,
and I'll do it more when we cover this in depth,
is that the variance-covariance matrix is sigma square I,
where I is the identity matrix.
So thinking about an N by N matrix,
the diagonal going from top left to bottom right,
everything on that diagonal is this constant, sigma squared,
and every other value is zero.
Let's talk in depth a bit more.
So our first assumption is that the dependent variable
can be calculated as a linear function
of the independent variable plus the disturbance,
and the violations are thought of as specification errors.
First, if you choose the wrong regressors;
Second, it's not actually a linear function.
It's not linear in the parameters,
or if the parameters change over time.
So the slope of the line,
you know, gets greater or lesser over time.
So we assume that these parameters are fixed
in the population for any given sampling period.
Next is the assumption of random sampling.
So one way of thinking about it is,
and the most common way is,
every individual in the population has an equal chance
of being selected into the sample.
We're not oversampling any group, again, by gender, by race,
by income, by where they live, or anything like that.
Basically, that you end up with a sample
that is representative of the population.
The third one, again,
is that the X is a non-stochastic constant.
So we have a random dataset and observations of Y and X.
The X's do not change over the sampling period.
Again, it's also important that not every individual
has the same value for X.
And these observations are fixed in repeated samples.
So if you drew the same sample again,
the X's would have the same value.
If you were in the sample
in one round and then we did it again in another round,
all of your X's would be the same.
Your Y's would not be the same
because there'll be the error term,
which sort of makes them go up and down again.
Again, it depends on sort of which bingo ball
that you choose.
The three main violations of this are errors
in the variables.
So if you make a mistake in how you measure X.
And we're gonna talk near the end of class
of what we can do about that.
If we have what's called autoregression.
So if last year Y is a regressor in this year's Y,
then that also falls apart.
And simultaneous equation,
where we actually have two dependents
that sort of mutually effect each other.
So in the model that we've been seeing,
we can sort of think of it as a one-way causality.
That a change in X drives a change in Y.
Whereas with simultaneous equations,
that they're sort of simultaneously driving each other.
It's sort of an arrow with
two arrow heads on each side.
Whereas,
in this model it's just X
and an arrow pointing to Y, if that makes sense.
Finally,
and again a really important assumption,
is that the expected value of the disturbance term is zero
and that the value of X has no effect
on the expected value of the error term.
That the error term that you draw
is sort of truly a random occurrence
not dependent on what your X is.
So high-income folks or low-income folks
or medium-income folks,
their income has no effect on what error term
that they would get.
And the mean of it must be zero.
And if that's not true, it leads to biased intercept.
So the big point here is that for every individual i,
the u i and the x i are uncorrelated.
If these four things hold, then the beta OLS,
the OLS estimator of a beta, is unbiased,
and that OLS will yield an unbiased estimator.
So those first four are needed for beta OLS to be unbiased.
And if this fifth assumption holds,
then it's also the most efficient estimator.
So the one with the lowest variance,
so that the one that,
if you can imagine the bell curve
where it's sort of the skinniest bell curve,
it's most sort of bunched up around the mean.
A tall, skinny bell curve would be
a more efficient estimator.
A short, fat bell curve
would be the less efficient estimator.
And we assume that there are so-called spherical errors
and that all the disturbances,
they have the same variance
and are not correlated with each other.
So
when they have equal variance,
it's called homoskedasticity
so that the value of x has no effect
on what the variance of the error term is.
When we have heteroscedasticity,
it means that the variance changes as X changes.
And again,
we're gonna spend a whole week on this in,
like, four or five weeks.
There's also autocorrelation. This is also a violation.
And that is where the disturbances are correlated.
So the bingo ball that you draw will then affect
which bingo ball that I draw.
That is autocorrelation.
And those are the violation, again,
where we're gonna spend more time on this.
But if we have these spherical errors,
so if we have homoskedasticity and uncorrelated errors,
then this is the most efficient estimator as well.
The way we express it is,
if 1 through 5 hold, OLS is blue,
it's the best linear unbiased estimator.
And again,
if this spherical error assumption holds,
it means it's the most efficient estimator.
There are a few other conditions that have to hold.
First, that the number of observations, so your n,
has to be greater than the number of your regressors, k.
So the degree of freedom
is the number of observations
minus the number of regressors.
And that is measured by the degrees of freedom.
It must always be greater than 1.
And more degrees of freedom is better,
it gives a better model.
Also, we're gonna learn more about this too.
There can't be an exact linear relationship among the X's.
So if your first X is income,
and another X is
5 times your income plus 1,
they're a perfect linear combination of each other.
And if you know one, you sort of know the other.
So each regressor has to add some new information
and not just be a simple function of each other.
So we're gonna learn a bit more about that.
A good example for that
is degrees Fahrenheit and degrees Celsius.
So they are linear combinations of each other
and you can include both of them in a model,
say, of how fast a plant grows or,
you know, how thickly the ice forms or anything like that.
So OLS kind of has a special place in the econometrics.
It's the
simplest estimator in some ways,
but it also scores high in a lot of other ones.
So think back, computational cost.
I think any econometrics software package is gonna have OLS.
So certainly SPSS does and R
and all the other well-known ones.
It minimizes least squares, basically by definition.
It also maximizes R squared for much the same reason,
and it is unbiased.
And if all of these assumptions hold,
it's the best unbiased.
That it will have the smallest variance-covariance matrix.
And if the errors are normally distributed,
it's not only BLUE, it's BUE, it's the best unbiased.
It may not minimize mean square error,
that you could have a biased estimator
with a really small variance,
which then makes a smaller mean square error.
And last, if the disturbances are normally distributed,
and we're gonna talk a lot about that
when we talk about hypothesis tests.
But if that holds true,
then OLS and maximum likelihood estimation
are exactly the same.
So this is what we did.
We looked at the single regressor example.
We talked about the betas, the disturbances,
how we get the OLS estimator and the five assumptions.
And again,
if these five things hold, then OLS is blue.
So I hope that you found this helpful as a review guide.
And thanks.