Simple Linear Regression: Checking Assumptions with Residual Plots - YouTube

Let's look into checking the model assumptions with residual plots in simple linear regression.

Recall our simple linear regression model,

where Y is assumed to have this linear relationship with X,

and epsilon is a random error component,

representing the fact that the Ys have some variability

and are randomly distributed about that line.

To do any statistical inference

we needed to make some assumptions about that random error term epsilon.

The error terms are assumed to be normally distributed

and homoscedastic, meaning they all have the same variance.

and they are assumed to be independent of one another.

These assumptions may or may not be true,

and so we should investigate them where we can.

If these assumptions are true then the observed residuals should behave in a similar fashion.

Epsilon_i would represent the true theoretical error term for the ith individual

whereas e_i represents the observed residual that we see in the sample,

the value of Y for that individual minus the predicted value of Y from the model.

And if the assumptions of our model are true

then these observed residuals should behave in a similar fashion.

In other words, they should be approximately normally distributed

with constant variance for all the different X values.

Strictly speaking the observed residuals are not going to be independent of one another

but that's not really a big problem for our purposes.

Here I've plotted the residuals for a simulated dataset against X

I could have plotted them against the predicted values (the Y-hat values),

and that would give us basically the same picture with just a different scaling on the x-axis.

But we shouldn't plot them against the Y values

We can't plot the residuals against the observed values of Y because they're related,

so that would give us a misleading picture.

We shouldn't plot that out.

One thing to note right off the bat is that the residuals always sum to 0 in simple linear regression.

These observed residuals are going to sum to 0,

and that's why I put this 0 line in for a little perspective.

The residuals sum to zero and so they have a mean of zero.

What we are hoping to see is simply a random scattering of points,

nothing giving any indication that the assumptions of our model are false.

So here for instance it looks like the variability is about the same at all these different values of X.

The variability here is approximately equal all the way along.

There also doesn't appear to be any curvature

or just any other indications that there is a problem with the model.

So I'm going to give this the big check mark, indicating that that residual plot

gives us no indication that the assumptions of our model are false.

Here's another type of plot we might see.

At first glance this might look a little bit different from that plot on the last page

because we've got all these different measurements at these individual values of X

We might see this type of thing in an experiment

where we have some control over the levels of X.

For instance, if we're giving different dose levels of a drug or something to that effect.

But here it's a similar story to the last page, there's not really much going on here.

The variability is approximately the same across the board.

There's no systematic curvature -- nothing indicating non normality or anything of that nature.

Overall this is a very reasonable residual plot

A fairly random scattering of points I'm going to give that the check mark,

indicating there's no real obvious problems with our assumed model.

In this residual plot there's a more obvious problem.

The variance of the residuals is increasing with X

and that's a violation of the constant variance assumption.

This type of situation is not uncommon.

It's not unusual in statistics for the variance to increase with the mean,

so we do this see this kind of thing. We do have ways of dealing with this.

One option would be to possibly use something we call weighted regression to deal with this changing variance

but overall the assumptions of our model are not reasonable here,

so I'm going to give that the big X.

Here's something that's a more obvious problem.

There's systematic curvature in the residuals

and that indicates that that assumed linear relationship between Y and X is not a reasonable one.

We may be able to deal with this by using a slightly different model,

but for now this residual plot indicates serious problems with our assumed model,

and I'm going to give it the big X.

If we recorded our observations in time order of some nature,

then we should also plot that against time order.

Here there is something going on

our residuals are small and then they're big and then they're small and then they're big.

there is some sort of time affect that has not been properly included in our model,

and we're going to have to deal with that in some way.

If we ignore it that's going to be a problem.

So this residual plot indicates there's something systematic going on,

and there's a problem with our assumed model, so it gets the big X.

Now let's look at a real-world data set that we've looked at previously.

This is activation level in the pain centres of the brain for 16 women

versus their score on the empathic concern scale.

and I've plotted in the least squares regression line.

Let's see what the residual plot looks like.

Here I've plotted the residuals against the explanatory variable X,

and I'm going to say there's no obvious problems here.

There's no systematic curvature, there's no major outliers.

One could make an argument I suppose

that the variance of the residuals might seem to be a little bit lower here than over here

but that's not necessarily an obvious effect here

and I'm not going to consider that to be a big problem.

So I'm gonna give this the check mark -- that's a reasonable residual plot.

Here's a normal quantile quantile plot of those residuals.

If you recall if the residuals are approximately normally distributed

then the normal quantile quantile plot of those residuals will result in an approximately straight-line

and I'm going to say that for normal quantile quantile plot that's a pretty straight line.

So I'm going to give the check mark there,

and say that the residuals are approximately normally distributed.

so those different residual plots don't indicate any problems with our assumed model,

and it would be okay to go ahead with some of our statistical inference techniques.

Here's another data set that we've looked at previously.

Janko hardness versus density for 36 Australian trees,

and we fit a least-squares regression line through those 36 points.

and it looks to be a pretty reasonable fit,

but let's see what the residual plot looks like.

Here I've plotted the residuals against the explanatory variable X

and there does appear to be some systematic curvature here,

as well one could argue that the variability over here

seems to be a little bit less than the variability over here,

so this residual plot is indicating a possible problem with our assumed model

And if we go back to this scatterplot

armed with the knowledge of what the residual plot looked like,

we might be able to see that curvature here.

We might be able to see wait a minute

it seems to be doing something like this

but the residual plot makes that a little bit more obvious.

The residual plot removes that increasing trend and then rescales the y-axis

so it's a little bit easier to see these issues sometimes in the residual plot,

even though the scatterplot gives us basically the same information.

So the residual plot has told us that there's a problem with the assumed model.

Perhaps that straight line relationship was a reasonable one to begin with,

but the residual plot told us that it's not perfect and that we can probably improve upon it

To improve our model we may consider adding a X^2 term, to fit a curve through those points,

or we might consider a transformation of one or both of our variables,

in an effort to come up with a straight line relationship

but those are other talks for other days.