Durbin Watson Test for checking Residual Autocorrelation - YouTube

Hello everyone.

After we've created a linear regression model,

It becomes really important to validate if the linear regression assumptions are met or not.

One of the assumptions of linear regression is

that there should be no autocorrelation in the residuals of your linear regression model.

Essentially implying that there is no lagged version of the residuals, which is related to the residual itself.

So, how do we go about doing that?

We have something called as Durbin Watson Statistic

The Durbin Watson Statistic is a test for auto correlation

in the residuals from a statistical regression analysis.

The Durban Watson statistic will always have a value between 0 to 4.

How do we infer that if there is any auto correlation in the residuals

based on Durbin Watson test is what we look at in this video.

So let's get started by importing the necessary modules.

I have a file called as durban_data.csv,

which I load into my data frame DF.

Let's look at the total number of rows & columns in my dataframe.

As you can clearly see I have 200 rows in my data frame and 4 columns.

Let's move forward and print out the first 5 rows of my dataframe.

As you can clearly see

My dataframe has 3 feature columns, which are X1, X2 & X3

& also has a target column called as y.

So it's a regression problem having all the features continuous as well as my target is continuous.

Let's now separate X and y values

where X will contain the data of my feature columns which are X1, X2 & X3

and y will contain the value of my target column which in our case is capital Y.

After we have splitted the data into X and y

Let's proceed forward and split the data into train and test.

I'll be using 70 percent of the total 200 rows in training

and 30 percent for my testing purposes.

Now without doing much of EDA or

pre-processing or feature engineering,

let's go forward and fit a linear regression model.

I'm using the statsmodels API to fit a linear regression model.

After fitting the regression model,

this is what I get.

I get an intercept value of 2.70.

For feature X1, I get a weight value of 0.04

For feature X2, I get a weight value of 0.199

for feature X3, I get a weight value of 0.006.

We can also print out the OLS regression results, which is given by the statsmodels API.

To find out how good or bad our model is fit.

As my results suggest

that the model has a very high R-squared value as well as adjusted R-squared

Both having a value of close to around 90 percent.

Rather than spending a lot of time on this report on which I've already created a lot of videos.

I'll link the videos in the description section of the video so you can have a look at it.

let's jump onto something called as a Durbin Watson value.

which in our case turns out to be 2.285.

How is this calculated?

what is the significance?

what does it imply in terms of autocorrelation of the residuals?

is what we look at and try to derive it from scratch.

So now let's spend some time in understanding the Durbin Watson test.

For any statistical test

You have something called as a Null Hypothesis

& an Alternate Hypothesis,

so in our case

The Null Hypothesis states that

there is no auto correlation of my residuals with any of its lagged versions.

On the other hand the alternate hypothesis states that

there is a heavy autocorrelation of the residual series with its lagged versions.

Now that the hypothesis statement is clear,

let's go on to calculate the statistic value for Durban Watson Test.

Now you might be wondering what is this formula exactly

But to give you a simple interpretation of the formula

The value dw which is the test statistic in case of Durban Watson test

is basically a ratio of the sum of squared of difference of residuals

upon the sum of squared of residuals.

So essentially

your numerator is nothing else, but the difference

of your actual residual values

& the lagged version of one unit of your residual values.

So essentially this term that is e of i minus one

is your residual values itself with a lag of one

so you take the difference of these two terms

square it

which gives you the numerator term

the denominator term is basically the sum of square of residual values

So essentially your

residual values are nothing else, but your

observations minus the predictions.

Which is what is denoted by this formula.

After you have computed the test statistic in case of Durban Watson Test

which would be stored into a variable called as dw.

If the value of dw is less than a lower threshold value of dl,

I reject the Null Hypothesis

which was that there is no auto correlation in my residuals compared with the lag version of the residuals

If my Durban Watson test statistic

is greater than an upper critical value called as du.

then I fail to reject the null hypothesis.

There is also case if your dw

that is if your test statistic lies between my upper critical value

and the lower critical value

then the test is inconclusive

so you can't reject the null hypothesis

neither can you fail to reject the null hypothesis

so this is that grey area which you have to keep in mind when you perform this test.

If this idea is clear to you,

let's try to compute these values from scratch.

To compute the values from scratch

Firstly, I'll make use of the model that I've created to find out the predictions

on my training data set

Just a disclaimer.

I want to come at a Durban Watson value of 2.285,

which has been computed for the training data set.

So whatever I'm doing is essentially on the training dataset itself.

So I calculate y_predict

which would be basically predictions of my X training itself using the model that I've computed.

Let's also visualize the shape of my y_predict

so as you can clearly see it as 140 samples.

& my residuals are nothing else, but my observed minus my predicted values.

I save all my residual values into a dataframe called as residual_df

and this is how my overall residual data frame would look like.

It will have 140 rows

along with one column which is specified as "ei"

where each row is basically the difference of my actual value minus the predicted value.

The next thing that I do is

I compute something called as ei_square

which is nothing else, but taking the square of your ei column.

now if you go up

the calculation that I've done which is ei_square

is for the denominator term.

So essentially what I do is, I take the sum values

of my ei_square column

and save it into a variable called as

sum_of_squared_residuals,

so i run the cell and print out the value

so the denominator that I have computed

which is essentially the sum of squared residuals

is having a value of 347.10.

Let's go forward and visualize the dataframe that i have

So, I have 2 columns at this stage I have a column of ei & ei_square.

Next thing that I have to do is, I have to introduce a lag of one

to my ei column in the residual dataframe,

so i run the cell.

& this is what i observe.

So i created a new column called as

ei_minus_one

which is basically the lagged version of my ei column

so this point which was at location or index 0 has shifted to index 1.

So if i now visualize the last five rows as well.

This value has disappeared in ei_minus_one.

& this second last value becomes the last value.

Since I have the first row as an NaN value

I dropped that row entirely.

& now if i visualize the shape of my dataframe i'll have 139 rows.

The next that was there in the numerator calculation is

finding out the difference of the ei column and the ei_minus_1 column,

so I do this operation in this cell.

Once I have the difference of ei and ei_minus_1.

I square that column & save it into a new column called

as square_of_ei_sub_ei_minus_1 column

& I visualize the data frame that I have.

So now this column is of prime importance

for the numerator sum calculation,

which was computed by first taking the difference

of ei column & ei-1 column and saving it into this column.

squaring this column gave me

this column.

So that's how I have been able to reach at the penultimate stage of my overall calculations.

Now, I have the column of all the square differences

of my ei

& ei lagged with a value of 1

Now, all I have to do is I have to take the sum of these values.

Which is a fairly easy task in Pandas,

so I call the dot sum function

pass in the value of the column that I want to sum

and I save this value

into a variable called

as sum of squared of difference residuals

and I run the cell.

So the numerator term

which is

sum of squared of difference residuals has a value of 793.31

Now the test statistic that I mentioned

is basically

the ratio of numerator by denominator.

So when I run the cell

and when I kind of visualize the value of

the test statistic,

it turns out to be 2.285,

let's go up

and validate if the value is correct based on our calculations.

So we are chasing a value of 2.285,

if I go back up.

& if I validate it against this value

I get the same exact value that is given in the statsmodels API report.

So we have successfully implemented a Durban Watson test

from scratch using Pandas

But the story is still far from over

because we have to also validate whatever

value we are getting

does it imply that our overall residuals

are non-correlated or auto-correlated?

For that we will have to make use of

something called as the test table.

So as I've already mentioned you have

the upper critical value DU

you have a lower critical value called as DL

where do you find it?

How do you find it?

is where the Durbin Watson table comes in handy,

so I'll switch over to a PDF

which kind of has the table ready.

I'll also link that table in the description section of the video.

So you can have a look at the PDF file as well.

So, this was the Durban Watson table that I was mentioning sometime back.

Now, how do you find out the values of dU,

and dL

that is your upper critical value & the lower critical value.

So if I keep going down, I'll start seeing a table like format.

So, this is the table that I was mentioning.

So you are something called a k mentioned here

k = 1, k = 2, k = 3 and so on and so forth

and you have some numbers N here.

K, basically signifies

the total number of features that are there in your overall linear regression model.

So as you can clearly see

whatever I have just mentioned is again validated in the PDF itself

where k is the total number of regressors

excluding the intercept term, so

if you remember we had three features X1, X2 and X3.

So, we'll only be looking at k equal to 3 column.

for finding out our upper critical value & lower critical value.

Now, we have finalized the column

Now, we have to finalize the row.

Now "N" is the total number of samples that are considered

that your testing for

So in our case,

we used our entire training dataset to come up with that value of 2.28.

So our training set if you remember consist of 140 samples.

So I'll go down.

& I'll pick 150 since it's closest to 140.

So the lower critical value

at "N" equal to 150

is 1.584

and the upper critical value at "N" equal to 150

for 3 regressors is 1.665

the statistic value that we computed was 2.28

our lower & upper critical values are 1.5 & 1.6 respectively,

so our statistic value

is greater than our lower and upper critical value.

Let's go back to the hypothesis that we framed.

I again repeat since I've changed screens

The lower critical value was 1.5

for N equal to 150

& k equal to 3

the upper critical value which in our case du was around 1.6.

for the same set of combinations of n and k

& the dw that we calculated was 2.28

so essentially dw is greater than dU

so I fail to reject the Null Hypothesis

which states that there is no auto correlation of my residuals

with any of its lagged versions.

So this is a good linear regression model that I've constructed

which does not violate the autocorrelation assumption of my residuals.

Now that we've proven this statistically

we can also validate this by using the ACF plot or the auto correlation plot.

So as you can clearly see from this autocorrelation plot

if I have my residual values as it is

and if I keep taking a lagged version of my

residual combinations & try finding out the correlation,

This is what I find.

There is no significant

auto correlation or correlation of the actual residual series

with any of its lagged series.

So this was my attempt at explaining how Durban Watson test works internally

& how you can use it smartly to validate your linear regression assumption

of autocorrelation of residuals.

I hope you found this video informative.

If you do have any questions with

what we covered in this video then

feel free to ask in the comment section below

& I'll do my best to answer those.

If you enjoy these tutorials & would like to support

them then the easiest way is to simply like the video

& give it a thumbs up

& also it's a huge help to share these videos with anyone who you think would find them useful.

Please consider clicking the SUBSCRIBE button to be notified for future videos

& thank you so much for watching the video.