Neural Networks from Scratch - P.1 Intro and Neuron Code - YouTube

What's going on everybody welcome to the Neural Networks

from Scratch series where we will be creating

creating Neural Networks end-to-end so from...

the Neurons, the Layers,

Activation functions, Optimization,...

Back Propagation all this stuff were going to be coding

that from scratch in python. Now...

everything we do we are going to show first in truly raw python

no libraries or no 3rd party libraries

and then we are going to use NumPy for multiple reasons

NumPy just makes a lot of sense here. It's a

extremely useful library and

it'll cut our lines of a full application down-a-ton

it'll make it much faster

and NumPy is a great thing to learn, so...

we will show everything from scratch in python first and then we are going to use NumPy. Now...

why would anybody wanna do this to themselves?

Well, first of all, it's very interesting umm...

the idea is not-even though we effectively we are going

to be programming our own Neural Networks

framework, it's not really the

purpose here. The purpose is to actually learn

how Neural Networks work at a very deep level so that

when we go back to whatever Framework

we actually use, be it PyTorch or Tensorflow's Keras

or maybe some library that doesn't even exist yet.

We actually understand what we are doing. So

from myself when I at least when I learned Deep Learning, it was all-

-yes it was hard but at the same time

everything was kinda solved for me you know, how

many layers, how many multiple layers, what Activation functions to use

but all that stuff was just fed to me

I just kind of-okay

and then I just sort of memorized like here's the activation function

used for this sort of task

But I didn't understand why and

this became a problem when I tried to solve problems that

had not yet been solved for me.

So classifying images of handwritten

digits? Pretty darn simple. Taking that

a step further and classifying images of cats and dogs?

Pretty simple.

But then taking that just a tiny step further

and classifying images that were instead frames from a video game

and trying to map that to actions that I want to take

in the video game. Suddenly, I'm lost.

and that's.....

...that's no good and there's no way to really know

where to go next and you can see that there is a problem

but actually trying to solve

more custom problems is going to require to have a deeper

understanding of how things actually work

Unfortunately though deeper understanding Deep Learning

can be very very complicated.

So you can just-just a simple forward pass of a Neural Network

Neural Network looks

extremely daunting. So looking at the calculations

for a Neural Network, it looks pretty confusing fairly quick

You've got your input data

and every unique input and every unique neuron.

The information coming through has a unique weight associated-

with it. Those get summed together per neuron

neuron plus a bias. Run through an

activation function and then we do that for

every single layer, giving us the output

information. From there we wanna calculate a loss

which is a calculation of how wrong the neural Network is,

so that hopefully we can fix it.

And at the end of that, even though that was just a forward pass

that looks already extremely-extremely daunting

Now let's take a look at the exact same formula just in code version.

You have all your inputs times your weights (Input x Weights)

And you don't have to follow along perfectly here just look at each element and see if any of these elements

look like they are over your head. They really shouldn't be.

So inputs times weights, you can make this even simpler by doing just a dot product so 'np.dot'

y1 is just the max of zero and the output

We do this for all the layers. Then we have

our softmax (activation function) at the very end.

And then the loss turns out to be

a negative log (logarithmic) loss

due to the nature of neural networks.

So this is the entire forward pass and calculation of loss formula

I urge you to look at all of the functions and things that we are doing here

to determine if any of these is really over your head because it shouldn't be right?

We've got very simple functions going on here

We are calculating a log, sum-if you don't know what a 'log' means btw we're going to explain it.

'Sum', you should know what that means.

'Exponential', again if you don't know what that means we are going to explain but it's very simple.

'Dot Product', again if you don't know what that means

no worries we will explain it, very simple.

'Maximum', is just the max whatever two values you pass here. Again very simple.

Some more dot products,

'Transpose', again very simple.

If you don't know it we will explain it anyway

That's it! None of this is over your head I promise you

As far as prerequisites go the only expectation I have from the viewers is that you understand

programming and Object-Oriented Programming (OOP)

otherwise, you're going to feel kinda lost.

If you're coming from a different programming language you will probably find

python simple

You can probably use the programming language that you are familiar with

Everything is so low level here that you should be able

to follow along in any other language that you want

So feel free to do that, if you wanna do that

otherwise, if you wanna follow along in python make sure you know the basics and OOP

I'll put links in the description for both of those

If you need to brush up

Next, is the version of python

If you follow along with python we are gonna do things like 'fstring'

I'm gonna be on Python 3.7

The NumPy version, I can't think of any reason of why it would matter.

I'll put that information in the description

just in case any function does change.

Like I said, everything's so low level this series should be good for like 10 years.

Let's hope for that :)

So those are the prerequisites

it's really not much.

I don't expect anybody to have any background knowledge of Deep Learning

So if you do know things about deep learning, yes we are going to cover hopefully quickly

the fundamentals, just so people understand

like what exactly are we aiming for here?

And then the bulk of your understanding of how Neural Networks work

is gonna come from us just building these Neural Networks

If things feel a little fuzzy to you

it's probably normal, to be honest.

I think once you build a Neural Network from scratch

that is all the understanding you are gonna need

So you're not expected to know math or anything like that.

For me in college, the only math class I took was 'Math Fundamentals'

And I don't think we did any math calculation at all in that class.

It was definitely a joke.

So if I can do this, I know you can do this too.

So If you wanna brush up on Math

There is Khan academy for Linear Algebra and the calculus stuff

But I wouldn't even suggest you go through the full series on either of those topics

You can use those to kinda like a spot check issues that you still find confusing

With that in mind, this series is also provided in conjunction with the 'Neural Networks from Scratch' book

We are going to be covering the same material for the most part.

The book might be a little more verbose.

The series is obviously free

but the book ha various prices depending on what you want.

The E-book, Softcover, Hardcover. We ship everywhere in the world

Access to the book gives you access to the E-book.

So whichever version you buy you always have access to the E-book.

That gives you the opportunity to access the google docs

draft, currently it's in a draft form

at some point, it won't be a draft anymore

but you can highlight, post comments, ask questions inline with the text

Also if you're impatient you can access even though its a draft right now

it's complete from basically end-to-end

of training the Neural Network

and we're doing testing right now.

So, we are already obviously quite a bit ahead of where the videos are

so if you're impatient you can also access information earlier with the book

I would use the book as either reviews or the videos as reviews

So I would maybe read the book before watching the video and then use the video as a review of what I learned in the book or vice versa

This is a topic that you're not gonna blow through this in a weekend.

It's gonna require multiple sittings multiple environments

and ideally multiple mediums so,

If you're interested in the book you can get that at 'nnfs.io'.

So we call these Neural Networks because they look visually like a network.

You've got your Neurons which in this case are the blue circles.

They're connected via those orange lines

and in this case, we have basically the input layer, two hidden layers of 4 neurons each

and then your output layer.

Now data is gonna get past forward through this,

so it starts at your input layer, so in this case, we only have two

pieces of data they are gonna come in.

That gets past forward to that first hidden layer

Then that hidden layer passes data to the second layer and then finally to the output layer

where we hope that it will output something that we want, for example:

based on some sensor data maybe we wanna predict

Failure or not failure. You could either have one neuron or in this case we have two neurons.

So the top neuron might be a failure neuron, the bottom neuron is a not failure neuron.

And depending on which one has the higher value

that's the supposed prediction.

Now the end goal of neural networks like most machine learning is to take some input data

and produce output data that is desired.

In this case, we've got images of cats and dogs.

We hope that we can pass it through in pixel form to our neural network

and if its a dog then that final output neuron on top

is going to be the strongest, if it's a cat

then that final output neuron on the bottom will be the strongest.

And we can do this by tuning the weights and biases

So all those unique weights and biases, we do that by tuning those

and that is the actual training process.

It's tuning those in such a way so that

hopefully, we can take data that this neural network has never seen,

give it pictures of cats and dogs that it has never seen

and have it accurately predict those

So how and why do neural networks work?

Well, if you just look at them and really consider what's going on

every neuron is connected to the subsequent layer of neurons in folds.

So each of those orange lines, that connection

that's a unique weight

and then every neuron is a unique bias.

So what this ends up giving us

is a huge number of uniquely tunable parameters

that go into this GIGANTIC function. So for example, with

64 x 3 hidden layers here we have

9164 tunable parameters

in this gigantic function. And each of those parameters

impacts the output of the next neurons

and so on. And so what we end up having

are these complex relationships that can in theory

be mapped. So to me the

impressive thing of neural networks is not

necessarily all of those connections its not

really complex to understand.

What the hard part of neural networks and deep learning is figuring out

how to tune such a thing.

Alright! since I think it would be lame to not post

any code at all in this first video we are going to begin to code neuron.

But first I wanna go over the version numbers real quick

because quite frankly I am gonna forget to put it in the description

So I'm using Python 3.7.7

NumPy 1.18.2

Matplotlib 3.2.1

Again all of this should work

very far into the future

But just in case not, there are

the exact versions if anybody needs to have the exact version to follow along

Now, let's go ahead and begin to code

just to say I am using Sublime Text

You can use whatever editor you want.

Everyone's got a really strong opinion on editors.

I'm gonna be using Sublime text. There might be a time where we use Jupyter Notebook.

Who knows, I'm gonna use whatever makes the most sense to me at the time

which may not make sense to you or me later

Anyway, continuing on, so every neuron basically-

Let's pretend we are coding a neuron that's somewhere in this-

densely connected feed-forward multilayer

perceptron model. Using all the big words.

Don't worry about knowing what those mean, by the end of this

you will know all of those words.

But part the problem with learning deep learning is people use the same 3-4 words

for the exact same thing and it can be very daunting. So for now,

we are gonna called this a neuron, it's somewhere in our neural network.

Now in this fully connected neural network

every neuron has a unique connection

to every single previous neuron.

So, let's say there are 3 neurons

that are feeding into this neuron that we are gonna build.

So, we don't know much about those neurons but we know that they are outputting some value

So first, their output becomes the neurons that we are coding's inputs

We are gonna just make up some numbers, we're just gonna say 1.2, 5.1, 2.1

Those are the unique inputs

So these are outputs from the three neurons in the previous layer

Every unique input is also going to have a unique weight associated with it.

So we are going to say weights and you should know how many weights we are going to have

Well since we have three inputs we know we are going to have three weights (3.1, 2.1, 8.7)

I'm just making up these numbers.

It's just for, beginning to code how neurons are gonna work

So you've got your inputs, your weights

and then every unique neuron has a unique bias

So bias equals to 3

So now, the first step to for a neuron

is to add up all the inputs times weights plus the bias

So this is relatively simple.

In very raw python no loops required at this stage.

We're just going to say basically

output so far of this neuron is going to be

*look at the screen now*

*You need to have some basic knowledge of 'list' here*

That is basically so far...

There will be other things that will happen soon enough, but so far

that is the output to our neural network

So we'll just print the output, run it. 35.7

If you're new to Sublime Text for some reason

'ctrl + b' to run. But you'll have to set up the build system to run with python.

But I would expect how

that people know how to work with a programming language

But if not feel free to

either comment below, join us in 'discord.gg/sentdex

And we'll be happy to help.

Anyway that's it for now

if you've got questions, comments, concerns, whatever

moving forward feel free to ask them below

otherwise, we're just gonna keep slowly shipping away

at Neural Networks from Scratch

As you can see pretty darn simple so far

and for the most part, it's just adding a bunch of-

as we've broken it down so far

We are gonna break it down to the point where every little additional step

is gonna look an awful lot like that

There's a couple of points that may be a little more challenging

but the goal is to break it down

so much that it is painfully simple

So I will see you guys in the next video :)