Hello guys.
So we are going to continue our discussion with respect to RNN.
And in this video we are going to talk about RNN back propagation with time already in our previous
video, we have seen how the forward propagation with time happens in RNN.
In a simple RNN, we have also seen about the basic architecture of a simple RNN.
Now with respect to this particular forward propagation, once we go at the end, once we calculate
the loss function, our main aim is to reduce this particular loss function.
Now, in order to reduce this particular loss function, we have to do a back propagation and we have
to update all these weights that are available over here.
Right.
So, uh, in this video we'll be understanding how the back propagation will happen and how the weights
will get updated in a simple RNN.
Now, before I go ahead, what I will do is that I will again quickly revise our forward propagation.
So let's say this is the basic architecture of a simple RNN.
And here you know that we use a self feedback loop.
And here also we pass some kind of inputs.
So here also some weights will be involved okay.
But when we unfold this with respect to time okay.
And let's consider I'm going to take my sentence one.
It has this X11X12X13.
Let's say that my sentence one has this three words and this is my output.
It can be one and zero okay.
Okay.
Similarly, my sentence 2nd May have another one like X21X22X23.
Right.
And let's say this is the output zero.
And similarly my sentence 3rd May have something called as X31X32X33.
Okay, I'm just considering three words at a time okay.
You can have any number of words.
That is up to you.
And this word may also vary because at the end of the day, we will be converting these words into some
kind of vectors by using one hot encoding.
By using word two vec, it is up to us.
Okay?
And we will be passing every input one at a time based on the time stamp.
But if I really want to give this generic format.
So here I can go ahead and write something like this.
I will make this sentence one as generic.
I'll say, hey, this will be x I of one, then this will be x I of two and this will be x I of three.
Okay, so here I is, nothing, but it will be the sentence number okay.
Sentence number.
Perfect.
So I'm just giving some basic information over here.
Now let's do one thing.
Um, let's unfold this entire RNN simple RNN with respect to time.
Now, you know, with respect to time when my T is equal to one, I will be passing my first input,
right?
So I will be going and passing my first word.
So let me just go ahead and define this.
And this will basically be my first word okay.
So at t is equal to one I am going to pass x of I one okay.
Now initially I will also pass some kind of weights over here.
Let's say this is my hidden weights.
By default, some weights will be getting assigned.
And uh, initially I will not put this.
Okay, let let this be empty for now.
Okay.
I will just go unfold this entire network over here.
We are just going to unfold it now, once we give a input at a time, step T1T is equal to one.
Here some weights will get assigned.
So let me just go ahead and write this as w of I.
Okay.
Now you know that, um, what will happen in the forward propagation.
We multiply with this specific weights.
Then we add a bias.
And after this we get a output.
So this will be my output right.
And this output will be nothing but oh one.
Let's consider oh one.
So this will be my oh one output.
Now you know that my output I need to send it back to this particular RNN, right?
To this particular hidden layer and all the other hidden neurons in that layer.
So this will be sent to my neuron.
Hidden neuron okay.
The same output will also be sent to my hidden neuron.
But at time step t is equal to two I will be passing my next input.
So here it will be writing right XI2.
Okay, that basically means we are just passing the second input with t is equal to two.
Again the same weights will get initialized over here.
Along with this we will go ahead and initialize one more weight over here w of h.
Okay.
So this is nothing but this is the hidden neurons wet weight right.
If you remember over here in this hidden neurons I told you right.
We are also going to assign this all weights.
Right.
This all weights will be also getting assigned.
Right.
And uh, the next step will be that we will multiply with this and this, and we'll add o one with w
h.
And finally, when we combine both of them, you'll be able to see that we will be getting our next
output.
That is nothing but O2.
Now with respect to this particular O2, again I will go ahead and use my uh, the the weights will
get assigned in this.
And here I'm actually going to get my next input again at t is equal to three I'm going to pass x of
I three okay.
And finally I will get my output which is called as o4 okay.
Now with respect to this particular output of O4 okay.
Again I will go ahead and pass this to my.
Sigmoid if it is a binary classification.
Right.
And finally I get my output over here, which is my y hat.
Okay, so this is how I had also written it over here earlier when you saw this entire steps.
Let me just show you once again.
So here, right.
We were getting this offer.
We send it to the sigmoid.
And finally, if it is a binary output we use sigmoid.
Otherwise we use softmax if it is a multi-class classification, we specifically use something called
as softmax.
And here also you know that with respect to this offer, I will also assign some weights, uh, uh,
with respect to this.
So let's say here also we go ahead and use w of H.
Okay.
Now and here uh there will be a weight of I that will get initialized over here.
Now, with respect to the forward propagation so quickly, let me just go ahead and write.
So here is my forward propagation.
Now in forward propagation what happens.
Okay.
Uh, one more important thing is that, guys, uh, initially I did not pass any input, right?
Because, uh, when we are starting this, we will also initialize some kind of input weights.
And here, when I am initializing some kind of input weights, obviously I will also initialize some
w h over here.
Okay.
This will be my w h.
Along with that, uh, you'll be seeing that since every layer has some kind of output here.
Also, we need to make some kind of output right.
We need to probably initialize some kind of output.
So here I will be getting my O zero.
This o zero can be initialized as zeros.
You know it can be initialized any values as such, right.
But randomly it can also be initialized.
So here we can also initialize it zeros or here also randomly any values can be initialized based on
some initialization factor right.
The initial output.
And again here also you'll be having this hidden weight okay.
Now in forward propagation what happens is that whenever we go with the forward propagation the first
thing that we need to compute is O one.
Right.
So O one needs to be computed over here now Now in order to compute oh one.
What I will do is that I will just go ahead and and in every hidden neuron I apply some kind of activation
function.
So let's go ahead and apply this activation function f.
Let's say the activation function in this particular case is tan h.
Right.
So I will go ahead and multiply x of I I x of I one multiplied by w of I plus.
Then you can see O zero is also there.
This this, uh, o zero is nothing, but it is zeros only.
Right.
I'll be just considering that we have initialized all the values zero, so zero multiplied by this hidden
weight will be zero only.
So that entire thing.
Right?
It will be zero.
Because we need to sum this up.
Then we have a bias.
Let's consider a bias over here.
In every hidden neuron we will be having a bias right?
So let's consider this as a bias okay.
Now this is my first operation with respect to this thing.
Right.
And one more important thing over here is that I will just go ahead and write this also so that you
will not get confused.
So I'll say, hey, my TO0 multiplied by w h plus bias okay b one.
And then we basically go ahead and apply an activation function okay.
Now coming to my next output O2 okay.
Now this is how the forward propagation will happen.
Now in forward propagation of O2 my next word is basically sent.
So this will nothing be but it will be a function of x of I two multiplied by w of I plus.
Now o one and w h is also there.
So I will go ahead and multiply o one multiplied by w h.
Plus they will again be a bias over here.
Let's consider this bias as b.
And here I'm going to use one another activation function.
This is how we can go ahead and calculate O2 right now.
Similarly when I go ahead and send my third word that is to calculate O3, it will be nothing but X13
multiplied by w of I plus o two multiplied by w of h plus b bias right o two multiplied by w of h,
w of h is nothing but my hidden weights.
Okay, so everything looks perfectly fine over here.
Now finally, when we get our output right, we send this output to a sigmoid activation function and
then only we'll be able to get my y hat.
So in order to get the y hat, let's say the sigmoid activation function, I'm going to apply over there.
it will be nothing but oh three multiplied by.
So here you can see 0300300 sorry I did not write I wrote oh four over here.
So it should be oh three.
This is my output oh three multiplied by W of H.
Right.
And then uh, we send it to the sigmoid to get my output right.
So here what we are doing.
We are just going to multiply oh three with w of h o.
So once I probably do all these things over here c o3 w of h.
Instead of writing w of h here I'll not use hidden weights because this will be my output weights.
So let me just go ahead and write w of zero okay.
So this will basically be my W of zero.
Perfect right.
So this is how the forward propagation usually happens.
And finally when we pass it to sigmoid or softmax I'm going to get my y hat okay O3 multiplied by w
of zero.
So here we are specifically going to apply w of zero because this is my output weights right.
And finally this is my output node where I'm getting my output okay.
Now this is fine.
This is perfectly till here everything uh I think we have discussed this in our previous video also.
Now the time comes is that after we calculate y hat now once we calculate y hat, what will happen?
We will take y minus y hat and we will calculate the loss.
Right?
There will be some loss value.
Now our main aim is to reduce this loss and how we will be able to reduce this loss only when only when
we update update all the weights.
Now what are the weights that are available over here?
Uh, as I probably see, there is, there are three weights, uh, that will be involved over here.
One is w of I.
The next one is w of h and w of h w of I.
And one more weight is w of zero.
Okay, so this three weights needs to keep on getting updated during the back propagation.
Right.
This is the most important thing that if we really need to update w of I, w of x and w of zero, right,
I need to probably update this weights itself and probably do the backward propagation, and then continuously
do the forward and the backward propagation in similar way.
And we'll be doing unless and until this.
Till when?
Till our entire convergence will happen in this gradient.
Descent will reach this global minima, or when my loss will be really less.
This is equivalent to when my loss will be really, really it will be less right?
Now let me just go ahead and quickly show you with respect to the back propagation, how does back propagation
actually happen.
So I will just go ahead and create one more cell over here.
Okay.
So this is my forward propagation.
Now in the backward propagation with respect to time.
Backward propagation with time.
Now see at the end you know over here when we do the forward propagation we finally go ahead and calculate
our loss.
Right?
So this is our loss.
So we go ahead and calculate our loss okay.
Now once we calculate this loss okay.
Please focus on this okay.
Once we calculate this loss the first output.
So our first weight that we need to update is w o.
Now in order to update w.
And as I said we have to update which all weights we have to update w of I, w of h and w of O.
Okay, we need to update this three weights.
Now in the backward propagation the first weight will get updated based on your timestamp.
So let's say timestamp is equal to three.
The first weight that is coming is w four.
So in order to update w four how do I do it.
So here I will be using a weight updation formula.
And I hope everybody remembers this.
What is the weight updation formula.
It is nothing, but you will be able to see that weight of new is equal to weight of old minus some
learning rate.
Derivative of loss with respect to derivative of w old.
Okay.
Now this formula I hope we are very much familiar because we have also discussed this in an but out
of it this is the most important one okay.
Because of this here when we are doing this, this is basically my derivative.
And why we are doing it.
Because we need to calculate the slope of.
Slope of.
Gradient descent.
We need to go ahead and calculate this slope of gradient descent.
Okay.
Now my question is that, uh, our question is that how do we solve this?
Now see.
First is derivative of loss with respect to derivative of w old.
So first weight that is.
W o.
So first thing that we really need to calculate over here is that derivative of loss with respect to
derivative of w old.
This is what I really need to calculate right.
If I calculate this then only see my.
This weight is nothing but derivative of loss with respect to derivative of w o.
Right.
And if I talk with respect to the weight updation formula, this will be nothing but derivative of O
is equal to derivative of O old or I'll write like this.
Derivative of o new.
Sorry w of o new is equal to w of o old minus learning rate derivative of loss with respect to derivative
of w o old.
Okay.
So this will be old.
So this is my entire weight updation formula for w o right.
But here I need to calculate this.
This is fine.
I'll be getting the old value now in order to get the new value I have the learning rate.
Learning rate will be a small value.
So let's consider learning rate will be something like 0.001 okay.
But I need to compute this.
So in order to compute this derivative of loss with respect to derivative of w old, how do I probably
write this entire or how do I find this?
Okay.
Now it is important, uh, for you all to understand is that here w o is directly dependent on y hat,
and y hat is also dependent on loss, because at the end of the day, we are calculating the loss.
So here you can see there is a direct relationship.
The loss is dependent on y hat and y hat is dependent on w zero.
Right.
So if I want to probably expand this equation based on chain rule.
So this based on chain rule of derivative.
If you remember this we have already discussed.
So based on chain rule of derivative you will be able to see that I'll be having derivative of loss
with respect to derivative of w old.
Instead of writing w old.
I know loss is dependent on y hat, so I'll go ahead and write derivative of loss with respect to derivative
of y hat.
And similarly I will go ahead and say derivative of y hat is dependent on.
Now I can go ahead and write.
Derivative of y hat is dependent on what w o old right?
Now once we do this, we will be able to get this value.
And this is nothing.
But this is a chain rule of derivative, right?
And through this you will be able to see that we will be able to calculate or will be able to update
the weights of w o new.
Okay.
This is perfect.
So our first first main thing is that how do we update.
w zero?
That has been very much cleared.
Okay.
Because there are three types of weight.
One is w zero, one is w of h and one is w of I.
So here with respect to time stamp here you can see that I have updated my W of zero.
Now coming to the second important case we need to update w of H.
And this is nothing but this is my hidden weights.
Right?
Hidden layer weights.
Now this weights are important because this weight is used in all timestamps, right?
T is equal to one, t is equal to two, t is equal to three, t is equal to four.
So we have used this in each and every timestamp.
Now how do we do that okay.
So here you can see now we what we are going to basically do while updating this is that it's very much
simple okay.
And first of all let me just go ahead and write the weight updation formula here.
So I'll say hey w of h new is equal to w of h old minus learning rate derivative of loss with respect
to derivative of w h old.
So this is the formula the weight updation formula.
And out of this formula the main thing that I have to compute is nothing but derivative of loss with
respect to derivative of w h old.
Okay, this we need to compute.
So in order to compute this how do I write.
Now derivative of loss with respect to derivative of w h Old.
How do I compute it?
I have to make sure that how many time stamps are there in my entire RNN, t is equal to one, two,
three, so I have to make sure that I have to find this based on my timestamp.
T is equal to one, two, and three.
Now how do I do it?
Okay, first of all let's in the back propagation.
My first timestamp is three.
Now I need to update which weight over here I need to update this w of H right?
I need to specifically update my w of h weights.
Okay okay.
Perfect.
Uh, so let's go ahead and do that okay.
And here you can see there is w of h and there is w of I.
Okay.
So uh I will just try to show you how we will update this w of h.
So w of oh we saw it how we can update it.
Now let's go ahead and see with respect to W of H now this hidden weight, you know that this is dependent
on O3.
See just try to always make a rule like this is dependent on O3.
O3 is dependent on y hat and y hat is further dependent on Los.
Okay.
So here you can clearly see this.
First of all Los like uh, if I go from this particular because I'm searching with respect to time is
equal to three, right?
And time is equal to three.
I have w h over here.
Now first of all this is dependent on O3.
O3 is further dependent on y hat right.
And y hat is further dependent on loss.
Right.
So there is a kind of dependency right.
And vice versa.
Also I can say right, I'll not say okay w h is dependent on O3 and I'll say O3 is dependent on w h.
Or you can also think in this way.
Let me reframe it.
Okay.
So I have my loss.
Loss is dependent on y hat y hat is dependent on o three, and o three is dependent on w h.
Now in order to update this I have to follow this chain okay I have to follow this chain okay.
So let me just go ahead and write it and how it will look like.
So first of all I'll go ahead and write derivative of Loss.
Let me go ahead and write this.
So here I'll go ahead and write derivative of loss with respect to derivative of y hat.
Because this is the first dependency.
Then derivative of y hat will be dependent on what.
Derivative of derivative of o.
Or let me just see this weights okay o w for okay.
So derivative of w for or instead of writing derivative of w for.
I know this will be dependent on derivative of O3 because O3 is the output over here, right?
O3 is the output.
Then similarly this O3 will be further dependent on derivative of O2, right?
Now I know that in time stamp t is equal three I have to first of all update this weight.
So what I will do instead of writing O2, I will go ahead and write derivative of O sorry derivative
of.
Derivative of w h w h.
Okay, now this is very much important because when t is equal to three I'm doing the back propagation.
And I'm finding out this particular derivative of this value in this way okay.
Now I will take this as one okay.
One entire thing and we'll update this will will calculate it okay.
Now we will go ahead and write plus y because at time stamp t is equal to two I still have w of h over
here okay.
Now what kind of rule we will follow or what kind of chain will follow.
So here you can see loss is dependent on y hat Y hat is dependent on O3.
O3 is now dependent on O2 and O2 is now dependent on w h.
Now in order to update this w of H, please see this.
Okay.
In order to update this w of H, I have to follow this chain rule.
Okay so here what I will go ahead and write again derivative.
So I will start from here okay I will go ahead and write at timestamp t is equal to two.
How I will update this.
First of all I will go ahead and write derivative of l with respect to derivative of y hat.
Then I will multiply derivative of y hat derivative of y hat with respect to derivative of o3.
Then I have derivative of O3 with respect to derivative of O2.
Because there is also a dependency over there, and then with respect to derivative of O2, I will be
calculating the derivative of w h.
So this will be my For time stamp t is equal to two.
How I will go ahead and calculate my w of h.
Right?
This is how based on the chain rule it will happen again.
Let me repeat it.
Loss is dependent on y hat Y hat is dependent on O3.
O3 is dependent on O2.
O2 is dependent on w h.
And that is how I have actually calculated my derivative.
Now comes to the next point.
And the next point is nothing.
But with time stamp t is equal to one.
Now how do I calculate it?
So here what I'm actually going to do I'm going to probably create this chain.
Now I need to update this particular value that is w of h right now.
How do I calculate it.
Now you can see there will be a dependency.
Now from here I'll say O two is dependent on 0101 is dependent on w h.
Right.
So first we'll say loss.
Loss is dependent on y hat.
Y hat is dependent on O3.
O3 is dependent on O2.
O2 is dependent on O1.
And finally O1 is dependent on w h.
And that is how this chain rule will be working.
Okay, now if I really want to write this again so it will be nothing.
But here I'm going to write derivative of loss with respect to derivative of y hat multiplied by derivative
of y hat multiplied by derivative of O3 based on the chain rule derivative O3 divided by derivative
of 020202.
Then again, I will go ahead and calculate derivative oh two divided by derivative of oh one.
I think it is oh one right oh one.
Then oh one is derivative of.
Then oh one is dependent on w f uh w of h w of w of h.
So here I have my next one.
And once we update all, once we add up all this particular values that we are going to get.
That is when we are going to get this entire value.
And we'll go ahead and update it.
And our w h new will be the value with respect to this old minus learning rate multiplied by this whatever
value we specifically get.
And that is how in one back propagation, all the hidden weights gets updated.
Okay, this is perfect, right?
So I hope you are able to get how to update w h, which is my hidden layer weight based on timestamp.
Now coming to the third and the very most important thing that is w of I.
Okay, so third is nothing but updating weights updating weights.
W of I.
Now let's go ahead and discuss about this.
So guys now we are going to probably see how we can update weights w of I again based on timestamp okay.
Because that t is equal to three I'm sending one word at t is equal to two I'm sending one word and
at t is equal to one I'm sending one word.
So we are just talking about these weights right.
So this weights we are we also have to update in the back propagation.
So let's go ahead and see step by step okay.
Now first of all for this what kind of chain rule I'll go ahead and apply.
So here you can see that uh first I have loss.
The loss is dependent on y hat Y hat is dependent on O3.
O3 is dependent on w of I.
Right.
So again for this the weight updation formula will look something like this.
So here I will go ahead and write w of I nu is equal to w of I old minus learning rate of derivative
of loss with respect to derivative of w of I old.
Okay.
Now as you know we have to go ahead and compute this okay.
And this will be with respect to time.
Now quickly derivative of loss with respect to derivative of w I old is equal to I know first of all
loss.
This uh derivative of loss.
So this will be with respect to time stamp t is equal to three okay.
We have to do the back propagation.
So here you can see derivative of loss with respect to derivative of w.
Um here you know that loss is dependent on loss is dependent on y hat.
Okay, then I have y hat.
It is in turn dependent on.
So y hat is dependent on what y hat is dependent on.
Oh three.
So I'll just go ahead and compute 030.
Then oh three is dependent on w of I.
Right.
And when I say w of I this is nothing but w of I hold.
Okay so this is with respect to my timestamp t is equal to three okay.
Similarly we'll go ahead and write plus.
Now let's go with respect to timestamp two.
Now with respect to timestamp two you know that how we are going to go y is dependent on y hat.
Y hat is dependent on O3.
O3 is dependent on O2.
O2 is dependent on w of I.
Right.
So we will go in this rule.
So first of all I will say hey let's go this.
So this you have one dependency loss with this.
Then oh three then oh three from oh three to oh two and oh two to this w of I.
Okay.
So we will go ahead and write this so quickly.
Let's go ahead and do this.
So here I'll be writing derivative of loss with respect to derivative of y hat multiplied by derivative
of y hat multiply divided by derivative of O3.
Then you have derivative of O3 divided by derivative of O2.
And finally you have derivative of O2 divided by derivative of w of I old.
So here is my total thing right.
So O3 to O2.
See I'll see O3 to 020 2 to 5.
So oh two to oh five.
Right.
And finally you'll also be seeing that we will go ahead and write my final four.
And this is for t is equal to two.
Now let's go ahead and write it.
For t is equal to one, t is equal to one is very simple derivative of w of y hat multiplied by derivative
of y hat with respect to oh three.
Derivative of oh O3 with respect to derivative of O2.
Then oh two to oh one and finally oh one to w of I.
Old.
Once we calculate this all values, you will be able to see that I'll be getting this entire value over
here and this will be replaced over here.
Right.
And finally we update w of I update W of I weights in the back propagation.
And this usually happens in one back propagation with respect to time.
So finally when we are using this weight updation formula what exactly is happening.
Right.
And I hope everybody knows about gradient descent okay.
Our main aim is to probably come to this global minima.
Right.
And there are different, different weights.
One is W of h, that is w of I.
And there is something called as w of o right.
I my main aim is that whenever this convergence basically happens we have to probably come towards this
global minima, right?
If we are coming towards this global minima, that basically means our loss will be very much near to
zero or our loss will keep on decreasing.
Right.
And this is what we had already discussed in an regarding gradient descent.
So I hope you like this particular video.
This was it for my side.
Uh, here in this video we have discussed about RNN, simple RNN, we have discussed about forward propagation.
We have discussed about backward propagation with all the equations at all.
Right.
So yes, this was it from my side.
I will see you in the next video.
Thank you.