Hello guys.
So we are going to continue the discussion with respect to RNN.
Uh, already we have seen about forward propagation in RNN.
We have seen about backward propagation in RNN.
Now we are going to understand what are some of the problems with RNN okay.
And because of this, you know, uh, the next neural network after RNN, which we are specifically
going to discuss about is LSTM, RNN and GRU RNN.
Right.
Why why we require this variance of our own when we already have a simple RNN.
Okay, so we'll be discussing about that.
And in the upcoming videos we will be discussing about LSTM, RNN and grew RNN.
So let me just go back and probably revise some of the concepts with respect to RNN.
So here is what my RNN looks like right?
A simple RNN I have a seed feedback loop and then when we unfold this, you will be able to see that
I will be getting something like this and this will basically be my.
The output will go to my hidden layer.
Again with respect to the specific neuron.
This is my input.
And again it will go like this.
And it'll keep on going to the next stage till we get the last word with respect to the time stamp.
Right.
And once we get this, what happens once we get the output here we are just going to apply a sigmoid
or softmax activation function.
I'll say soft or sigmoid.
And finally we will be getting a y hat.
And later on we go ahead and calculate our loss.
Y minus y hat.
And our main aim is to reduce this loss.
So what we do we basically do the back propagation.
And if I just talk about over here I will be having w of I here I will be having w of h.
Okay.
this will also be w of h.
And here I will be having one input uh which will be like initialized to zero initially.
Then similarly this will also be w of h.
Here I will be getting w of I w of I w of h.
Similarly w of I.
And here finally we get w of zero.
And with respect to this particular output here, you will be seeing that I will be having 010203 like
this up to oh N right.
So we discussed about this entire process and we have discussed about forward and backward propagation.
Now let's understand what is the problem with RNN okay.
And I hope in an uh when we were discussing, we discussed about a very important topic which is called
as vanishing gradient problem.
Vanishing gradient problem.
Okay.
So let's go ahead and discuss about this again.
And here also in our simple RNN we may face this problem.
Now the thing is that you really need to understand how do we face this particular problem okay.
Now let's consider in in my data set I have some text I have some output okay.
Till now we just saw smaller sentences.
Let's say I have four words like the food is good right.
So my output will be one.
The food is bad so the output is zero okay.
When we are we're using this all kind of words right.
Or these all kind of text or sentences.
This sentences were small.
This sentences was specifically small.
Right.
And instead of taking this particular use case let's consider one more use case which is called as text
generation.
Okay.
Let's say some of the example with respect to text generation.
I will go ahead and write.
Hey, um, I'll take a very simple example.
I like to play Dash.
Okay.
So here I need to probably go ahead and predict my next word.
It can be cricket, it can be football.
It can be anything.
Right.
And over here this word will be predicted based on the context or based on the previous words that we
have.
Right.
For this particular sentence, if I have this sentence S1 over here, you can see this word may be actually
dependent on this word, or it can be on this word, or it can be on this word, or it can be on this
word.
Right.
If I consider and if I try to train my neural network right here, you can basically consider that this
has a kind of short term dependencies.
Now when I say short term dependency, what does this basically mean?
Right.
Let's understand this with a very simple example.
Right now here I have specifically written that I like to play.
I need to predict the next word.
Now this word may be dependent on this word or this word or this word or this word.
Okay.
So here you can see based on the length of this entire sentence.
Right.
Based on this length of this entire sentence, they hardly four words.
Right.
So I have a dependency within this shorter sentence.
Okay.
But let's say I will go ahead and write a new sentence.
Let's say I'll write.
Hey, my name is Krish.
And I like sports.
Like cricket, volleyball okay.
And I.
Like.
Let's say I'll go ahead and write and also like to make.
And this will be dash over here I need to predict this particular word.
Now this word can be videos.
I like to make food.
It can be different.
Different words.
Now here, in order to determine this specific output.
Now the dependency is on this word right.
It may be on this word.
It may be on this or this or it may also be on other words.
But here you can see this sentence.
Sentence is long is really long.
right now when this kind of sentence is long.
Just imagine if I am passing through my RNN.
So how many timestamps I will be requiring I will be requiring T is equal to one.
I may require t is equal to three four.
Like this.
If I keep on counting one, two, three, four, five, six, seven, eight, 910 1112 1314, 15, 16.
So at t 16 I will be sending this word for my forward propagation.
Okay.
Now similarly this is just a smaller set.
This is still I'll say a sentence with a good length.
But let's say if I have a sentence which may have 100 words.
Which may have 100 words.
So at t is equal to 100, I will be sending my last word for the forward and the backward propagation
right to calculate the loss.
Now what will happen as the sentence keeps on getting longer and longer, right?
The final output that I really want to get right.
This may be dependent on some words that may have started earlier, right?
That may have come up in time.
Step T is equal to 432.
And this dependency cannot be captured by simple RNN okay.
Okay, so let me just go ahead and write it down.
White cannot be captured.
I will be explaining about it.
Right.
The long term dependency.
Cannot be captured by RNN and hence RNN cannot provide a better accuracy in this kind of use cases where
it has this kind of dependency, where it has this kind of dependency.
Now, you may be thinking, Krish, uh, whatever things you are actually explaining over here, it
it it hardly makes sense.
Please try to give some example.
Obviously I'm not going to stop over here right now.
Let's take one example.
Let's say I'm going to use this entire neural network again.
Okay.
I will just go ahead and copy this.
I will go ahead and copy this and let's paste it over here okay.
Let's go and do this okay.
Let's say for the first instance I give my
XI1X2XI3.
Okay, so in one of the sentences I just had four words right?
Or I just had three words.
And after that I got my output I passed this it to my sigmoid.
So if I really want to show you with respect to, let's say in one of the sentences, I just had three
words.
So my unfolding will basically happen like this.
So at t is equal to one I will be giving my X11 as t is equal to two, I'll be giving X12.
Then my X13.
And finally when I get my output this will be passed.
This will be my output here.
Instead of writing hidden I will be writing something like this.
I will be sending it to my.
I'll be sending it to my users.
Consider this as sigmoid.
Okay, so I'll be sending it to my sigmoid.
Let me just draw this properly.
Okay.
So here I'll be sending it to my sigmoid to perform the to capture or to calculate the y hat.
Okay.
okay.
So this is pretty much easy.
And here I will be saying that this will be my W zero okay.
Now in this case when I try to find out the weights right I need to update weights over here as w or
h w of I and w of zero.
Now in order to calculate this weights again the generic formula is nothing, but w new is equal to
w old minus learning rate of derivative of loss with respect to derivative of w old right.
And out of this, what is the main thing that I really need to calculate is this particular value.
Right.
This.
And this value needs to be calculated for first of all I will go ahead and write w of zero right.
This uh loss with respect to zero here I will go ahead and calculate my loss Y minus y hat.
Okay, now, when I calculate w, uh, derivative of loss with respect to derivative of w zero.
So here you will be able to see the equation will be very much simple.
So I will go ahead and write derivative of loss with respect to derivative of w uh y hat derivative
of y hat with respect to derivative of um w ho.
Okay.
So for this the calculation will be very much easier.
But now let's go ahead and consider one of the example.
Let's say I want to update or I want to update this hidden weight okay this this w h okay.
So let's go ahead and update this hidden weight okay I'll take this example.
So for this I will go ahead and write w derivative or uh l with respect to derivative of h old.
Now you know that my loss my loss, which is getting calculated, is dependent on.
In this particular case, it is dependent on 0303 is dependent on O2, and O2 is finally dependent on
w h.
Okay.
So here what I will do I'll write simply like derivative of loss with respect to derivative of y hat.
Sorry.
Loss is dependent on y hat, Y hat is dependent on y hat is probably dependent on O3.
Okay.
So so now I will go ahead and write derivative of y hat with respect to derivative of oh three.
Okay.
And this is simple chain rule that we have actually discussed.
Derivative of oh three.
Okay.
Uh then we'll go and write derivative of oh three to oh two.
Then we'll go ahead and write derivative of oh two to derivative of w h old.
Now remember one thing here we are.
Calculating with timestamp is equal to t is equal to one.
Right.
If I really want to calculate all the derivatives I need to keep on adding for t is equal to two.
I need to add so I can go ahead and take this entire value.
Okay.
So at t is equal to two I can go ahead and add that.
So at t is equal to two.
Let's say I want to go ahead and do this.
So I will be having a different derivative.
Or I'll be having this differential uh separate chain rule over here.
I'll be able to get it.
Then at t is equal to three I will be able to get it a different chain rule okay.
But I just want to focus over here right now in this right now here when you are seeing this this this
is a specific chain rule that I have okay.
And here you can see that with respect to t is equal to one.
When my total number of words were just three, when t is equal to one, I was able to get this big,
uh, chain rule of derivatives.
Okay.
Now, if I go ahead and consider if my length of one of my sentence, if my length of one of the sentence
is having 50 words.
Now, in the case of 50 words, now just imagine what will happen to this particular W of like if I
go ahead and try to calculate the derivative of loss with respect to derivative of w hidden, please
focus on this.
Okay?
If my length of the word is 50 words, so at t is equal to one, how am I equation will look.
So I will go ahead and write like this.
So at t is equal to 50 or sorry at t is equal to like total number of time stamps will be 50.
And I want to find out derivative of loss with respect to derivative of w old with respect to t is equal
to one right.
Now this same equation it will come right.
And here what it will become.
See now my derivative of loss will be dependent on derivative of y hat.
Next step will be that derivative of y hat will be dependent on what derivative of O.
Whether it will be three or whether it will be 50.
So now I'll go ahead and write 50 because I'm going to go ahead with till t is equal to 50.
So my last step over here will basically be O 50.
The output 50.
That will be calculating right.
Then similarly we will go ahead and calculate derivative of O 50 with respect to derivative of oh 49.
Similarly, we'll go ahead and write derivative of oh 49 with respect to derivative of oh 48.
And similarly.
Like this.
We'll keep on going till what step.
We'll be going till we get this one right, till we get derivative of oh two with respect to derivative
of w h old right.
So at t is equal to one.
This will be my first step.
Right at t is equal to one.
This will be the value that I really need to compute.
Then again at t is equal to t two.
I'll keep on writing it right.
And again over there.
Hardly.
From 49 it will start.
From 49 it will start and it will end till here.
Then for t is equal to three it will start from 48 and then it will end till here.
Right.
Same thing will basically be happening along with this right now.
Let's focus on this important things right.
Let's focus on these values.
These values okay.
Let's focus on these values.
These are very important values.
And we will take this up okay.
Let's let's consider one of the equation.
Like uh out of all this I will go ahead and take one more over here derivative of O3 with respect to
derivative of O2 plus uh, sorry, multiplied by this last one.
Okay.
So what I'm saying is that if I keep on doing this right, if I keep on going this like this, it will
continue till this particular value.
Right.
And before this value, I also have something like derivative of O3 divided by derivative of O2.
Now let's go ahead and calculate this okay.
We will just go ahead and calculate it.
Now derivative of O3.
What is O3 over here.
O3 is nothing but it is very much simple.
We basically take this w of h c.
Let's go ahead and see what exactly is O3 okay.
O3 is nothing but derivative of.
Let's say I'm going to apply a sigmoid activation function on this node.
Okay.
Let's consider this for for O3.
I'm just considering okay I'm just taking an example.
So O3 is nothing.
But first of all, I'll just go ahead and write about oh three.
Okay.
What is oh three over here?
Oh three is nothing, but it is, uh, how we are getting this particular output.
It is very much simple.
First of all, I will go ahead and multiply x of I three multiplied by w of I.
Right.
So let me just go ahead and write it down over here x of uh I three multiplied by W of I plus.
Okay, plus we are multiplying with w of h multiplied by o two.
Right?
So here I will go ahead and say o two multiplied by w of h.
Right.
So this is my derivative of this o three value okay divided by derivative of o two divided by derivative
of O2.
Now once we perform this derivative, since we are doing it with respect to O2, this will become one.
Okay, this will become one.
So I will be getting this w h and this will be considered as a.
This will be considered as a constant.
But before this you will be able to see that when I am applying derivative over here on top of it,
I also have to perform a sigmoid activation function on this node.
I have to perform a on top of it.
I also need to perform sigmoid activation function because there will be a sigmoid activation function
over here.
Along with this, you'll also be seeing that I will also have one bias parameter.
So this will be plus bias okay.
Now this will how the equation look like.
So I'm saying hey derivative of sigmoid of this x of I three multiplied by w of I plus o two multiplied
by w of h plus one plus b okay.
And this is entirely my o three right.
This is what I have replaced o three by.
Right.
So o three over here means what o three is nothing, but it is sigmoid of sigmoid of XI3 multiplied
by w of I plus o two multiplied by w of h plus B, right?
Now if I go ahead and find out the derivative how it will work.
Derivative.
If you know I will get this is sigma dash I will right because this is the derivative.
Now further going ahead you will be able to see that.
It's like if you use the derivative for this entire thing okay.
If you probably use this entirely, this particular derivative, this will be nothing.
But here you can specifically write one multiplied by w of h, okay, one multiplied by w of x.
Because this will be a constant, this will be a constant.
This will become zero.
Okay.
So I have something called as one.
Multiply by w of h.
That's it.
Okay.
So here is what I'm actually going to get okay.
So this is nothing but derivative of sigmoid.
And here I'm just going to get one multiplied by w of h.
Now this will be nothing.
But whenever we try to find out the derivative of sigmoid.
So I have actually explained this in my activation function.
So derivative of sigmoid ranges between 0 to 0.25.
The output of the sigmoid ranges between 0 to 1.
But whenever I try to find out the derivative it will be ranging between 0 to 0.25.
So this value that you are going to get is nothing but 0 to 0.25.
In between this values you will be getting.
Now here you can see that you observe one thing, right?
One of the value that I calculated in this multiplication, which was this one okay.
This one was ranging between 0 to 0.25 okay.
Now as we go ahead right now this is a decimal.
Let's say that we went and calculated here.
We got a decimal as .015.
Okay I just calculated it randomly.
Then let's say for this I got 0.02.
Right.
For this I got 0.25.
Right.
For this I got point.
Uh, not for this.
For one of the internal one, I got 0.01.
Now, when I keep on doing this multiplication, you'll be seeing that at after some point, this will
become a very small value.
And it will be approximately equal to zero.
Now when this is approximately equal to zero, that basically means what at t is equal to one.
Where I had this first word that I'm actually giving it is not participating much to update the weight
value.
Right.
So what is happening over here?
The t is equal to one.
The word.
is not participating.
Participating to update the weights.
Value.
Do you agree or not?
Just see this white is not participating because this entire operation will be approximately equal to
zero.
Because we are just multiplying with these numbers, right?
These numbers.
And as you know that if you keep on multiplying with decimal numbers, you will be seeing that, hey,
we'll be getting a value which will be approximately equal to zero.
It will become smaller value, a very, very small value.
Now, since this entire value is becoming zero, you'll be able to see that when I will try to add this
number, or when I will try to add this with my next timestamp, it is not probably going to impact
the weights, right?
It is not going to change this specific value, right?
You will be seeing so what is basically happening over here, here.
In short, it is saying that the word that you initially gave in our simple RNN, when compared to the
last word, this word is not playing a good or amazing impact in the simple RNN to probably predict
the outcome or the output.
Right.
It is very much simple over here, right?
Whichever will be the nearest value with respect to the time stamp, that may create a bigger impact.
The reason is very much simple.
Why?
Because when you perform the operation, the operation will be very small.
So here when I am trying to calculate, let's let's calculate for O3.
Right.
Just just for this last time timestamp.
If I'm calculating if I'm trying to find out O3 loss is dependent on y hat, Y hat is dependent on O3,
right.
Or loss is dependent on y hat y hat is dependent on w o and w o is dependent on further dependent on
w if I.
If I want to update this, the chain rule will become very smaller.
But in this case the chain rule is big.
Now when the chain rule is big, when the chain rule is big.
right?
Big, big basically means the chain is very big over here.
Right over here you can see that whichever is our first word, that will not play a very big impact
to finally predict the last output.
Right.
And that is what we will be seeing because here we have a long term dependency.
Long term dependency.
Why?
Because in this case I took 50 words.
This was my first word.
This was my first word that I passed with respect to time stamp.
And if I'm updating this weights, it's hardly zero and it is not creating any impact for my final weight.
Right.
So this is the major problem.
And whenever we face this kind of problem right.
This is basically called as vanishing gradient problem okay.
Okay.
Vanishing gradient problem.
Right.
So this is very much important.
Let's consider at t is equal to 50.
What is going to happen okay.
So at t is equal to 50.
You'll be seeing if I go ahead and calculate my derivative of loss with respect to derivative of w h
old.
Now first I know my derivative of loss will be dependent on derivative of y hat.
Then I have derivative of y hat.
It will be dependent on what if I really want to just play with the w th last w h 50.
Right.
So this will basically be dependent on uh derivative of y hat will be dependent on my final output.
My final output will be o 50.
And derivative of O 50 will be dependent on derivative of w h old.
Right.
This is going to this is what is going to happen at t is equal to 50.
Now here you can see this is the right.
So this is basically t is equal to 50.
Right.
This is what I'm actually going to add at t is equal to 50.
Right.
So this here you can see my chain rule is very small.
And here I may be getting some value because the chain rule is small.
I don't have to probably do so much of calculation, so much of multiplication, like when we did it
for t is equal to one.
So similarly I will go ahead and multiply ad with t is equal to 49.
Then we will have t is equal to 58.
We'll keep on adding it till t is equal to one.
Finally we'll be having t is equal to one.
Here you could see that almost my value was approximately equal to zero.
Right.
And as we keep on going towards this timestamp, which is the last timestamp, you'll be seeing that
these words will play a very important role while predicting the next output.
Right.
So here you'll be able to see that these all values will never be equal to zero.
Right.
It it will have some values because this will be creating an impact on this.
And it this nearest words with respect to the timestamp will be responsible in updating the weights.
But some of the weights that are present in t is equal to one, t is equal to two.
These all values will be approximately equal to zero.
And that is what vanishing gradient says.
And this is the problem in simple RNN.
It is just focusing on it is just having a dependency in this entire chain rule.
It is having a dependency on the nearest word, not on the forest word that is available in that entire
sentence.
Right.
And this is the major problem.
Now how do we solve this problem?
See, instead I told here we are using sigmoid.
The derivative of sigmoid is 02.25.
We can also change this to tanh activation function.
But again in time you will be getting a derivative between 0 to 1.
But as the chain rule keeps on increasing, the multiplication will keep on happening and your small
you'll get a smaller value.
Okay.
So in order to solve this, uh, you can use other activation functions like ReLU okay, or leaky ReLU.
So this, uh, here, it makes sure that the derivative is always, uh, near to one.
Okay.
The other case is that researcher came up with an amazing one more network, which is called as LSTM
RNN.
Okay.
Now this LSTM RNN solves this problem of long short term memory.
And that is why it is called as long short term memory RNA.
Okay.
Not only this, uh, you'll also be seeing.
Along with this, we'll discuss about GRU RNA.
Okay, GRU.
And so both this RNN will solve the problems with respect to a simple RNA.
And that is what we are going to discuss about in the upcoming videos.
But I hope you have got an idea.
Right.
right?
Uh, if I have a long term dependency with respect to the sentences, then your simple RNN will not
be working well, right?
That is the major problem.
And I have also explained you why?
Because at time is equal to 50 steps.
You can see my chain rule was so, so big.
And when we did just one of the derivative problem statement, when we try to solve it here, you could
see that, hey, I'm getting this particular value with respect to weight, so this will be nothing,
but this will be weight.
Over here we are multiplying with the specific weight, but we are multiplying a smaller value okay.
Smaller value.
So I hope uh you are able to understand this particular video.
This was about the problems of RNN.
Yes.
I will see you all in the next video where we will be discussing about LSTM, RNN and Grant.
Thank you.