Hello guys.
So we are going to continue the discussion with respect to RNN.
And in this video we are going to see a variance of LSTM RNN.
Now uh we have already seen the entire LSTM architecture.
But in some of the research paper you will also find this kind of architecture that I have actually
mentioned over here.
Okay.
Now please focus on the diagram.
I would always suggest, uh, whenever you want to check different variants.
Right.
You know, what is RNN?
You know, the generic representation of RNN.
You know about LSTM, RNN.
Now just pause for hardly 10 to 15 seconds and just focus on this diagram okay.
And again suggest if you really want to learn all these things pretty much properly, you really need
to focus on all this representation of different different variants.
Right.
And just think within yourself like how it is different from LSTM RNN okay.
Because here also you have this forget gate, you have this input gate, you have this output gate.
But along with this you will be seeing some lines.
So what will happen if you get this kind of line.
Just just take 10 to 15 seconds and uh uh, you know think about it.
You can pause the video and you can think, but I will go ahead and continue the explanation over here.
Now, if I talk about LSTM, RNN.
So it's not like LSTM RNN is very new concept, right.
So LSTM RNN came in somewhere 1970s to 80s okay.
At this point of time then different different variants.
LSTM variants.
Uh came like this is one of the variant we will discuss about this.
And this was introduced by Girs and, uh, uh, Schmid.
Huber.
I guess I'm pronouncing it right.
If I'm not, I'm extremely sorry.
Okay, so this type of variant specifically came in 2000, and I have seen a lot of research paper where
they have specifically used a LSTM RNN.
So in some of the research paper I had seen this diagram.
Okay.
That is the reason I'm teaching you this different kind of variants also.
Okay.
Um, so obviously, uh, this is another kind of variant.
And, uh, you can just consider that.
Okay.
In some of the research paper you will find this.
And in some of the research paper we will find the normal LSTM, RNN what we had actually seen.
Okay.
Now let's go ahead and focus on this entire diagram okay.
Now here you can see that the input gate, the sorry the forget gate, the in forget the forget gate,
the input gate, the memory candidate, uh, the candidate memory and the output gate are almost same.
You know, you also have the CT minus one, you have CT, ht, minus one, you have HT, you have XT.
Everything is same.
But the additional thing that you specifically have is this line that you are sending from the memory
cell, uh, memory cell to both the forget gate and also to the out, uh, to the to the input gate,
along with the candidate memory.
And you are also sending it to the output gate.
Okay.
So what is basically happening in this variant from this particular variant you is you can actually
see a line that is probably coming up from the memory cell to the.
Again, let me repeat it to the forget gate, to the input and the memory candidate candidate memory,
and also to the output gate.
Okay.
So this is all information are specifically coming.
This connections, this connections, this connection from memory cell.
Two.
Forget gate.
Two.
Input gate.
And, uh, you know, you are also combining later on.
This is just going to the input gate right now okay.
So I'll not go ahead and say that.
Hey, we have to include candidate memory.
Candidate memory is done with a point wise operation Along with this, you also are sending it to the
output gate, right?
This connections are called as very important word that I'm going to use.
And you'll also see in the research paper which is called as peephole.
Connections.
Okay.
So this is basically called as peephole connections.
Very important.
Okay.
Peephole.
Connections.
Right.
So this is specifically called as peephole connection.
And as the term says.
Right.
What is this?
Peephole connections.
Let me just give a simple definition over here.
Peephole connection is nothing.
But, uh.
And you see this, okay?
They will be providing this entire information, uh, in some of the research and some research paper
will not be getting it.
Okay.
So here peephole connection means that through this architecture we let the gate layers look Look at
the cell state.
Okay.
So this is one additional thing, right, that we are specifically doing.
We are saying that hey, when you are from this architecture, we are just giving some functionality
to the forget gate, saying that, hey, if you want to make this memory cell, forget some of the information.
You can also take the previous context or previous information that is available in the memory cell
to decide what information you need to make it forget.
Right.
We are.
We are just saying that.
Hey, go ahead and look at the cell state and then probably take a decision.
By this, they will be able to take much more good decision with respect to forgetting, with respect
to adding some information to the memory cell and with respect to getting the output.
Okay.
So this is the entire funda behind this particular variant.
And here you can probably see the equations right.
We need to calculate this f of t.
This is there.
We need to calculate I of t and we need to calculate o of t.
Now in f of t only.
This term is basically getting added right.
This three important terms are basically getting added right before it was this two.
Then we also are combining this right.
We are also combining C of T.
Let's say C of t are four dimensions sorry three dimension.
Let's say t of minus one are three dimension.
Let's let's consider in this way.
So it will be three dimension.
And you also have x of t.
Let's say this is four dimension.
So we are going to combine everything.
And then we are going to give it to the neural network.
Let's say there are three neurons.
And let's say we are after this we are going to pass it to the sigmoid for the forget gate.
And then finally I'm going to get the output.
So this will get connected to each and every thing right over here.
And you can further do the connection based on this.
Right.
So this indicates my if I'm just considering three dimension.
This is my CT minus one.
This is my PT minus one for the forget gate for the first operation to calculate f of t.
So this is my f of t.
And then this is my x of t okay.
So this is how we are combining.
And similarly I of t we are combining CT minus one over here o of t.
We are combining CT because after both this operation we will be able to get CT over here.
We will be able to get CT over here, right?
So this was about one of the variant.
Uh, let's talk about one more variant okay.
So this specific variant okay.
Now this uh, variant uh, is also amazing.
Um, over here, this is, uh, I can just go ahead and write.
This is another variation.
Okay.
Where we are coupling this is very important where we are coupling or where we are combining forget
and input gates.
Okay.
Where we are combining forget and input gates.
So here you can see that, um, I'll just go ahead and provide you the definition, what we are specifically
doing.
Here we are saying instead of instead of separately deciding.
See, in my previous LSTM we had a separate forget gate.
We had a separate input gate.
With the help of forget gate we were separately deciding what to forget, right?
And with the help of input gate, we were separately deciding what to add to the memory cell.
Right?
So instead of separately deciding what to forget and what we should add, We should add in.
Add new information.
New info.
Okay.
We make this decision together through this particular variant.
Right?
We make this decision together, right?
And this is what we are doing.
Forget information is also going over here and some kind of operation is basically happening okay.
So here the goal is okay.
The goal is very simple.
I will just go ahead and write my goal.
The goal is very simple.
The goal is just to make sure that we only forget.
We only forget when we are.
When we are going to input something.
In its place.
So what does this sentence mean?
Is that whenever we are inputting something, then only we are going to forget something.
Okay, so if I want to probably give a conclusion over here, I can also write.
We only input new values.
To the state.
When we forget.
something older.
So this is the entire funda.
Okay, uh, I have to, uh, say this is one kind of operation which is basically doing this, this,
this entire functionality.
That's it.
Okay.
And by this, you probably get another variation.
And this is nothing.
But we are trying to combine forget and input Get in such a way that whenever we don't have any information
to remove from the memory cell, then we don't add new information.
When we remove some information, then only we add new information from the input gate.
So I hope you like both this particular variant.
Now there is also one more variant which is called as GRU gated recurrent recurrent unit.
Okay.
And this is also again a different variant of RNN.
Um, why we use this.
We will talk about the advantages and disadvantages.
And here also if we see uh here you can actually see what was the kind of operation that we did.
So when we are calculating okay C of T that is this we take f of t multiplied by.
That is point wise operation.
So this is my point wise operation over here with c t minus one.
So this is fine.
Plus what we do we say hey one minus f of t we are going to multiply with c of t whatever output we
are getting from this particular tanh edge.
Right.
And here you can see c one minus f of t is nothing but this.
We are just subtracting one.
Over here we are multiplying with C.
Uh, we are doing a point.
Uh, we are basically taking the C of T, we are multiplying it over here.
And then we get this entire information.
Right.
So this is nothing.
But this is my point wise operation.
So this basically indicates that through this operation we we add new values.
We add new values only when we forget.
Only when we forget from the forget date.
Right.
That's it, you know.
So I hope.
Are you able to understand this video with respect to variance?
In our next video we will be talking about gated recurrent neural networks or gate gated recurrent unit.
And we'll try to understand what are the advantages and disadvantages.
But I would suggest you please just check the diagram, see the variation from the other diagram that
is like LSTM as a.
And I will definitely see you guys.
LSTM RNN is must to understand anything.
The 5 to 6 videos where I have explained about input output, how the training basically happens, everything
you really need to know.
Okay, so yes, this was it from my side.
I will see you all in the next video.
Thank you.