Hello guys.
So we are going to continue the discussion with respect to the LSTM working of LSTM, RNN neural network.
Now in this video we are going to discuss about the forget gate, uh, which is the first and the foremost
important component in the LSTM RNN.
Okay.
Now yes, we are going to discuss about this.
We have already discussed in the previous video we gave some uh example.
What is the importance of this memory cell.
Then we understood what each and every operation is doing right?
Everything we discussed.
Now let's go ahead and discuss about the forget gate.
Now let's focus on this part okay.
Let's focus on this part.
So on the left hand side of this part.
Now what is exactly happening over here.
Right.
Let's, let's consider that um, I am probably doing, uh, functionality.
Uh, I'm trying to solve a use case which has text, and we need to predict the next word.
Okay.
Let's say right now for every word that I have over here.
Right.
Let's say this is X11.
This is X12.
This is X13X14.
And this is my next word which is nothing but my y y word specifically.
So this will be my Y15 okay.
This will be my output now for every word you know that we have to convert this into vectors, right?
So let's consider that X11 or any word that we are going to use.
We are just going to use it with the help of for 3 or 4 vectors.
Let's say I want to probably represent every word with four vectors.
Let's say it will go like zero, two, four, six.
So this is my first word vector.
The next word vector can be 4512 okay.
I'm just defining okay.
And these are the vectors that I am actually going to give with respect to timestamp okay.
Now let's focus on this operation okay.
Let's focus on this operation.
So guys now let's take this first step.
And uh I will show you like how in terms of neural network it will look like okay, so let's consider
this x of T.
Uh, I've taken as four dimension or let's say that okay, it is the every word.
We are just going to represent it with three dimension okay.
So here is my three dimension.
Every word that I am representing is with respect to three dimension I may use any technique.
And that is what I'm going to pass in the LSTM RNN okay.
Now when we say x of t and HT minus one.
Now when we say my x of t is represented, each and every word is represented by three dimension.
So you will also be seeing that this will be my x of t.
You'll also be seeing my HT minus one, which is the hidden state of the previous neuron.
Right.
This will also be a three dimension okay.
This will also be three dimension.
Why?
I'm saying it will be three dimension.
Because later on when we do this entire calculation right now, I'm just saying that this is the previous
state, right?
I will be getting three dimension.
We have to take three dimensions and you'll be able to understand it.
Why I'm saying this will be three dimensions.
Uh, can't it be four dimension?
Yes, it can be.
See, I will just go ahead and write it over here.
Let's let's consider.
Okay.
My HT minus one is also three dimension.
Let's say one, two, four.
Okay, I can also have four dimension.
I can also have five dimension.
But if I have RT minus one which is the previous hidden state as three dimension, then what you see
over here has a memory cell.
This CT minus one will also be three dimension.
And let's put some values.
Okay.
Uh, in order to clear your confusion, let's consider that I'm going to use my word with four dimension.
Okay.
Four dimension.
Every word I will just go ahead and use with four dimension.
And, uh, let's say if in our previous state my HT minus one is three dimension, then my KT minus
one will also be three dimension.
Why I'm saying this because we'll do the calculation.
Every calculation will be shown to you okay.
Now let's take this xt and HT minus one.
And we are concatenating over here.
What does concatenating basically mean.
So I will I'll, I'll take just as this input.
So this will be my input.
Let's say input words that I'm giving I know first of all I need to give HT minus one.
Right.
So this is my HT minus one.
Now in the case of HT minus one as I said this is three dimension right.
So I will be having three inputs that I will be giving over here.
And then you have the next one that I'm passing XT next.
Since we are concatenating it, I have already told you we have to combine this, right?
So concatenating basically means I'm going to combine HT minus one.
Along with this I'm also going to combine my XT.
Now xt I will just give this as four dimension 1234.
So this is for every word right.
So at t is equal to two.
Let's say I'm passing this information HT minus one basically means at h one the previous hidden state.
I'm getting this values.
So this three dimension is specifically coming from h t minus one.
And these four dimensions are basically coming from my x of uh, my uh current word in this timestamp.
Okay.
When we combine, when we concatenate like this, that basically means we are just combining it, okay.
We are just combining it.
Perfect.
Till here it is.
Fine.
Now this basically becomes my input.
Okay, now in the next layer, once I go ahead, you know there is a neural network.
Now when I say this is a neural network, in short, what I am actually going to do is that I will be
having this kind of hidden layer, okay.
Where neurons will be there, hidden neurons will be there.
Okay.
It can be two hidden neurons.
It can be three hidden neurons.
It is up to you.
Let's for right now, let's consider that I'm just going to use three hidden neurons okay.
So I'm just going to use this.
Three hidden neurons okay.
So in this hidden layer I'm having this three hidden neurons.
And top of this hidden layer what is basically getting applied and activation function is getting applied.
Okay.
An activation function is basically getting applied in every hidden neuron.
And that is what it looks like, right?
Uh, if uh, so I'm just applying an activation function on top of it okay.
And you know in all this neurons also we'll be having bias.
We'll be having bias okay.
So till here it is very much simple.
Now what I will do I will just go ahead combine every node which looks like this.
Okay.
So here I'm just going to combine this.
Okay I'll combine this to let's say this is here.
This is here again.
You can combine each and every layer.
It is up to you okay.
And you can complete this diagram.
So this mostly becomes like a, uh, simple connected uh neurons.
Right.
So this is my input.
This is my hidden layer.
Now once we apply the activation function then what we are going to get, we are going to basically
get f of t f of t.
Okay.
So in short this part that you will be able to see which is called as forget gate.
I have changed into something like this, and this is the entire operation that is probably going to
take place.
Now let's understand how many weights is basically getting assigned.
So here you will be able to see how many, how many inputs.
I am giving one comma seven.
That is one cross seven.
Right.
So this will basically be one cross seven because one row seven inputs.
And then with respect to this since I have three, since I have three, uh, neurons, hidden neurons
in the hidden layer.
So this is basically going to be seven cross three right.
Seven cross three weights.
Right.
This will be my seven cross three weights.
And once I perform this particular operation, if I do this entire dot operation so it will be one cross
seven uh dot seven cross three, which will be nothing.
But here this two will get combined.
So I will finally have one cross three.
So that is the reason this f of t that you will be seeing is that I will be getting a output dimension
of one cross, three right?
One cross.
This is what we are basically getting.
Okay, now you can understand why I basically took CT minus one as three dimension RT minus one as three
dimension.
When this is three dimension, this has to be three dimension.
And just by doing this calculation here, you can see that I'm able to get three dimensions right.
Uh, if this is four dimension, this also has to be four dimension because based on the calculation
it will happen in that way.
Right.
So here my f of t what I have done is that I have able to find out this output vector which will be
of one cross three.
Okay.
Perfect.
Till here.
I think everybody is understood this forget gate operation.
Now going forward, you will be able to see that there is a point wise operation.
So here, if I go ahead and take this forward, there is something a point wise operation.
Now this point wise operation is along with a cell, uh, CT minus one, which is with my memory cell
and CT minus one is the previous, um, previous memory cell information that we are getting with respect
to CT minus one.
Now, once we do this point operation, what does this point operation basically means?
Whatever vector we are getting, we are doing a point wise operation with the previous cell state.
Now what does this mean?
Let's let's consider this from my previous cell state.
I get this information CT minus one.
CT minus one.
And let's say I have some values like um, 689.
Okay.
Now when we do this point wise operation, point wise multiplication operation.
Okay.
And let's say my f of t that I have actually calculated is something like 000.
Let's consider like this okay.
So what we are doing, we are basically doing a point wise operation with 000 okay.
Now what does 000 basically mean?
Zero zero basically means that whatever operation we are doing over here till finding this f of t,
I'm saying that now let's see that the complete sentence context has been changed.
Now, when the complete sentence context has been changed and let's say I'm getting the value 000,
this is the value that we are getting.
What does this basically mean once we do the dot operation here, the previous context, whatever was
present, we are just making it entirely to zero.
That basically means we are removing all the previous context.
Now here you know that when we apply a sigmoid activation function, it will give you a output between
0 to 1.
Right.
It is going to give you an output between 0 to 1.
Right.
And let's say I got all the outputs to zero.
And I if I'm getting the outputs to zero, that basically means whatever operation we have done.
Right.
And with respect to the new sentence, the complete context has been changed.
So I have to remove all the context that is already remembered, uh, remaining in the previous cell
state.
Okay.
Now similarly, let's go ahead and consider that if my F of t is one, let's say if my f of t is completely
one.
Now if I do the dot operation.
So this is my first scenario.
Now this will be my second scenario.
Let's say my CT minus one is nothing but 689.
nine.
Now, if I do this point wise operation with one one, one, what I'm actually going to get, I'm going
to get the same vector 689.
So this actually means that I am not going to remove anything from here from the CT minus one memory
cell.
I'm not going to remove anything because all the context is important for me.
Okay.
Now similarly I may also give you another scenario.
Let's say if I go ahead and say my CT minus one is 689, now I'm going to do a dot operation with,
uh, the dot operation will be with uh, let's say 0.51.5.
Now when I do this dot operation, what does this basically mean?
So six divided by 0.5 is uh, one by two is 0.38 divided by one will be eight, nine divided by 0.5
will be 4.5.
Right?
So 3.5 will be 4.5.
Now here it says that from the memory cell I am removing some context from this vector and some context
from this vector.
But I am not removing any context from this vector.
Right.
So in short, what we are doing by this operation is that we are adding some information or removing
some, some information.
Sorry, I'm not adding some information, but I am removing some information out of it.
Right.
I'm not adding over here, but I'm removing.
Right.
Removing some kind of information.
So that is the reason this is entirely called as forget.
Right.
So the conclusion with respect to forget get is that.
Here based on the context.
Based on the context this forget gate.
Will let go some information.
Or will not let go.
Not let go some information.
And that is what is all about forgetting information.
It is making to forget some of the information based on the context, right?
And this is how you basically understand about forget gate.
Right.
So I hope you were able to understand this entire LSTM architecture with respect to forget gate.
I broke this down completely and I also helped you to understand this.
What how does this neural network look like?
Okay.
But with the help of forget gate, what we are doing is that we may remove the complete, uh, we may,
uh, tell the memory cell to forget some of the information or to remember some of the information or
forget some information.
Something can be actually done.
And this was the importance with respect to the, uh, forget gate.
Uh, now, in my next video, we are going to basically go ahead and discuss about something called
as input gate.
Right now, what exactly is this input gate?
We'll get to know.
But here I think we discussed till here.
And this multiplication operation is just like you are forgetting some information you are allowing
based on the context.
You are telling the memory cell to forget some of the information or let go some of the information
by remembering it.
Okay, so yes, this was it for my side.
I will see you all in the next video where I will be discussing about input gate.
Thank you.