Hello guys.
So we are going to continue the discussion with respect to LSTM, RNN.
And in this video we are going to discuss about input gate and candidate memory.
Now uh, one thing that I missed out in the previous video is talking about this entire operation.
Right?
When we are calculating f of t, what does this equation basically mean?
Right?
I have explained you each and every thing, uh, with respect to working, but let's go ahead and write
this equation.
So first of all, you'll be able to see that, uh, I am I'm first of all, combining this concatenating
x of t and h t uh, minus one.
Right.
So this way we can go ahead and write like how it is written over here h t minus one comma x of t.
That basically means we are just combining all the inputs.
Right.
Like how we combined over here.
Then once we combine this, then you know that we are just going to have some kind of weights.
So weights included over here is f of f w of f that is actually given over here.
So we are just going to multiply with w of f as my weights okay.
And along with this you'll also be seeing that in my neurons right.
I will also be adding a bias neural network.
So here you will be having plus B of F right.
And then finally you will be able to see that I'm applying a sigmoid activation function.
Right.
So this is how the entire operation has basically taken place.
And this is how the notation looks like.
Okay.
I hope uh, pretty much clear with respect to the explanation I have done, I think it should be very
much easy to understand.
Now let's go ahead and make you understand about the input gate and candidate memory.
Right now, the input gate from this particular diagram is nothing but this specific gate where we are
combining x of t hidden state t minus one, and we are passing it to this particular neural network.
Okay.
Now here when we are passing to this particular neural network, it is same thing like how we have actually
done over here, right.
How we have actually done over here.
And instead of calculating f t, we are specifically getting um, let's, let's consider over here as
I of t.
Okay.
So this is nothing but I of t okay.
So we know how I of t is basically getting calculated.
But let's talk about this tan h right.
Now in this case of tarnish, what will specifically happen.
This I of t let's say that I am actually getting some vectors.
Okay I of t let's say I'm getting a three vectors like two four, six.
Okay.
Now with respect to this I of t over here you can see the operation is almost same like how we did it
for the forget gate.
The similar operation will be over here where I will be having all my inputs.
Let's say three, four, five, six, seven.
So this is my HT of HT minus one.
And this is specifically my x of T okay.
Then the next layer I will just go ahead and pass it to my hidden layer which will have this three neurons
okay.
And here we are also going to add.
So there will be one bias that will be added over here.
Now apart from that we apply a sigmoid activation function on top of it.
If I want to just go ahead and apply it.
So here I'm going to apply a sigmoid activation function.
And finally I get my output which is nothing but I of t okay I of t.
Now let's go ahead and do the same thing for this candidate memory also.
Now in candidate memory what we do is that again the same way.
First of all, we will take our HT minus one.
Let's say it is a three dimension.
Then I have XT minus one.
Let's say this is sorry XT.
It is four dimension.
We take this entire thing and here this time we are using a hidden layer inside this hidden layer.
Let's consider that I'm using three hidden neuron.
After this I pass to an activation function which is called as tan h okay.
And here we add a bias okay.
In short we are just combining each and everything.
So this will basically get combined to this this this.
And you can further draw this diagram I'll leave it up to you okay.
Let's say I am just drawing some of the diagram so that you face this thing right now here also, and
just connect all the other lines, okay.
Here also, you'll be able to see that I'm passing one cross seven.
This will be uh, one cross seven.
And this will also be something like uh seven cross three.
So the final one that I'm actually getting is one cross three after I apply a tan H.
So here also I am basically going to get my C of T, which will again be a three dimensional vector
because this is what I'm actually getting one cross three.
And let's say that I am getting some values like 010.
So again three dimension vectors.
Now you need to understand whenever we take this input gate and along with this candidate memory we
have to combine this together.
Right.
So whatever output I am getting from it uh it and CT, we are performing an another operation which
is called as pointwise multiplication operation.
Okay.
We are performing this pointwise multiplication operation which is basically mentioned over here.
Right.
So this is my pointwise multiplication operation.
Now with respect to the pointwise multiplication operation.
Why do we say this as input gate.
This is called as an input gate because from here based on the context again.
So here I'm just going to write it down based on the context, if any new information needed to be added.
Vision needed to be added to be added in the memory cell.
In the memory cell c t minus one.
Then after this point operation, the information will get added.
The information will be added.
So in short, forget gate.
If you remember, forget gate the first gate when we are trying to do the multiplication operation over
here.
Pointwise multiplication.
We are forgetting some information.
And from this specific gate we are.
We are adding information.
And that is where I get my final cell state that is called as C of t.
This is what we get as a final state C of T.
Okay.
So this is uh, more about a basic information of, uh, input and candidate memory.
You know, uh, again, uh, to explain you, it's very much simple.
When I do the point wise operation, you know, so here I'm saying, hey, there is some new information
that I really want to probably add it up.
I can add it up over here with along with this particular information in my forget gate.
I know just by using this forget gate, I can remove some of the information and that is the entire
context of using input gate and candidate memory gate.
Right?
So, uh, I hope you are able to understand this thing and you're able to make sense.
Okay.
So let's talk about these two operations.
So first of all, uh, you can see with respect to uh x of HT minus one and with respect to sigmoid.
So we are using this one right.
This is basically calculating I of t.
Then you have uh we are combining see for this we are using another weight w of I.
And for this I'm using another weight w of CT okay w of c.
So for the tanh operation I'm specifically using WFC.
So here you can see WFC with HT minus one.
With x we are combining it.
And then for that we are using another bias.
So this is also another neural network.
This is also other neuron.
So that is the reason I created this.
Write this here.
My weights that I have will be w of I here.
The weights that I have is nothing but WFC and I'm combining them together.
And finally I'm getting my c uh c of t okay.
But once we do this dot operation of both of them, uh, it's just like the, uh, the input and the
candidate memory that I'm specifically having with this operation.
Any new information that I want to add.
Right.
You need to understand with respect to the candidate memory, what important thing it is doing.
Any new information based on the context that needed to be added.
We will just try to do a point wise operation over here.
Right?
So let's say if CD has this important context in over here.
Right.
So I may probably put some new information.
I may go ahead and write 020.
And when we do this dot operation here, you can actually see that this will become zero, this will
become eight and this will become zero.
So new information is basically getting added, uh, in my CT minus one cell state along with the previous
state, and we are adding it up.
This is also very much important.
We are adding it up.
Okay.
So let's do one thing.
Let's discuss about both this.
The combination of forget gate and input and uh, candidate memory.
So here we saw that uh, with respect to this particular information that I had right over here, I
had.
So first of all, if I want to get to this final memory cell state C of T, first of all, we will multiply
f of t multiplied by c t minus one.
So here we are removing.
Or forgetting some information if it is, if the context is switching.
Otherwise we'll just make the cell state pass in that way.
Along with that, when we multiply I of t and c of T that we are doing a point wise operation.
So this c of T is nothing, but it is a it is a what kind of cell it is.
Basically it is a candidate memory.
It is nothing, but it is a candidate memory.
And the task of the candidate memory is to add new information.
Right?
Add new information to this particular cell state.
And here you can see point wise operation.
We are just adding it up.
And finally we get a new cell state that is C of T.
Okay.
So I hope you got an idea with respect to the input gate and candidate memory gate.
So if I take a very good example, let's say I go ahead and write uh, I stay in India.
Okay.
Some sentence and I speak dash.
Okay.
So if this kind of sentence, I want to, uh, probably, uh, find out what is the next word.
So all the words will be going line by line.
Right.
But here you can see we will be saving this Indian context in the memory cell Right?
When the context switches, we may be forgetting it.
We may forget some other information, but for a longer term, we will be keeping this particular information
over there to predict my next language.
And obviously.
But just by seeing this context, my neural network will be able to predict, okay, I do speak Hindi
or English or anything as such, but right now this particular prediction is based on my this context.
That is which country I'm actually staying.
Okay.
So I hope you are able to understand it.
I hope you are able to understand this entire equation.
I hope you are able to understand the entire working right to simply give you.
Till now we have discussed about forget gate and we have discussed about input gate, input gate plus
candidate memory.
Candidate memory when we do a point wise operation over here.
Candidate memory.
Whence we do the point wise operation.
And then we do the addition operation with CT minus one to finally get C of D, which is my final cell
state.
Memory cell state.
So yes, this was it for my side.
In this video we have discussed about input and candidate memory along with the explanation along with
mathematical intuition.
This was it for my side.
I will see you in the next video.
Thank you.