Hello guys.
So we are going to continue the discussion with respect to LSTM RNN.
Now in this video we are going to discuss about output gate.
Now output gate is the third important module in this LSTM RNN, specifically in the architecture.
And if I probably see with respect to this diagram, this is the part that is related to output gate
over here.
So we are going to discuss about the working of this okay.
Already uh we discussed about forward forget gate and forget gate.
Our main intention was to uh forget some information.
Information based on context from the memory cell.
So all the operation that we are actually doing, it is actually leading to that specific thing.
Now, uh, in the second one, which was specifically in my input gate, our main aim was aim was to
add information based on the context.
Okay.
Add information based on the context.
Now let's go ahead and discuss about the third output gate.
That is the third component over here.
Now there are two things.
One is again the same operation which we have with respect to sigmoid.
Here we are passing x of t dt minus one and we are adding a bias.
So here we specifically we will be using b0 as our bias.
And we finally do the sigmoid activation function.
So if I consider this this is nothing but it is my neural network right?
It is the hidden neural network.
Perfect.
Now after doing this specific operation, whatever information that I have, you know, after I forget
some of the information and then add some information from the input gate through this candidate memory.
From that we will be passing one information that is called a CT.
Now this CT that you'll be able to see.
This is my memory cell, right?
This is the information in my memory cell.
Now this will continue on right.
And along with this you should understand one thing right.
So see let me just go ahead and discuss this.
So this is my CT and this is my PT okay.
Whenever I talk about CT this is my long term memory.
Long term memory.
Now in the output gate what we do we again do this tan h operation or tan h uh operation with respect
to the output that we get from here.
And we apply on each and every element like point wise, point wise operation is specifically done to
this sigmoid output, right?
That output that I have with respect to O of T by doing this, what will happen is that whatever output
is probably going to come from here, this will be retained in my h of T.
H of t is nothing but my hidden state.
And whenever I talk about hidden state, I'm actually talking about my short term memory.
Right.
Short term memory.
And this short term memory will be related to the just the current context or one context before it.
Right.
So this operation is specifically done to get the output one.
Once we get this particular output we will be sending it, uh, you know, we'll be sending this h of
t which denotes our short term memory, to my next timestamp.
Right.
And similarly, with respect to the operation that we are doing here, the point operation, uh, point
wise operation from CT minus one, I will be getting CT, right.
So this will basically be my long term memory.
And this will be my short term memory which will be passed to my next layer right.
Next timestamp.
And if I say this, uh, this line also probably I can just go ahead and connect this over here.
Right.
This is the h t itself, right.
I can use this also.
So uh, that is the entire operation with respect to output.
Now if I go ahead and see over here, here you can see I'm using sigmoid of w of o w of o is the weight
that is assigned over here.
Then I have h t minus 1XT plus b zero.
And then I'm performing the same sigmoid function.
I am getting the odd value.
Then when I do point wise operation with tanh over here and c of t is nothing but uh, C of T is this
specific value that I have.
Then through this I will be able to get the context that is available in the short term memory.
And then we continue this process.
Okay.
Now you can just imagine like this, right.
Uh, let's say this is my LSTM.
This is my LSTM in T minus one.
Then in t t is equal to two.
Sorry I'll say one and t is equal to two.
I will be having another LSTM and t is equal to three.
I will be basically providing this particular loop right.
So this will keep on going on.
Right.
And finally you'll be able to see that once I get the output, I may pass this to my sigmoid function
based on the kind of output that I really want Y hat.
Right.
And this operation will keep on going with two cells.
As I said, one is the long term memory cell and one is the short term memory cell.
So here this is my long term cell.
And this is my short term.
Right.
Memory cell.
And with respect to the information what I really need to forget I will be removing some of the information
from here.
I will be adding some of the information from here.
Again I may see both these things may happen over here right from one neural network.
You know, from sorry, from one hidden layer.
I may add some information.
I may remove some of the information.
Right.
Adding the information basically is done by, uh, the input and memory cell.
And removing some of the information will be done by forget cell right.
Forget gate I'll not say cell but forget gate.
And adding some information will be specifically done by our input gate along with candidate memory.
Right?
So that is how the entire process will be going on the forward.
And the backward propagation will keep on happening unless and until our loss keeps on getting reduced.
Right.
And that is how we make sure that we update all the weights.
Now in this particular case, what are all the weights that we saw.
One was w zero.
Then uh here also uh, we saw some more weights w of I and then you had w of C, right.
So these three weights were specifically there.
Right.
So all these three weights w of I, w of C and w of.
Oh.
So let me just go ahead and do this and write down all the weights that we are updating w of I w of
c, w of 00I is with respect to input, C is with respect to the memory cell and O that you will be
able to see is with respect to the output, which I'm actually getting back my hidden state over here.
Right.
Sorry, not hidden state, but here when I'm putting it last on my.
So this this neural network that we are using over here.
Right.
This will specifically be my W of zero.
So we need to keep on updating this weight with the help of back propagation, back propagation.
But at the end of the day, you will be able to see the memory cell is playing a very important role
because it is keeping some of the information for a longer context.
Uh, whichever needs to be remembered, it will be remembering.
Right?
And uh, let's say if the context is switching and we really want to remove that, the entire training
will be happening in this specific way, right?
So, uh, I hope you are able to understand about the output gate LSTM rnn.
Now, in my next video, I will be talking about one more variant of RNN which is called as GRU RNN.
Okay, so this is just another variant of LSTM, LSTM another variant.
And here uh we will be understanding some important things with respect to this.
What all things will be changing right.
This operations will be changing with respect to the GRU RNA.
So yes, this was it for my side.
I hope you liked this particular video.
I will see you all in the next video.
And, uh, I hope, uh, you're able to understand, but just to give a summary, you have this forward
gate.
Forward gate is, uh, responsible in removing some of the information from this particular memory cell
input and candidate memory is responsible in adding some of the information.
And finally, output gate is used to distinguish properly all the memory cell and the uh, hidden state.
Hidden state which represents the short term cell okay, short term memory.
Or I can also say short term memory.
Not right short term cell, but instead I can basically go ahead and write short term memory.
So I hope are you able to understand it?
I will see you in the next video.
Thank you.