Hello guys.
So in this video we are going to discuss about gated recurrent unit which is again a variant of LSTM
RNN.
This uh gru rnn was introduced by Koi et al at 2014.
Okay, now just imagine in 1980s LSTM RNN was there.
Then we saw some variants in 2000, right?
Uh, now in 2014 you will be able to see that, hey, we have this green variant.
Now, since this variant, this this green has just come recently in 2014, hardly ten years back.
Right now it is 2024, right?
It has some very important meaning.
And we'll be discussing why do we use this green.
Okay.
So let me just go ahead and explain some of the things.
So see in LSTM RNN you know that we have a separate cell or memory cell and we have a separate.
So this is for my long term memory.
Long term memory which we or let me just go ahead and draw it below so that you will have sufficient
space to understand this right.
So let us say this is my long term memory.
This is my short term memory, okay.
And I have already shown you we have this forget gate.
We have this input gate.
We have this output gate.
Right.
So I'll just give some notation over here.
This is CT minus one.
This is C of t.
This is HT minus one.
This is h of t.
Then we do some point wise operation.
This will be plus this will be plus over here.
And uh you know let me just go ahead and check it because, uh, it's been see, this was this.
Right?
Then we use the tan H, and then we specifically do this.
I could use this particular image.
Okay.
So here, uh, what I will do, I'll just go ahead and use this tan h and then a point wise operation.
So here you have all the information going right.
Right.
Now understand one thing.
This line is for my long term memory, long term memory.
And this is for my short term memory, because this is my hidden state that is basically mentioned over
here.
So this is my short term memory.
Now, you know, for every gate.
So as I said this is my forget gate.
Right.
So this is my forget gate.
This is my input gate and this is my output gate.
Now, for every of this gate you know that we have some weights.
So here I will go ahead and write w of f here w of I.
Then here you have w of o.
Okay.
So at the end of the day with respect to this particular gates, this LSTM RNN variant, you will be
seeing that one of the major problem is that this LSTM RNN architecture is quite complex, right?
Why?
It is quite complex because here we are using three gates.
First of all right.
Here you'll be able to see that we'll be using this forget gate.
We'll be using this input gate.
And obviously along with this we also have this candidate memory.
And we have this output gate.
Now, when we have this many number of gates, you know that we also have three important weights.
One is w of f, one is w of H and one is w of oh this many number of weights.
Right.
Sorry I'll just go ahead and write this weights again.
So I have w of f w of I and I have w of.
Oh o right.
And even though if I consider that in my input, I have 100 dimension in the hidden layer, I have uh,
100 hidden nodes.
Right.
So these weights will keep on getting increasing.
So whenever this weights and obviously bias will also be there.
Now what do we consider this weights and bias.
This weights and bias are nothing.
But these are called as trainable parameters.
Trainable parameters.
Right now, because of this complex architecture, you will be seeing that this trainable parameters
will keep on increasing.
Right.
Because that many number of parameters is specifically required.
And whenever trainable parameter increases, uh, obviously the training time will also be increasing.
And because and this is basically happening just because of the complex architecture, we have so many
number of gates and we are we are having so many number of weights.
So we are having so many number of bias and forward and backward propagation when we keep on doing with
respect to time.
Right.
So see LSTM, RNN does a good job in solving long term dependency.
That is very much clear.
But still the architecture is very complex.
And because of this you will be having more trainable parameters when you have more trainable parameters,
you have more training time.
Right now, what researchers do.
Uh, researcher Co et al did in 2014.
I hope I'm pronouncing it right, guys.
Uh, if I'm not pronouncing it right, please apologize.
Right.
I, uh, sorry for that.
Okay.
So what in GR you specifically done is that instead of using two memory like long term memory and short
term memory here, what it has been specifically done is that, hey, we will see again for handling
this long term and short term memory.
There will be different different parameters that will be assigned.
More operations will be specifically done.
You know what they have actually done in GRU.
They have combined this two memory cell.
See this is just one memory cell which is representing hidden state.
And here there is no long term memory cell.
Right.
This cell, this cell that you will be seeing, it acts as both long term and short term.
It acts as both okay.
And from this only we pass this information.
And this is my input gate over here.
This is my.
You can you can see over here, right.
Uh, this I'll not say input gate.
So see this is basically called as.
And here you also need to understand with respect to the gates okay.
So Z of T over here I will denote it over here.
Like how we had this forget gate input gate and output gate here.
Uh, when I consider this architecture here, you'll be seeing that we first of all find out Z of t.
So z of t is nothing.
But we take this t minus one.
We take this x of t, and then we pass it to the sigmoid activation function.
And here we go ahead and apply this w of z weights.
Right.
So when z of t is basically calculated this z of t is nothing.
But it is basically called as update gate.
Okay, let me just go ahead and write this information.
This is called as update gate.
Update gate.
Okay.
Now when we go to the next one here you can see that it is r of t.
How do we calculate r of t.
We take s x of t dt minus one.
And then here we are assigning and another weight another weight over here.
And this weight will be nothing.
But it will be uh w of r right now this value that we are computing r of T, this is basically called
as reset gate.
Very much important.
This is basically called as reset gate okay.
Now reset gate.
What is the functionality?
I'll give you a brief functionality and information.
What this specifically does we will be getting to know all about that okay.
And then uh the final one that you have is nothing but h of a temporary hidden state.
I can basically say this is a temporary hidden state.
I'll just name it.
Okay?
Temporary.
Hidden state.
So let's do one thing.
First of all, let's understand how this operation of R of T is there, Z of T is there and h of T is
there h h mode of t?
Is there h hash.
Right.
I'll say h this upper symbol.
Okay I'll give this kind of notation over here which I'm writing it as temporary hidden state.
So first of all, uh, everybody is clear with Z of T, because here it is very much simple.
We are combining x t minus one.
We are combining x of T.
We are passing it to this, uh, sigmoid activation function with uh sorry.
Over here.
This one.
Right.
We are passing it to the sigmoid activation function with z okay z over here z uh uh z of t.
We are basically getting the output here.
We are going to apply a weight which is called as w of z.
So let me just go ahead and write w offset okay.
So this is one.
Now let's say r of t r of t which is nothing but reset gate.
This is nothing but update gate right now.
When we go to the reset gate and reset gate, what we are doing, we are taking this W of R, whatever
is the weights over here and we are combining HT minus one and x of t.
So this is also very much simple.
And then we are passing it to the sigmoid activation function.
And finally I get my r of t.
Now when I get my r of t you will be seeing that I am doing a point wise operation over here.
So point wise operation with what?
With T minus one.
Right.
Once I do this point wise operation with T minus one along with x of T, I am passing this entirely
to my tanh activation function, and with the help of an activation function, this is where I'm getting
my temporary.
Hidden state.
Okay.
And then what we do by doing another point operation with z of T, right.
So z of t is also there.
And by doing another point operation with z of t I'm updating my final h of T, which is my memory cell
for both long term and short term memory.
So this is the entire operations that is specifically done right.
Understand the dependency right one or the other dependency x of t z of t is used over here along with
h of t right.
So here we are doing dot operation.
So this is my dot operation that we are doing.
And then in order to calculate a h of t I'm also subtracting some information one minus z of t one minus
z of t.
And I'm doing dot operation with h t minus one.
So that is what you are able to see over here.
Okay.
Step by step again let me repeat it and we'll try to make it much more clearer.
Once I make you understand why this entire steps is done.
So first of all you need to understand Z of T is nothing, but it is called as update gate.
R of T is nothing, but it is called as reset gate.
H mod of T is nothing but temporary hidden state.
So first of all, we go ahead and compute z of t.
How to calculate z of t.
It is nothing, but we take this t minus one.
We combine x of t, then we pass it to a sigmoid activation function.
Before passing it to the sigmoid activation function, we multiply with weights w of z and finally I
get z of t.
Now this value so z of t will be used where I'll talk about it okay.
So first of all we have computed z of t okay.
Now the next one we are we are going to compute r of t r of t is very small simple.
In case of r of t we take t minus one x of t we apply weights.
Multiply of weights w of t because this is our neural network right.
Which you have already discussed.
Then we pass it to the sigmoid activation.
We get r of T.
Okay.
Then what we do we in order to compute this h dash uh t right.
We take this h t minus one, we take this r of t we do point wise operation.
So that is where this point wise operation is done.
Then we take x of t.
We combine both of this input.
Then we multiply by what we multiply by weights which weights.
So that will be w weights over here.
Then we pass it to the tanh h.
Then I get h dash of t which is my temporary hidden state.
Now what I do, I take the output from my update gate.
I take the output from this temporary hidden state.
I do a point wise operation and then I send it over here.
And then we do a point wise plus operation with HT minus one and get the memory back before that.
Also, from the update gate, I subtract some of the information from one minus z of t.
So from this particular information, and then do a point wise operation and send it to this particular
information.
Right.
So finally in order to get h of t I do this two important operation right now let's go ahead and understand
what does update gate, reset gate and temporary hidden state actually do.
Okay.
So let's go ahead.
So guys now let's go ahead and discuss more information about this update gate reset gate and temporary
hidden state which we also see it as candidate hidden state.
All the notation is specifically given over here okay.
Now what is the importance of this reset gate?
Okay.
Here I'm going to basically denote by r of T which is basically given over here r of T.
And you know r of t how it is calculated sigmoid of w r.
And we are combining ht minus one and x of t.
And we are passing it to the sigmoid activation function.
And then we specifically get r of t Now r of t.
We basically say it as a reset gate.
And its main responsibility is let's say for my HT minus one line.
Right.
Which is my current memory cell okay.
I want to see this R of T as I said.
Right.
Let's say I will just go ahead and right over here.
Let's say I'm getting the R of T after this sigmoid.
Right.
And for this I'm passing my h t minus one and x xt.
Right.
Now once I give this specific output, once I get this particular output from this sigmoid right, you
will be seeing that this reset gate will be responsible in resetting and resetting some information.
Resetting some information from h t minus one, which is nothing but our memory cell.
And when I say memory cell, this is the combination of long term memory and short term memory.
Okay, so this is what R of T will specifically do.
So if you go ahead and see over here how we are calculating R of T, it is nothing but sigmoid of w
of r with uh w of r multiplied by this h t minus one comma x of t.
Right.
We are combining both these things.
Now the main aim of this reset get why I'm saying you once we do this calculation here, you can see
that I am taking this T minus one and we are combining or we are doing a um point wise operation, point
wise multiplication operation.
See, whenever you see point wise multiplication operation, always consider this as resetting some
information or forgetting some information.
Now what does this resetting some information basically mean?
Let's say from h t minus one.
If I just go ahead and consider some four dimension vectors.
Right.
So let's consider that this is my h t minus one.
And here I have values like .6.5.3 and point nine okay.
Now I will go ahead and calculate my r of t.
Now r of t may be some values like this.
It can be point of two.
Let's say that we are just going to calculate it .4.8 and uh point two okay.
Now this indicates when we do this when we do this point wise operation.
In short we are just going to multiply this.
Right.
We are going to multiply this when we are going to multiply it.
Understand one very important thing over here When we multiply these two values, what we are specifically
saying that, hey, there is some 0.6 value and we are just multiplying with the 20% of this value.
Yes.
Right.
So once we probably multiply it I will be getting something like this 0.12 okay.
So here you can see that from this point six I'm just taking 20% I'm resetting this particular value.
I'm I'm forgetting some information through this reset way.
Right.
Similarly over here, if I say 0.5 multiplied by 0.4.
So here you will be getting 0.20.
What does this basically indicate from this 50%?
I'm taking only 40%.
I'm resetting by 40%.
Right.
So when I'm resetting by 40% I'm only getting 0.20 over here.
I'm saying, hey go ahead and reset this point three by 80%.
So this will be nothing, but it will be, uh, .24.
Right.
So something like this.
So you can just consider that I am trying to reset some value out of it.
Okay.
And this is how I'm actually able to do it.
You know, now when I'm resetting my 80%, I'm getting this specific value.
If I'm resetting by 40%.
So 40% of 0.5 is nothing but 0.20 when I'm resetting this point to that, 20% of 0.6 is nothing but
0.12.
Similarly, what is 20% of 0.9?
You know, so you can go ahead and do the multiplication.
It is 0.18.
So here what I'm saying I'm basically resetting the value.
I'm removing some of the context.
And this resetting is happening based on the context.
I'm saying hey I got my new x of T, let's say.
And based on this x of T I need to reset all this particular value.
And this is what this point wise operation will specifically do.
Okay.
Now once we reset it, you'll be seeing that we are sending this information to our tan H.
Right.
So tan H over here it is sending and we are calculating h h t which is nothing, but it is our temporary
hidden state.
Or we can also say it as candidate hidden state.
Right.
So that is the reason you will be able to see that when we pass this information.
In short, we are just passing the resetted value right after we probably perform the Resetted value.
We are passing this to this particular Tanitch.
Okay.
And we will be getting this edge bar of T okay.
Once I pass this okay.
Now let's go ahead and see.
I think this is very much clear for everyone I guess.
Uh, so the first operation where we calculated R of t.
Now let's go and see with respect to z of T how do we.
Z of T is calculated.
So we'll take this t minus one.
We'll take this x of t.
We'll pass it to sigmoid activation function.
And here another weight is applied w of z.
Okay.
So here we have calculated this.
Now this z of t.
And whatever h bar of t will be coming up.
Right.
This both are combined together to get this value.
So guys now let's go ahead and talk about this gate which is called as update gate.
Right.
So update gate is nothing but this gate.
And here you have basically temporary hidden hidden state okay.
Now see guys, uh, it is very much simple.
First of all you need to understand from h t minus one I am able to get h t hat.
And then from this we are getting h t okay.
If I combine combine x of T, right?
If I combine both of them right.
Let's let's go ahead and combine this.
Both of them.
And uh, if I, if I combine this and if I pass it to a sigmoid activation function with some weights,
I will be getting, first of all, which gate over here I'll be getting r of T right.
R of T is my reset gate.
And with respect to this reset gate, uh, further, you'll be seeing that once I get this r of T,
right.
Once I get this r of t, I'm doing a point wise operation.
So here you can see I'm doing a point wise operation again with this.
I'm passing this entire information.
Passing this entire information to my tanitch to get this value.
Okay.
But here you should also remember here I will further pass this x of t also.
Okay x of t over here.
So once I do this operation that is what it is happening right x t minus one with this operation.
And here you can see x of t is passing and we are passing to the tanh neural network.
So neural hidden neural network where the activation function is tan h okay.
And here I'm actually able to get h of t.
So we basically have to do all this particular information.
Now this is very important from h t minus one to h t.
We are making a transition.
And this gate uh that we will specifically say uh I'll talk about that also.
Now from h of t to come to h t this is the most important step.
Okay.
And for this step what we do is that we combine both of this input.
Then we pass it to the sigmoid activation function.
And an sigmoid activation function basically means here I have a hidden layer okay which will have a
sigmoid activation function.
Then I get the output over here This output is nothing but z of t, right.
So here I'm just going to go ahead and write z of t okay.
Now this z of t along with the output of h t.
We will do this point wise operation.
Now this point wise operation that we are specifically doing over here.
Right.
It is between an update gate and a temporary hidden state.
My final state is H of T, right?
Update gate basically says that what context information needs to be added needs to be added, but this
will be completely dependent on my candidate.
Hidden state.
Okay.
And candidate hidden state basically talks about the current context.
I will say, hey, if the current context is important then I will add this information with less information
from with less information from Z of T.
So I will take less info from this and I'll add more info from the correct current candidate.
Hidden state.
Okay.
And that is the reason you will be able to see over here my H of T, which is my final h of T, which
I'm adding it up.
It is nothing but one minus z of t multiplied by one minus z of TC1 minus z of t basically means what?
First of all, I'm giving this input on the top memory cell.
Right.
So here you will be able to see that I am coupling for forget gate with this temporary hidden state.
That is what we saw in this variant, right?
If you remember coupling forget and input gates here.
How how we coupled are two gates.
Similarly, here we are coupling this particular this particular update gate with the temporary hidden
state.
And the operation is same thing.
Right.
So what I'm actually doing over here, one minus f of TC1 minus f of t is further getting multiplied
by c of t okay.
And sorry c of t over here.
And uh this f of t is basically getting, uh, multiplied like a pointwise operation with c t minus
one.
Similarly, over here, one minus z of T will basically get done, uh, as a point wise operation with
t minus one plus the remaining information.
Based on this context, whatever context needs to be added right is getting multiplied by h of t, right?
So here sorry h bar of t.
And finally both this value will get added over here When this both the value gets added.
I'm getting h of t now.
What does this basically mean?
This basically means very much simple right?
So if I'll just take this equation over here one minus z of t multiplied by h t minus one.
That basically means from this particular information, from this particular information, from this
particular update gate, what information we should include that is completely decided based on this
temporary hidden state.
So in order to understand just this equation here, this equation basically indicates one minus z of
t multiplied by multiplied.
Basically means point wise operation with ht minus one plus z of t h dash of t.
So this basically includes the information of the current context.
Let's say the current sentence that we are getting is very important.
I want to include this.
How much I want to include it.
It will be decided by this value.
And the more context that I include, the lesser context I should be including from here, the lesser
context from here.
If I include the more context I will be including from here.
So this is this works in a much more coupled way.
This is what I really want to talk about.
Coupled way.
So in short to update from t minus one to h t this memory cell whenever we need to update this is exactly
the, uh, you know, this is exactly decided with the combination of update gate.
And your candidate memory gate.
Candidate memory gate.
Okay.
It is decided from both of them.
This indicates h bar of T.
This indicates.
Let me talk about the symbol that is nothing but z of T, Z of t, and one more gate that we specifically
had was nothing but our reset gate.
This basically indicates how much information we need to reset from this memory cell.
Right.
So this is the entire context behind GRU.
And already I've spoken about how the training usually occurs in LSTM.
And similarly the training will also occur based on all these equations.
And here there are less number of weights when compared to LSTM RNN.
So this architecture uh over here, the training also with terms of optimization is better when compared
to the LSTM RNA.
So yes, this was it for my side.
I hope you liked this particular video.
I will see you all in the next video.
Thank you.