Hello guys.
So we are going to continue the discussion with respect to NLP in deep learning.
And in this video we are going to see about a new variant of RNN which is called as LSTM.
LSTM is nothing but long short term memory RNN.
Okay, now already in our previous video we have discussed about RNN and we also understood that RNN
is not able to solve long term dependencies, right?
And that is where we got to know about something called as vanishing gradient problem.
Right.
We got to know about this.
Now in vanishing gradient problem, whenever we had a long chain rule at that point of time, what was
happening?
Whenever we are trying to find out the derivative of one output with respect to the other output, at
that point of time I was getting a smaller value.
So whenever we had a very bigger chain rule with respect to time, the value was approximately equal
to zero.
Okay.
So that all things we discussed in our previous video.
Now again I will go ahead and revise some important questions.
So first of all, I will say again I will talk about the problems of RNN in this video.
Okay.
Um, then we will discuss about why LSTM RNN instead of directly using RNN.
So this is the second topic that I will discuss about.
Third thing we will just how LSTM RNN is solving the problem right.
So here two important topics we will be talking about.
One is what is this long long term memory.
And then we will also be talking about the short term memory right.
Short term memory along with this.
Then in the upcoming videos we will be seeing the LSTM architecture.
Architecture.
And finally we'll see the working of LSTM okay.
Fifth will be working of LSTM.
With examples.
Now everything will be discussing.
But before I go ahead I really want to quickly give an amazing shout out to this amazing blog which
is called as Koalas Koala's blog.
Again, this blog if you probably go ahead and see who's the author over here, is an amazing author,
right?
Uh, his name is Jay.
Okay.
And definitely do check out this particular blog again.
Uh, because from here I will be taking most of the diagrams, because here all the diagrams are given
much more in an amazing way to make you understand right about LSTM Again, a quick shout out to this
amazing blog.
The credit goes to this.
I'm giving the proper credit.
Uh, and I will be referring to this particular blog.
So, uh, let's go ahead and let's start our session with respect to LSTM RNN.
So guys now let's go ahead and discuss about the basic problems with RNN.
And already I have discussed this in our previous video.
but here I will try to give you some more examples.
Okay.
So let's say that I am planning to predict, uh, the next word in a sentence.
Okay.
Next word in a sentence.
So this is my task that I really want to do.
Okay.
Now with respect to this particular task, Let's say one of the example that I have actually taken,
let's say my sentence is something like the color of the sky is dash.
Okay.
So I have to predict this particular word.
Now you know that in order to predict this word, let's say the answer is blue.
Okay, I will just go ahead and write.
The answer is blue over here.
So the color of the sky is blue.
Now when we need to predict this particular word, you know that we do not need further context, you
know, based on the previous word.
Right.
So based on this particular word, since we are as we know that.
Right?
Obviously the color of the sky is blue.
Right.
right?
So this word is mostly proportionate to this has a dependency to this particular word.
And obviously to get the output we don't require any further context, any further context.
That basically means there is no dependency on other word over here.
Right.
Uh, since we are talking about the color of the sky is blue, uh, you can see that the sentence is
also very small.
And when in terms of dependency, let's say there are two words there that are dependent to probably
find out the output.
One is sky and one is color.
The gap between this particular word is also not that long, right?
So obviously with the help of RNN, we will be able to solve this problem and it will even not face
anything like vanishing gradient problem.
So vanishing gradient problem will not be there, right?
But if I take one more example, let's say uh, I will go ahead and write something like this.
I grew up in India and let's say there is some more text over here.
And then finally I say I speak fluent Hindi.
Okay, now here you can actually see.
Obviously just by seeing this.
Right.
And let's say this is this is what I really want to find out.
This is what I really want to find out.
I want to find out this specific word.
So this is the word blank that I really want to find out the next word.
Now just by seeing this you can see Okay, fine.
If I probably go ahead with this entire sentence, obviously a word that should come over here should
be a name of a language.
Let's consider it should be a name of a language.
Right.
But which language?
Right.
They are different.
Different languages that actually exist throughout the world, right.
So obviously it requires some more further context.
Now, in this case, the context that it requires may be the name of the country.
Right.
And as I said that this sentence I grew up in India, this probably came at the first.
And then we had a lot of sentence.
And then finally we ended with, I speak fluent whatever language over here it is.
Right.
So obviously this word is dependent on this particular context, right?
And obviously you can also see that there is a long term dependency because there is a huge gap.
There is a huge gap between the output that I want to find and the word which has its further context.
So there is a huge gap over here.
So in this particular scenario.
So here you could see that uh uh, if I, if I just take this as an example okay.
So I will just go ahead and probably take this as an example over here.
And I will write it over here.
The reason is very simple because this is the architecture that we are looking at.
And similarly over here okay.
Let's go ahead and probably do this right.
So here uh, the problem was very simple that whenever I was trying to perform a task, okay, whenever
I was trying to perform a task over here with respect to this.
So just imagine that I want to find out this particular output.
And there is there is a dependency on this particular word to probably find out the output.
Right.
So here the gap is less So when the gap is less, so the dependency.
And it may also not be probably requiring any further context.
It is just directly dependent on this word or this word.
Right.
And it is being able to find out the output.
But if I have something like this kind of scenario where there is a huge gap between the output and
the further context word that is required, right?
Like in this scenario, let's say I want to predict this, I want to predict this.
Right.
But this is completely dependent on the first initial word.
And here you can basically see that there is a huge gap when there is a huge gap, I can basically say,
hey, there is a long term dependency.
And obviously RNN cannot solve this, right?
RNN cannot solve this long term dependency.
And it's very simple.
Why?
Because it faces this problem, which is called as vanishing gradient problem and vanishing gradient
problem.
What it does, it basically says, hey, whichever is the nearest word to this.
Like let's say this word is nearest.
This can really change the weights and it will be much more responsible in finding the next word.
Right?
But if we see the words that came earlier at t is equal to one, at t is equal to two.
This word, because of the huge gap that is present over here.
Right?
The huge gap is between the relevant information.
Right.
The output that we really need to find out, and with respect to the further context for the context
basically means it is having some kind of dependency on this particular word.
So there is a huge gap, right?
So this kind of scenario what happens vanishing gradient problem is basically faced.
And there the updation of the weights will not happen.
Because whenever we try to find out the derivative of loss with respect to derivative of words at this
particular timestamp, right, the weight will be there will be some value.
But at this particular timestamp it will be approximately equal to zero.
Right?
Why?
Because the derivative of a activation function that we are specifically using is between zero to 0.25.
Let's say in the case of sigmoid, in the case of uh, tanh it is 0 to 1.
Right.
And whenever we use this in our derivative chain rule of derivative right, it will be a big rule.
Right?
It will be a big chain right?
At that point of time, the value will be approximately equal to zero that we discussed in our previous
video.
Right.
So I hope you understood this.
And this problem that I would definitely like to say is something called as long short term sorry long
term dependency.
This is the problem that is basically faced with RNN.
And I've also taken one example.
If this is a smaller sentence, and let's say this particular word is just dependent on, you know,
the nearest word, the gap is not that much.
It will not require any further context.
Right?
So, uh, I hope you got an idea what was the problem with RNN?
And obviously now previous video we have discussed about the vanishing gradient problem, mathematical
intuition.
Now let's go ahead and see how our LSTM RNN solves this problem okay.
So before I go ahead, this on the left hand side is a basic representation of RNN okay.
Now here you'll be able to see that okay.
This is with respect to t is equal to t minus one.
This is t is equal to t.
Let's consider if I give some numbers right.
So let's say this is with respect to t is equal to one, t is equal to two and t is equal to three.
Right.
We are passing some information.
And over here you can see that uh the previous output from the timestamp is combined along with the
input right here.
There will be some weights that will be assigned.
So if you really want to just see this it will be looking something like this.
Right.
This is the same thing that we discussed.
Right.
So I have I have a neural network or a neural network which looks like this.
Right.
So I'll just take this as an example.
So what exactly this is.
Right.
So at timestamp t is equal to Ma one I'm passing this as an information over here.
Also will get some kind of information here.
There will be some weights that will be assigned.
We talked we spoken about this.
This will be my input weights.
Right.
Uh, here we are going to pass uh XI1.
This will be XI2 this will be XI3.
Again I'll be having w of I w of I.
And here it will be t is equal to two.
The here will be t is equal to three.
Okay.
Now just imagine when we are passing this input along with the weights I will be able to calculate my
O one over here.
And this is nothing.
But this is my O one right?
I'm passing this information to my, uh, next, next time stamp right over here.
Now, what will happen?
This will be assigned with another hidden neuron weights.
Sorry are done.
Other hidden weights okay.
And this both are basically getting combined over here along with the weights.
And then on top of it you apply a tan h activation function.
And finally we get an output.
So this is my h of t right.
And I'm also getting an output over here which is passed to the next state.
And same representation is basically used okay.
Now if I go ahead towards the LSTM RNN representation see guys nothing is different.
So this is the same thing like how what we discussed.
Because internally over here inside this hidden layer within each each and every neuron we basically
apply tanh activation function or sigmoid activation function.
Right.
So here it is basically given as tan activation function okay.
Now if I talk about LSTM RNN.
So this is what is the generic representation okay.
So don't don't get don't get worried.
Like it looks really complex when compared to this I will break down break this down each and every
thing as we go ahead.
Uh, but uh, just to give you a smaller idea, you know what exactly is done by LSTM RNN?
Okay, so let's say this is my RNN.
This is my RNN over here.
Now with respect to this RNN okay.
When I'm passing this information okay.
See RNN usually has this short term memory okay.
It has this short term memory.
So since we know that whenever we work with RNN, how is the generic representation?
It looks something like this.
And there is a feedback loop okay.
This feedback loop is specifically for short term memory okay.
Short term memory now with the help of LSTM RNN, what we are doing is that we are also adding a LSTM
memory.
Sorry, we also adding a long term memory.
Long term memory along with the short term memory.
Now we know that in RNN all already short term memory is there.
That is this particular feedback loop right.
In RNN we know that right.
So this is this this line that is specifically going to all the other hidden neurons.
This is nothing but your short term memory.
But what about long term memory.
So over here the long term memory is added something like this.
There will be a big line and how this line will be like, you know, uh, this line will be like, I
hope everybody has visited airport, right?
So in airport you will be seeing something like a conveyance belt.
Right.
Conveyance belt basically means, uh, where you specifically put luggages, right?
Luggages.
So there will be a continuously moving line.
Right?
Whenever an airport arrives, people will start, uh, the worker will start putting out the luggage
over here.
And when it reaches here, let's say there are some human beings.
They'll take out the luggage from here, right?
They'll take out the luggage from here.
So this is with respect to all the human beings who are specifically using it.
Similarly, in LSTM, RNN, there will be this long term memory Long term memory.
And the main work of this particular memory is to just make sure, add what context is required and
remove what context is not required okay.
So in this particular case, let's say if I take this particular example, uh, I grew up in India.
I speak fluent this.
Right.
So here you'll be able to see when we pass with respect to time XI1XI2XI3.
Like this when we go ahead and pass as soon as we talk about India.
Right.
Uh, let's say the word over there is India.
And we need to predict what is the language that I speak.
Okay.
So whenever till the context is, see this?
This, this memory cell will make sure that it will maintain the context till whenever it is required.
Whatever information is specifically required, it will store over here.
If it is not required, it will be removed from there.
Right.
That is what is the main importance of memory cell.
So that if I really want to predict this particular word, this India context will be saved in this
long term memory, right?
Right.
Even though the information is not there in short term memory.
See in short term memory.
If I probably consider the same sentence, first of all, I will go ahead and give my word.
I then grew, then up right in India.
Now once we went till India over here some other sentence came.
So the context switch has actually happened.
So what will happen?
We'll forget all this information.
Right.
But I know that we are going to go ahead and probably this sentence is going to come.
I speak fluent in so and so.
Right.
So what memory cell is going to do?
It is going to save this India word, or it is going to save this context in the memory cell.
Right.
So finally when it comes like I talk in whatever language it is, it will go in and check in the memory
cell what all information is specifically there, right.
And if that context is available based on that we will do the prediction.
Right.
So this is just like a book of library okay.
And obviously we as a human being right.
If I tell you hey um, let's say I am learning, I'm studying something.
You know, after some time I forget something.
So what I do, if I want to probably refer that I will remove the book from this particular shelf,
and I will try to go ahead and revise it.
Okay.
And then whenever I do not require, I may forget it.
Right?
I may just say remove from my memory because it is not important.
But when exam day comes, I really need to remember all these things.
So this is also like my brain cells which we also call as our memory cells.
Till what extent we can basically remember.
Right.
And sometimes, uh, based on the further context, based on the exams, based on the test.
I need to know like how much things I really need to remember, right?
So that is the entire funda about LSTM RNN.
I've just given you an idea.
So this is basically called as a memory cell.
I will be discussing more about it because in the next upcoming videos we will break down this entire
architecture and we'll see how it is probably solving this long term dependency.
Right.
So I hope, uh Uh, you understood the specific video.
Uh, this was it for my side.
So in this video we have discussed about, uh, LSTM.
What is the problem over here?
Why LSTM rnn?
Okay.
And uh, we have also seen the basic representation of both LSTM and RNN.
Now in my next video I am going to talk about how RNN works.
I give you an idea that okay, we will be having a long term memory.
Separately, we'll be having a short term memory.
Whichever information I need to store for a longer period of time, I'll put that information in longer
long term memory.
Whichever I do not require it, I'll remove it from the long term memory.
Okay.
And whenever I need to reuse it, I will go ahead and reuse it.
Okay.
So, uh, yes, let's go ahead and discuss about this in the next video where we'll be talking more
about how our LSTM RNN works.
Thank you.