Hello guys.
So we are going to continue the discussion with respect to our deep learning and NLP.
Already we have discussed about this four important neural networks that is simple RNN, LSTM, RNN
grown and bidirectional RNN.
Right.
And here you should definitely know what is the problem in simple RNN.
The first problem in simple RNN is about vanishing gradient problem, right?
And because of this vanishing gradient problem, what we did is that we saw the different variant uh,
of RNN, specifically LSTM, RNA and RNN.
Right.
Here, uh, you could see that by using this two techniques we were able to add long short term memories.
Right.
Long short term memory.
And because of that, you will be able to see that this was very much efficient in solving problems
which were of, uh, you know, many to one RNN type.
You know many to one like sentiment analysis or predicting the next word.
And many more things as such, right?
Then we understood there were also some problems with LSTM, RNN and GRU RNN wherein like if I really
want to do a prediction of a word, if there is some context of the further words that is required,
or if there is some kind of dependency of that particular output.
Uh, with respect to the upcoming words, at that point of time, we cannot actually use LSTM or GRU.
So for that we had this bidirectional RNN.
Right.
And we also understood the architecture.
Now let me talk about one type of RNN which is specifically called as many to many.
Right.
So many to many RNN specifically indicates let's say if I have this specific RNN like this, right?
Many to many RNN basically says that I have Multiple inputs, right?
I have multiple inputs and with respect to that I have multiple outputs.
Y1Y2Y3.
Right now in this scenario you will be able to see.
Let's talk about some of the use cases.
Some of the examples let's say machine language translation okay.
Now in this specific example let's say if I give a sentence in English I need to convert this into French.
So here I know there will be many words as input in English language.
Similarly, when I try to convert this back into French, you'll be able to see multiple words with
respect to French language.
So this basically becomes many to many type of RNN right.
So here I will just write many to many RNN.
Now whenever I talk with respect to this kind of architecture or this kind of use case.
This is nothing.
But this is called as sequence to sequence neural network, right?
Now why do I say this as sequence to sequence neural network.
This is very much important to understand right.
I will not say neural network itself, but yeah we can also say sequence to sequence neural network.
But here we will be seeing that we are trying to solve a problem which involves sequential data as in
my input.
And the output is also sequential data.
Right.
So use during this kind of use case where my input and output are both sequential data.
I cannot specifically use LSTM and GRU or simple RNN.
It will not give us that much good accuracy, right?
So considering this, we will be learning a new architecture which is called as encoder decoder architecture.
Okay, again, let me repeat what I am specifically talking about.
Right.
So everybody knows about this one type of RNN, which is called as many to many RNN, right?
In many to many RNN.
If I take some of the examples, let's say I want to create a model which will be able to convert one
language into another, right?
One language to another.
So some of the examples with respect to this is Google translation, right?
So let's say if I give a sentence with respect to English this should be converted into French.
Now here, if I'm giving multiple inputs over here, let's say over here, I give a sentence saying
that I like to eat food and in French it will automatically get converted right in French language.
So in this kind of case, this is a kind of use case which is called as many to many RNN, right?
Where I have multiple sequences as the input, multiple sequences as my input.
Similarly, my output is also a sequence of words, right?
Another example, which I can probably consider is in LinkedIn chat bot or in LinkedIn chat.
Right.
Whenever you're chatting with your friends, right?
Let's say if I say hi, how are you?
Now with respect to this, hi, how are you?
You will be able to see that you'll be getting some suggestion.
I'm good.
You know which LinkedIn will automatically provide, right?
Okay.
Thanks.
Something like that.
So that is also a kind of sequence to sequence problem statement.
Right.
Because my output automatically it is being able to suggest some kind of sentence.
Right.
So this is some example of sequence to sequence data set.
Right.
Whenever I have this kind of use case where I have multiple inputs and I want to probably get multiple
outputs right with respect to sequence, then we cannot use simple RNN or LSTM or GRU.
It will not be that much efficient.
So for that we specifically use a new architecture.
Even we cannot use bidirectional RNN, right?
If I have use cases like which is like many to one right, 1 to 1, I can use this kind of, uh, neural
networks.
So for this we specifically use a new architecture which is called as encoder and decoder.
Now we will try to understand how does an encoder and decoder architecture look like.
And how is it different from all these RNNs that we specifically have?
Okay.
So let us go ahead and let me just give you a simple working, simple working of encoder and decoder.
So as the name suggests encoder.
So there will be one encoder block over here okay.
There will be one decoder block over here right now with respect to this encoder and decoder.
Right.
We will be giving our input sentence.
Obviously we will be applying an embedding layer over here.
Once we give this input sentence it will be converted word into array of numbers, which we basically
see it as embeddings.
Then we create a hidden state.
This hidden state is nothing, but it is basically called as context vectors.
Context vectors okay.
We will talk more about this.
What exactly is context vector.
And then this is forwarded to the decoder layer.
Now in the in the the main aim of the decoder layer is that it generates the output word by word and
keep feeling of the words.
Keep feeding the previous words into the decoder again.
Okay, so this is what exactly happens in the decoder.
The decoder generates the output word by word and keeps feeding the previous word into the decoder.
Again okay.
So this is a simple working about encoder and decoder right.
Just get this particular understanding that I have an encoder.
This encoder work is to get an input right.
And with respect to this particular input, its work is to probably finally generate a context vector.
And if I take this context vector and give it to my decoder now okay.
Or if you want some other diagram, I will just try to draw it for you.
So if I give you one more example with respect to this encoder decoder diagram.
So here I will just show you one more diagram over here.
Okay.
Now see in this diagram what we are basically doing.
This is my input text I am giving it to the encoder.
It is generating a context vector.
And this context vector will be given back to the decoder along with the output text.
And finally we should be able to get the output okay.
Instead of summary I will not say summary, but I'll probably talk about this as an output.
Okay.
Now I know this diagram looks very much simple, and I hope you may be getting more confused just by
seeing this.
So what I am actually going to do is that in order to show you one more better diagram, this is how
it exactly happens.
So I need to show you what exactly happens in an encoder.
So as I said, this entire architecture has one encoder.
Encoder takes the input and converts all this input into a context vector.
This context vector talks more, has the entire information about that particular input in the form
of vectors.
Right?
Then the second thing is that once we have this particular vector, we have to pass all this vector
to the decoder to solve our use case.
And this context vector will also pass get passed with this particular decoder.
And finally we will be able to create our output over here.
What kind of use case will be specifically used over here.
Some of the use case like like language translation.
Language translation.
Text generation.
Right.
Text generation or sentence generation?
I'll say not word generation, but the complete sentence or text generation.
Right.
And some more use case you will be able to see as text say like how we have this LinkedIn chat bot suggestion.
Right.
Or I can also go ahead and write it as text suggestion.
So all these kind of use cases will be able to get solved with the help of encoder and decoder.
Now let me do one thing.
Let us go ahead, one step ahead and try to understand what is inside this encoder and what is inside
this decoder.
You really need to understand that.
So for that we have this amazing diagram again.
And this we basically see it as sequence to sequence encoder and decoder neural network.
Usually inside the encoder and decoder we have LSTM, RNN okay why LSTM RNN?
Why not RNN?
Because c RNN already has a problem which is called as vanishing gradient problem, right?
Vanishing gradient problem.
Now because of this problem we cannot use vanishing gradient.
Instead you can go ahead and use a LSTM, RNN or GRU inside this okay.
Now see this one.
Very much simple.
And here you will be able to get the complete meaning.
What does encoder actually do and what does decoder actually do okay.
So here you see that it is an LSTM layer.
And this is basically ruled based on the output.
Right.
Or sorry based on the input that I have.
So here this line that you see this is basically your long term memory right.
If you remember in LSTM it is your long term memory.
And this is your hidden state right.
So along with this input whatever operation basically happens inside the LSTM.
If you remember we in the LSTM we have something called as forget gate Gate.
We have this input gate, right?
Uh, we have something called as candidate input.
Candidate input.
We have discussed all about this.
Third thing that we specifically have is our output gate.
Right.
So all this gate, whatever operation, what operation it basically does, it does like what context
needs to be added in the longer memory cell and what context needs to be removed from the memory cell.
Along with that, what needs to be retained in the long short term, sorry short term memory cell,
which is nothing but the hidden state.
As I said, this is my this is my long term memory, right?
And this is my short term memory.
Right.
We use this long term and short term memory.
Now whatever operation is basically going to take with respect to that input.
Now see there is an input which is called as thank you.
Right.
And I need to convert this word.
Thank you let's say into French which is basically written as gracias.
Okay, now in order to convert this how my input data will look like.
So this will specifically be my data set.
In my data set I will be having my English word, and I will be having my French word right now with
respect to this English and French word, the first word will be something like thank you.
And the conversion that you specifically have is something like gracias.
Okay, I hope this is written in French.
Okay.
Now what does over here you will be able to see that inside this LSTM.
I will be passing my entire sentence.
Okay.
I will be passing my entire sentence along with this sentence.
What I will do, I will add a special character like SOS and end OS.
SOS basically means it is just to indicate it is the start of this particular sentence, and iOS basically
indicates it is the end of the statement, right?
Similarly, for this French word, I will also go ahead and write SOS.
Gracias, and we will write it as EOS.
This is important just to make sure to indicate the neural network.
What exactly is the start of the statement and end of the statement?
Okay.
Now this thank you is basically getting converted into a vector okay.
So let's say over here by using all these characters.
Now how many characters I have 1234.
Now if I am using a one hot encoding right.
So first word that is probably going to go into the LSTM.
So what we are basically going to do this will basically be my LSTM in my encoder okay.
So let's go ahead and draw this.
So if you want I will draw it properly okay.
So this is my entire encoder just to give you an idea how the entire training basically happens.
So this is my encoder.
Now inside my encoder you know that what I use I use LSTM right.
So this is my LSTM inside my LSTM.
You know that I should have two memory cell.
One is for long term and one is for short term.
This is indicated by CT.
This is indicated by DT okay here forget gate, input gate and output gate.
All the operation is basically going to happen right.
Whatever output that is probably coming from here I will not be taking that.
You know.
So what we do in the encoder stage is that very simple.
We give every word one by one to the input over here to the LSTM.
Right.
So let's say the first word that I'm actually going to give is nothing but SOS.
So this SOS word how do I give it okay.
Let's say if I'm passing SOS word right.
First word is SOS.
So in the first instance SOS when I pass this entire embedding will basically happen.
Right here we can use word two vec.
Here we can use any kind of embedding techniques.
Right.
But the most efficient one is embedding, uh, embedding technique with respect to embedding layers.
Right.
So here I have SOS.
So for SOS since I have total four words this will be one and remaining all will be zero.
Let's say that I'm just using one hot encoding right.
So in this particular word my SOS is getting passed okay.
Then in the next step what it will do in time is equal to one.
It will go to my next LSTM.
Right.
Now in this LSTM what I'm actually going to pass my next word.
My next word is nothing.
But thank you.
Right now in the case of thank you this will be 010 and zero right.
So this will basically be thank you according to one hot encoding.
Right.
If I directly use word two vec I will be getting a different vector dimensions.
Right.
Then coming to the next third word okay.
So this operation is specifically happening.
Then in the third instance I will pass my third word right.
In this particular third word, I will just go ahead and pass you.
You is nothing, but it will give this particular input as like this.
And here, instead of writing thank you, I will just pass you over here.
This will be my third word right now.
Similarly my fourth word what it will happen again.
It will go to the next LSTM.
And this is my entire encoder now okay.
Encoder.
Right.
So here I'm actually going to pass my fourth word which will be my end of statement 0001.
This is nothing but end of sentence right now.
This is how we go ahead and pass it over here.
And after this, the operation that we are basically going to do, we will be getting our CT and DT.
So this will be my CT and DT.
What does CT indicate?
CT indicates my long term memory and DT indicates my short term memory Right.
Short term memory.
Now, this long term and short term memory combined is basically called as context vectors.
Context vectors.
In short, all these vector operations will probably be taking place inside this all the operations.
Because inside this we have this forget gate.
We have this input gate, we have this output gate.
So with respect to all this time stamp, when I'm passing with t is equal to one, t is equal to two,
t is equal to three, t is equal to four.
Right.
I'm passing this particular sentence one by one right.
So this is what exactly happens inside the encoder.
Okay.
It is not like you have to learn decoder separately.
Decoder will also happen at the same time.
Now see this okay.
So encoder what it has done is that it has taken all the words inside the sentence.
And it has helped us to generate the context vector.
How did it generate the context vector?
Whatever operation is present inside this with respect to forget gate with respect to input gate with
respect to output gate.
Write all these three operations that it has write.
And with respect to that I need to combine this.
I need to add some input in the CT.
Otherwise remove some context from the CT right?
We finally get this entire CT and DT right?
Now we will go ahead and create our decoders.
Now in the case of decoder what will happen.
See this okay.
So in the case of decoder let's say in our training data set in the French word is gracious right.
So what we will do over here again we will go ahead and use one LSTM.
Right.
Now in this particular case of LSTM what is basically going to happen.
Now see this in decoder stage.
What is basically going to happen.
This entire CT and DT will get passed.
We are going to pass this entire vector to this.
Now you know that we have transformed our training data.
So this I'm considering as my training data okay.
now for the training data.
Also my output.
This is my output, right?
And this is my true output.
We basically say it as y truth, right?
Not y hat.
Y hat is your prediction.
Now in your y truth you know that there is SOS, right?
There is grass and there is iOS.
Along with this context vector, what I will do, I'll training it.
You know, so for the first instance I will go ahead and pass my input in this case first one that I'm
going to pass is my SOS.
Now, since there are only three words in the output like that in the entire data set, you may have
many words, but I'm just considering.
As an example, there are three words, so SOS is indicated by 101100.
In this particular case this will basically get passed over here, right?
How did I convert this?
Again by using the same one hot encoding technique.
We can replace this entire technique by embedding layer.
But just to give you an idea what exactly is happening, I'm actually writing it over here right then
with respect to this particular output.
What we do over here, we pass it to a fully connected layer with softmax activation function.
Okay.
So let me just go ahead and talk about it.
Right.
You know what operation is basically going to happen over here.
Because here also there will be forget gate.
There will be input gate and there will be output gate.
After we get the output we basically pass it to softmax.
Why softmax?
Because you will be able to see that I am having this three words, right?
I'll be showing that I'll not be having three words, but I have added this start of sentence and end
of sentence.
Right?
So with respect to softmax, I will be getting let's say three outputs over here one, two and three.
Okay.
Now with respect to this output, let's say my vectors look something like this I will I will get the
vector something like this.
Values let's say over here.
Or let me define it something like this.
So this will basically be my y hat.
Y hat basically means predicted value.
And this y hat.
Since we are using softmax y, we are using softmax because this is becoming a multi-class classification.
So with respect to shows that, I'm actually giving what should be the output that I should get along
with this context vector.
So let's say that here I'm going to get three coordinates.
Let's say I'm going to get 0.5 I'm sorry I'm going to get .0.1.
I'm going to probably get 0.4.
I'm going to get 0.5.
And I'm going to get 0.4.
Right.
So when I got this three vectors, we will just go ahead and see which value is the maximum in this
vector.
If it says 0.5, that basically means it is going to give us the output as gracias.
Okay.
So here what we do, we will convert this back into this particular output because the second second
index or second vector that is basically there it is for this particular word gracias gracious.
Because it is present over here, right?
So this is how we are able to get the y hat.
Okay.
Yeah, it there may be scenario that I may not get this particular value.
Also I may get that third value having a highest value.
Highest value with respect to probability.
Right.
Because when we are using softmax here we are basically going to get the probability of y hat right.
And right now since this is a multi-class classification based on three different vectors, whichever
vector value is high, uh, with respect to the probability, let's say vector value.
But I'll say this is the probability value, right.
Again let me repeat this.
So here once I give get the output I'm going to pass this through a softmax activation function.
And we know that over here there are three words S.O.S. gracias and iOS.
With respect to this particular S.O.S. along with context vector, it should predict something.
And let's say if it gives this three output that I have got over here, let's say if it gives this three
output okay, since we are actually getting this y hat, if it gets this three output, one is 0.1,
one is 0.61 is 0.03.
Now, in this case, the probability of 0.1 is.
For this SOS word.
The probability of 0.6 is for gracious, and this probability of 0.3 is for iOS.
So whichever will be the highest probability that will be my predicted output.
So here I will be specifically getting gracious okay.
Yeah.
There may be scenario that I may get another probability I may get 0.1 over here, 0.3 over here and
0.6 over here.
In this particular case, what will happen?
The prediction will be end of sentence.
So if I am getting end of sentence that basically means I will not go further.
Right.
But again during the back propagation I will propagate.
I will update all the weights that is involved over here.
And then we will again do the forward propagation.
And we will make sure that this y minus y hat should be less right or it should be reducing right.
So let's say I got gracious over here.
So this is my start of the sentence I got.
I did all the calculation inside this whatever happens in the LSTM layer.
And finally I got the softmax and I got y hat.
So in the next step what happens is that it will probably go to my next LSTM over here because I've
got the right output that is called as gracious.
And now with respect to this particular gracious, what I will do, I will pass this output back to
my next LSTM.
Right.
And you know that this HT and KT will also be going over here.
Right then along with this again, we will go ahead and pass this to our softmax.
Now with respect to the softmax, if my training is going on well, I should be able to get my y hat
over here with respect to the probability that I have.
And here my output should be end of sentence.
Right.
Because gracious after gracious, I don't have any other word.
Right.
But here my next word that is probably be going will be nothing but gaseous vectors.
Okay.
Whatever vectors is over here, that is zero, one, zero.
Right?
And finally, when I get end of the sentence, that basically means I have to stop training over there.
Okay.
Now this is the best way of training, right?
While I'm getting the exactly accurate value.
But here you will be able to see that if I try to break this up.
Right.
With respect to the decoder, I will be having y y hat.
Right?
I will be having y hat.
This y hat will again be happening with respect to the timestamp of t is equal to one two, three.
Right now with respect to t is equal to one.
Let's say over here with respect to t is equal to one, t is equal to two, t is equal to three.
We will be having our y truth value.
We will be having our y hat.
Right.
There will be this three vectors.
Right.
Again here will be three vectors.
Here there will be three vectors.
Again there will be three vectors.
Here there will be three vectors right now during this particular scenario, after we do the forward
propagation, what we do.
We basically calculate the loss.
And loss is nothing but y minus y hat whole square right.
It can be whole square.
Or we are just trying to find out the difference.
And our main aim is to reduce this loss.
Right.
So what we do in this particular case, once we find out the difference, we will try to calculate the
average.
And then what we will do is that we will try to reduce this particular loss.
And when we are reducing this loss, how do we reduce it.
We again have to use something called as optimizer right.
Now when we are using this specific optimizer it is just going to update all these weights that needs
to be updated inside all these things.
Then in the back propagation all the weights will get updated.
Then again we will do the forward propagation.
But you understand one thing over here is that over here during the input we need to pass all the inputs
first, then the output from the last layer or output from the last LSTM with respect to time.
That context vector.
Gets passed to the decoder.
So this basically becomes the encoder.
And this is my decoder right.
So this way the entire training.
Of the sentences happens.
And that is what is basically shown over here we have the softmax activation function.
Again we pass it to the same softmax activation function.
And we are unrolling it right.
If the loss is high then we have to again do the back propagation.
Update all the weights in this LSTM from till this encoder from the first stage of encoder.
And again we have to probably do the forward propagation and do the same thing again.
Find out the softmax activation function, what values I'm actually getting.
The most important thing is that how do we convert this into vectors okay.
So if I really want to probably draw this particular diagram in a much more easier way.
So this actually becomes my encoder.
Okay.
Then this is my decoder okay.
And with respect to the encoder and decoder, I will be having my LSTM rnn unrolled with respect to
time.
Okay, now you know, in this LSTM I have two important lines that gets passed and this two important
line will be responsible in creating my context vector.
This is the inputs that I'm actually going to give.
Here we are going to use an embedding layer which we have already discussed about embedding layer.
Right.
And with respect to this embedding layer I will be passing my X11X1.
Sorry.
Here embedding layer I will be passing my X11X12X13.
And let's say my X13 is end of sentence okay.
Then whatever output that I get this is nothing.
But this is my context vectors right?
The same output is my context vector.
Then here we again go ahead and create my LSTM for the conversion sake.
Whatever we need to generate this will be passed towards this right.
Along with this we will start over here with Soas right here again we will be using another embedding
layer.
In this embedding layer I will be passing my Soas first.
Okay.
So here also first Soas needs to be passed.
Okay.
Then whatever output I get after performing my softmax.
So let's say here I'm going to write my softmax.
Right?
The same thing will probably get passed to my input.
Okay, so here I will just rub this because this input will be only till here.
Here only I'll be using my embedding layer.
But the same vectors will get passed over here.
Then again I will be getting an output again.
The same vectors will be passed over here and here.
Finally we get our iOS end of statement.
And this is how we probably go with the entire forward and backward propagation.
This part is basically called as encoder.
This part is specifically called as decoder.
Right.
So I hope you got an idea of encoder and decoder.
Uh, what exactly it does.
And usually we use an LSTM over here.
Context vector are nothing.
But it is just like an arrays itself with respect to the calculation.
But whatever sentence we are probably passing right, it probably will give you the entire summary of
this particular sentence or context of the sentence.
Context of this sentence.
Right.
So, uh, I hope, uh, I know there are so many things written, but understand the context, but other
than this, the forward and backward propagation will be almost same.
So yes, this was it from my side.
And here we have actually discussed about the encoder and decoder architecture.
In the next video we will talk about uh the problems of encoder and decoder architecture.
So yes this was it from my side.
I'll see you in the next video.
Thank you.