Hello guys.
So we are going to continue our discussion with respect to encoder and decoder, uh, sequence to sequence
architecture.
Already we have seen how it exactly works.
We have seen the architecture with its complete in-depth intuition.
Uh, but let us go ahead and let us try to understand the problems with encoder and decoder architecture,
sequence to sequence architecture.
And I will just try to understand the problem.
And in order to solve that particular problem, what we are specifically going to use and what architecture
change will specifically happen in this encoder decoder sequence to sequence architecture that also
we will be seeing in the upcoming video.
So, uh, let me just go ahead and let me quickly go ahead and create a generic structure again.
So this is my encoder right now with respect to the encoder, you know that let's say I'm using a LSTM.
So over here I have my LSTM here.
Also I have my LSTM and similarly here also I have my LSTM.
Let's say with respect to time I have just drawn with t is equal to 1234.
Right.
And here you know that we specifically use embedding layer.
So here I'm going to basically use an embedding layer.
So embedding layer okay.
Now with respect to embedding layer uh here we are going to give X11.
Let's say this is just for the first sentence X12X13.
And this will basically be my X14 which is end of sentence okay.
Now you know that, uh, this entire information will be going over here in the terms of vectors.
So here a vector will be created here.
Also a vector will be created based on this embedding layer.
Here also a vector will be created.
And finally here also vector will be created.
Okay.
Now when we are discussing specifically with respect to this, uh, you know that here, whatever output
we are going to get, okay.
We do not consider this.
So here the final output will be considered, which will basically be my w.
Let's consider this as w.
This is my context vectors right.
And usually this context vector is represented by your uh two important lines.
One is your long term memory cell and your, uh, hidden state.
So I will just combine this specific line over here.
Right.
And this line only needs to be forwarded to the, uh, next decoder.
Right.
So with respect to any use cases which have this entire encoder and decoder architecture is usually
solving a sequence to sequence problem statement.
Right.
So here, uh, let's consider this is my decoder.
And here again I will be having my LSTM LSTM okay.
And let's say in this case I'll just go ahead and draw three right.
So what is basically going to happen.
This entire line will get passed to this okay.
And here I will again pass my S.O.S..
Let's say uh start off the sentence.
Okay?
I will get my output.
This state will still be passed till the last layer.
But this particular output over here, let me just go ahead and create this.
This will basically go ahead and pass to a softmax activation function.
Softmax activation function.
Right.
And with respect to this softmax activation function, I'm going to get the output.
Then this output will further get passed through the input to the next LSTM.
Similarly this will get passed over here.
Again with respect to the output I will again go back to the next one.
And finally I get my last output, which may be my end of sentence.
Right?
And this will be my y hat y hat.
Right.
It can be y one hat y two hat.
Right now, this is the entire architecture of an encoder decoder.
Right.
But now let's focus on this context vector okay.
So when I say this is my context vector I'm also considering it over here itself because there will
be two lines that will be passing.
Right.
One is my KT, one is my HT.
Okay I will if I combine both of them this act exactly becomes the context vector, right?
So that is the reason why I'm representing this, because I'm just going to consider the output of the
last, last uh, LSTM with respect to time.
Let's say this is the last time where I'm getting the sentence.
So this will be the context vector.
But understand what is the importance of this context vector.
This context vector represents represents the entire sentence, right.
This entire sentence that we are passing right now.
If this is a smaller sentence, write this context vector will be more than sufficient to represent
this particular sentence.
But what did researcher do?
Researcher.
When they were testing right, they used sentences of varying length To test it, write sentences of
varying length.
Right.
And they used the performance metrics.
Again.
If you go ahead and see the research paper with encoder and decoder, this research paper was something
related to Bleu score okay.
And here on the left hand side this will basically talk about sentence length.
Now when they use this encoder decoder architecture, you could see that they drew a graph which looks
like this.
Right.
That basically means as the sentence length kept on increasing, you know, after a certain point,
you know, over here, let's say this is 30, 40, 50 and the sentence length went till high, the accuracy
started decreasing.
The blue score started decreasing.
Right.
And this specific score is the performance metrics to test how your model is performing with respect
to sequence.
To sequence data.
Sequence to sequence data.
Right.
So this was the problem.
And why this is specifically happening.
Because they understand the context vector that we are probably considering.
We are not taking this output right.
We are not taking this output.
Instead we are just taking the last output, whatever context vector it is basically creating.
Now, this context vector, you know, will have more information with the nearest word that is just
passed in this particular timestamp, in the nearest timestamp, if this is my t for t is equal to three,
t is equal to two, t is equal to one.
You will be knowing that this context vector will talk about more, will have more information about
those words which will be to the nearest timestamp.
Right at t is equal to four.
But what about at t is equal to one?
And if my sentence keeps on becoming bigger and bigger, let's say if this is t is equal to 100.
If there is t is equal to one, the single context vector will be having more context information of
the recent words that are there with respect to the timestamp.
But what about those words that came at t is equal to one, t is equal to two.
It will have less context because here the context vector will be having limited information about it.
So the the the context vector that is used to represent the entire sentence will not be sufficient.
Right.
And that is the reason why for longer sentences the blue score was very, very less.
And this was the problem with the encoder and decoder sequence to sequence architecture.
So how do we specifically fix this problem?
For fixing this problem we have another new amazing mechanism which we will be talking about which is
called as attention mechanism.
And this attention mechanism.
We will talk regarding it sequence to sequence networks.
We'll try to understand the architecture in the next video.
What exactly change we really need to do in this encoder and decoder.
Because this is my encoder, right?
This is my decoder.
The work is very simple.
My encoder will encode the information and my decoder will decode back that particular information.
That is why we use use cases like text generation, sentence generation, uh, language conversion.
Right.
Different kind of use cases.
So we will try to understand what exactly is this attention mechanism.
Just to give you a brief idea.
Right.
Whenever we have a longer sentence or longer paragraph or wrong with this context vector, right that
we are specifically passing, we will pass additional context.
How do we pass this additional context?
We will see about it.
What architecture diagram will be changing?
We'll see about it.
But just to give you an idea, here is the research paper that we will be discussing about neural machine
translation by jointly learning to align and translate.
And over here, if you go ahead and see this architecture, this is my encoder.
This is my decoder We will be discussing about this particular architecture and how this architecture
is different from the previous one.
Here you can see in the encoder you just don't have only one LSTM layer.
You also have you have bidirectional LSTM layer.
So here also you have this here also you have this and then you probably combine all the information.
You pass that additional context to your decoder.
And that is what we are going to go ahead and have a look.
Right.
And then we'll be understanding about each and every questions as we go ahead.
Right.
Because whatever output you will be specifically predicting with respect to the decoder here, you will
be having the context of the previous state.
You will have this additional information that is probably coming up.
Uh, and along with that, whatever you will be your previous output that will also be there.
Right.
So we will try to combine all those things and we'll try to see.
But before that, uh, let's uh, go ahead and see.
And I hope you have understood.
What is the problem with respect to your, uh, encoder and decoder sequence architecture?
And that is what I'm going to discuss.
In the next video, we will discuss about attention mechanism, how we will provide some additional
attention while training this entire models.
So yes, this was it from my side.
I will see you all in the next video.
Thank you.