Hello guys.
So we are going to continue the discussion with respect to attention mechanism.
Already in our previous video, we have seen the problems with encoder and decoder or decoder sequence
to sequence architecture.
And we also understood if we have a context vector, like if we have a longer sentence right or longer
paragraph.
Whenever we try to give this specific paragraph over here, word by word, the context vector that is
created by the encoder, sometime it will not be sufficient enough for the decoder to probably get the
output.
So based on that, we also saw that okay, uh, the performance metric which was used by researchers,
which is called as Bleu score.
As the sentence length was increasing, you could see the Bleu score was also going down right.
So in order to overcome this, uh, advantage or disadvantage, we are specifically using something
called as attention mechanism and the main idea behind attention mechanism was that for longer or shorter
paragraph, along with the context vector, we will go ahead and provide more context, right?
So this was the entire idea over it.
Now for this, we will specifically be using this particular research paper to understand this is an
amazing research paper, uh, neural machine translation by jointly learning to align and translate.
Anyhow, I will be giving you the specific link.
You can also go ahead and explore it, but if you just go and search, you will be able to get the link
right.
And over here the researcher was Yoshua Bengio.
I hope you know him is a famous famous researcher and scientist in this specific field and specifically
in the field of NLP.
Along with that, uh, you have uh, it's very hard to pronounce, so I don't want to really pronounce
it, but I'm extremely sorry.
Uh, you know, I may pronounce it wrong.
Right?
And they are respected personalities.
I don't want to do that.
Okay, so, um, we will discuss about it, like how this, uh, attention mechanism is solving the
specific problem.
And if you go down, you know, this is the main decoder, general description.
Uh, we will try to change the architecture of encoder and decoder.
What?
It will basically be changing.
We will talk about it okay.
So first of all, uh, let's go ahead with a simple example over here.
Right.
So before we go ahead, let's go ahead and probably discuss the encoder decoder architecture once again.
And then uh we will try to understand the problem.
And quickly we will see what kind of changes is basically there with respect to this architecture.
The reason why I'm again, going back to encoder and decoder architecture, because here in the research
paper there is some different kind of notation that is used.
Okay.
So here uh, what I'm actually going to do, let's go ahead and probably create the LSTM in the encoder.
So this will be my H1 tag.
Uh sorry not H1.
So this is my LSTM uh neural network.
So this will be my H2 with respect to time.
Then I have my H3 okay.
So let's say that I have this three text that is probably going on.
So this is my x one.
This is my x two.
This is my x three okay.
Let's say over here I will just go ahead and write.
Hello.
This is the text that I'm actually sending.
Hello.
What's up okay I'm giving this particular text.
Now if I consider this as my encoder.
Right.
So the outcome of this particular encoder will be that obviously we'll take the sent text and we'll
convert uh, we'll use a word embedding layer and convert that into vectors and pass it to the hidden
layer right hidden layer, which is LSTM.
Then uh, once I probably go till the end of the sentence, that is what's up?
So at the end of the day, from here we will be able to get two important information.
One is s zero, S zero is nothing but your hidden state.
And along with this we will also be getting one more uh, very important parameter that is called as
c.
C is nothing but context vector.
Now see why I'm using this notation?
Why I did not use the previous notation like this.
Because in the in the research paper also, I have this as zero and all.
So it will be very easy for you all to understand.
So from this I will be getting two important uh, information.
One is one is S zero.
That is the hidden state.
Okay.
And then I have something called as c okay.
C is nothing, but it is my context vector.
Okay.
Now the next step will be that from this S0.
Now this is my encoder stage right encoder.
And first of all we will wait for all the inputs to come in the encoder.
All the internal processing will basically happen you know.
Or because they have three amazing gates over here for for get input and output gate.
Once we get this context vector this context vector will further get passed to the decoders.
Now with respect to the decoder, first of all, I will just go ahead and create one decoder over here.
In this I will just go ahead and create three LSTM neurons.
Okay.
With respect to time, you know I need to probably create this also with respect to time, right.
So here I will use some another color let's say.
So this is my first, second and third.
I need to convert this three words.
Right.
So what used to happen in the decoder over here.
So this is my entire decoder right.
Let's go ahead and divide like this okay.
Inside this decoder first of all we used to pass this as zero.
So s zero will get passed over here right then the C used to get passed over here okay.
Okay.
And along with this context vector, what used to happen is that, uh, our first, um, first input
that we were passing.
Right.
Our first input used to be y zero.
Y zero is our truth value.
Let's say this is the source.
Start off sentence okay.
So this used to get passed over here.
If I'm not passing this source then whatever is the truth value with respect to my first sentence.
Right in my training data, you know, let's say in my training data, I have this values, right?
I have something like this.
And inside this I have this English text.
Hello?
What's up?
And inside my French I'll say gracias.
Something.
Okay.
Some text.
Okay.
Now in the first y zero, I should pass.
gaseous.
Okay.
So this was gaseous that was passed over here.
Now as soon as this gaseous was passed by using this context vector along with this context vector,
it will passed.
It will get passed in the um LSTM.
Okay.
And this LSTM will be responsible in finding your y1 right, and y1.
How do you calculate it.
Because here another softmax will be there.
Right.
So if I just go ahead and see there will be another soft max over here.
Right.
So this will basically be my soft Max.
Okay.
Now once I calculate the y hat or y one okay.
This is y one hat.
Okay.
Then we used to pass this y one hat to my next next LSTM.
Right.
To calculate the next word right along with this.
What we used to also do is that uh we used to take this entire context vector and again pass it over
here.
Right.
And by using both of them, we used to go ahead and calculate our y two hat.
Right.
Similarly to calculate the y three hat.
Uh, I will go ahead and pass just a second.
I will go ahead and pass this to my next neuron.
Along with this, I used to also pass this my context vector.
And then finally I used to get my y three Y3 hat and which can be actually my end of sentence, right?
So this way was the entire process of the encoder decoder.
I missed this line previously, so I'm again writing it down.
So this was my encoder decoder architecture.
Right.
And what was the problem with this architecture.
It is very simple.
This if my sentence is very long, this context vector will not be able to capture the essence of the
entire sentence, right?
Because obviously, if my sentence is of a length of 100 timestamp, right, 100 timestamp.
So I'm probably going with till t is equal to 100.
During that particular case, whatever context vector will be created at that point of time, only this
will be able to capture with respect to the nearest timestamp.
When it is t is equal to one and all will not be able to capture the essence.
Right?
And because of this, you could see that from the blue score.
When the context, when the sentence length was increasing, it was not able to give us a good accuracy.
So what exactly were the changes that were made in this, uh, you know, in this attention mechanism?
Okay.
Now I will just follow this particular architecture.
So here you will be able to see this amazing diagram.
Okay.
And this amazing diagram will try to understand what exactly it is okay.
So first of all, uh, over here you'll be able to see that this is my encoder.
Okay.
This is my encoder and this is my decoder.
Okay.
Just to give you a clear idea, this exactly is my encoder.
Okay.
And this exactly is my decoder.
Now, what exactly is the difference between this encoder and decoder and why there is so much architecture
difference?
Okay, so to make you understand again what is the architecture difference that we will discuss in attention
mechanism.
Okay.
So in attention mechanism now attention mechanism main idea is to provide more context to the decoder
to probably do the prediction okay.
So here what we will do is that uh in the encoder first of all I will go ahead and use, let's say over
here.
Instead of just using a simple LSTM, I will be using bidirectional LSTM.
So let's say this is my one LSTM.
This is my second LSTM.
I hope everybody knows what is bidirectional LSTM right?
So here I will just use this.
Okay.
Please focus on the diagram.
And this architecture will completely make sense.
What we are exactly doing.
Okay.
Perfect.
Now over here, you'll be able to see that I will be giving my input over here.
Right.
And this input will also go over here.
Right?
This is my H1 okay.
And I'll say this is my H1 with information going in this direction.
This is my H2 going in this direction.
This is my H3.
Then the information going in this direction and in bidirectional what happens.
We will also be having our H3 same way over here right with respect to time.
Right.
So here you can see h1 h2 h3 h3 right h3 h3 h2 h1.
So here what I will do I will go ahead and write h3.
But my information will go from the reverse h3 h2 reverse h1 reverse right.
So what exactly this architecture is of this architecture is of a bidirectional uh RNN or LSTM RNN.
Right.
Wherein I will take up any text.
Right.
If I say hello.
What's up?
Okay.
What's up.
So in X11 here I'm going to pass.
Hello.
Right.
This information will get passed to both the both the LSTM RNN okay.
But before this is getting passed you know you'll be able to see from the reverse direction your text
will get passed also.
Right.
So here is my second input.
Here is my third input.
So with respect to t is equal to one I'm going to pass.
Hello.
With respect to t is equal to two I'm going to pass what's okay t is equal to.
And what's up at t is equal to three right.
Similarly at t is equal to one.
We are just going to pass this information first of all in the reverse direction.
Similarly as t is equal to two we are going to pass this information.
And at t is equal to uh t is equal to two and t is equal to three.
I'm going to pass this in the reverse direction.
Right.
So this is what happens in the bidirectional uh LSTM, RNN.
And the major aim is that we will also be able to provide the further words context.
Right.
Just imagine this in terms of when you have a larger word.
Right.
So once we do all this things right and you know that this will be interconnected, right.
So here I will be able to get the output.
So let me just go ahead and define this.
So this and this will get combined.
Okay.
This and this will get combined.
Uh over here this and this will get combined whatever output we are specifically getting and this and
this will get combined okay.
Now if we go ahead, right.
As you all know with respect to this, uh, the further information that we specifically have from this
hidden layer, right when it is getting combined here, we will be able to get S zero right, which
is my hidden state.
This is perfectly fine.
Okay, let me do one thing.
Let me just take this diagram little bit below.
Okay.
So I will copy this and let me take it over here completely okay.
Because I need to draw a lot of things on the top okay Okay, so I will just copy this over here.
I'll paste it over here.
Okay.
Now what will happen is that as I get this particular value as s zero okay.
You know that from here I will be getting my output, let's say my output over here H one here my output
is H two.
Here my output is h three.
Now previously what we are doing is that uh in the encoder decoder we were not taking this particular
output.
Right.
We are not taking this.
We just created one context vectors.
Right.
But this output also we can specifically use and we can we can uh, provide this particular output or
context or will convert this output into some kind of context and give it to our decoder.
That is the main idea over here okay.
Now what we will do from this, you know, see further thing what we are going to do over here.
And this will be very amazing.
So first of all, in the decoder, this is the.
So first of all in the encoder this is the chain okay.
So this is my entire encoder here.
I'm actually passing my bidirectional LSTM RNN so that I will be able to also provide the further words
context.
Right now in the next step what we are going to do is that we will create something.
Okay.
So let's say uh, I will just go ahead and combine some information.
Let's say I'm going to probably go ahead and use some notation.
This is my first notation.
This is my second and this is my third okay.
Now in this in this nodes right.
We are going to pass this h one along with h one.
We are also going to we are also going to pass this s zero.
Okay.
So here we are specifically going to pass a zero.
So this will basically go over here along with this.
It will also go over here.
This will also go over here right.
And this H2 tag information.
The hidden state from the output of the H2 will be going over here.
And S3 will also be going over here.
Right.
So this is the first information that is basically getting passed so that we'll be able to convert this
into entire context.
Now what is happening.
The output of this is going over here.
Along with that my hidden state right is also going over here on this particular node right now, the
next important thing that actually happens and let me just denote this okay.
So let's say with respect to words this is my E11.
This is E12 and this is E13.
I will explain you what exactly is the importance of this okay.
Now what we do is that once we have this entire information that is the combination of h one, S zero,
h two and S0S3 and s zero.
Okay, we pass this entirely to a softmax.
When we are saying that, we pass it to a softmax.
That basically means here we are going to train it with a a an softmax okay.
Softmax.
So in short, what is basically happening is that here we will create a feed forward neural network.
And we will try to create a and the output will be using the softmax.
Right.
So here inside this feed neural network we will be giving the combination of h one and s zero which
is my hidden state.
And then this will probably getting trained.
And over here in our output, whenever we try to pass it to the softmax, it is basically going to have
a multi uh multi classification problem statement.
Now with respect to this you will be able to see that here.
We will be getting some important nodes okay.
And what are the nodes or what are the information that will be getting.
It is nothing but A11A11.
And let me just go ahead and write this as A12.
And this will be finally A13.
I'm giving some notation over here right now.
See, see this entire architecture.
And imagine over here, right.
Same architecture over here.
We had our bidirectional right LSTM, RNN.
And from this we used we we generated this 8182 8382.
Right.
And this is what we have actually generated from it.
And this is how the entire generation is basically happening, right?
Uh, we are taking this hidden.
Uh, we are basically taking the output of each and every LSTM hidden output, considering both bidirectional
and, uh, over here, you'll be able to see with respect to this LSTM, we are combining this with
our S0.
Combining basically means together we are giving this to our an.
And then finally you will be able to see that we are passing it to the softmax and we are getting this
particular output.
Okay.
Now this this step, right.
This step that you will be able to see over here where I'm combining all these things.
These are basically called as alignment scores, right.
Yes.
We are aligning all the scores okay.
And uh over here you'll be able to see that this is a, an a this is a entirely an that is artificial
neural network.
Okay.
I'll talk about what is the importance of this.
Then in the next step we basically create our attention weights.
And I'll talk about the importance of this attention weights.
This attention weights are very much important.
C attention weights is all about how much context we really need to give it to the decoder in order
to do that specific prediction.
So here what we do is that whatever output we get from H1 right.
This output, this output, we will take this output and we will combine it with A11.
And here we will be doing a point wise multiplication operation okay.
Similarly over here you will be able to see with respect to A12.
We will take this H2 output.
And we will do a point wise operation with respect to this.
And similarly with H3 output, we will do a point wise operation with respect to this.
And this is what this diagram basically says when we say this point wise operation.
Right.
So here the same thing is basically happening.
Now let me talk about the importance of this A11A12A13.
What is the importance of this.
This is basically used in deciding.
How much context of H one.
I should probably consider how much context of H2I should probably consider right?
How much context of H3I should probably consider this entirely A11A12A13 is created from this particular
an wherein we are combining h one, and we are combining the hidden state right together and then passing
to the softmax activation function by using a feed forward neural network right feed forward neural
network.
So here, according to the research paper, we are just trying to say that how much context you really
need to pass from one stage to the other stage.
Okay.
Now if you are clear with this, this finally, what we will do is that we will combine all these things.
We will take all this values, and then we will again do a point wise operation over here with respect
to plus.
Okay.
And here we are going to compute our context vector.
We are going to compute our context vector.
And the context vector formula is very simple.
It is nothing but CT is equal to summation of I to whatever value it is one to t okay.
A of t and then you basically a of t of I h of I right a of t of I basically means 1 or 1 1213 like
that h of I right h of I is nothing.
But we are specifically multiplying with all the hidden states.
Okay.
So once we get this and this is finally the operation that you will be able to see over here with respect
to plus, right.
This is what you are able to get it.
I know see there are a lot of hidden things that you will be able to see.
And this is where see, your context vector is basically calculating.
And this context vector is coming from a I j a I j is nothing, but it is a softmax activation function.
Right.
And then you also have this e I j.
And that is what I have actually written over here.
E I j e I j is what e I j is nothing but a multiplied by um, sorry.
A activation function on s of I comma h s j right s of I is nothing but your hidden state, right?
And s of j right s of I is nothing but your hidden state, along with uh, h of j, h of J is nothing
but all your hidden states.
Right?
And this is where, step by step we have basically calculated.
But now the time comes is that you need to understand what happens in the decoder.
So once you get this, this c this plus once you add it here you are going to get your context vector
okay.
Or let me just go ahead and draw it downwards okay.
So I will just go ahead and take this.
And this will be your context vector C right.
Now we need this as zero and SC for our first decoder.
Right.
So in the decoder my first node will be S1.
Let's say this S1 this S0 is getting passed over here okay.
Now when it is getting passed we will also have to give our C okay.
Along with the C we will also give our y zero right y zero value which will be our truth value.
And then again over here our softmax layer will be used.
And with respect to this softmax I will be passing it over here and getting my Y1 hat.
Okay.
Now here you'll be able to see that how much context we are getting.
See, all this context is basically getting combined based on proportion, like a one on a one to a
one.
Three is basically saying how much from each one.
I have to probably take the context from which to how much.
I have to probably take the context from x3, how much context I need to take, right?
And here you will be able to see with respect to X1, I am I'm able to get this right.
Then uh, you will be also able to see Y1 over here.
Now what will happen in the next step?
See in the next step I need to pass this information to S2.
Right which will be my second layer.
But now here what will be the information that will be going.
Think over it okay.
What will be the information that it will be going right.
So first of all I need to see over here.
Very simple.
You'll be able to see I'm getting the information of S zero to S one right.
Then if I really want to do it for s two.
Right.
First of all again I need to generate this particular context vector.
Now for generating this particular context vector.
What I will do is that very simply I will instead of passing just from s zero, I will remove the line
from s zero.
And I will take this and pass it from S1.
Now this will go and generate my new context completely, right?
New context completely.
And this new context will then pass to my S2.
Right.
And this context that I'm actually going to generate.
It will be my C2 right.
So this was C1.
Now this will be C2 which I'm actually passing.
C1 was getting passed over here to S1 right along with this.
I will take this y1.
I will also pass this information to my S2.
Right.
So this will basically be my Y1.
Again I will go over here, calculate my y2 hat right y1 and y2 hat.
Then again when it goes to the last timestamp which is S3, this will go over here.
But again what we are basically going to do instead of S1 passing the new information or state information,
we will go ahead and pass from S3.
Right.
So here we will go ahead and pass from here.
And this will probably go ahead and generate our new context.
C3.
Now this C3 is going to get passed over here.
Along with this I'm also going to give it pass Y2.
And finally I will be able to get my output Y3 hat right.
So this is the entire process of this entire research paper.
Isn't it amazing?
Right.
Now this is a beautiful diagram that I tried to explore from multiple research paper and all.
Uh, and got the complete idea about this, how things basically happen.
Right.
And, uh, this is really important for every one of you out there who specifically doing.
But with respect to this, what is the nice thing that is happening every time?
Whenever a decoder is predicting something, it is again going and calculating the new context information
right for every timestamp.
Because of this, we are able to also perform our work with a larger data set.
Right?
And this is the most important thing with respect to this particular architecture that is called as
attention mechanism.
Okay.
Now, uh, one final thing, uh, that I really want to show it to you is that in this research paper,
when we are discussing.
Right, there is also something called as this particular score that is blue score.
And here you can see with the help of a sentence length when we did our own search, uh, of 50, the
blue score of the generated translation on a test set with respect to the length of the sentence, the
result on a full test set which includes sentences having unknown words to the model.
It's it's blue score remained almost constant.
Right.
I did not come down.
Right.
And there is also one more thing over here.
Uh, if you probably see the example with respect to A11, A12, A13.
Right.
So here, uh, with a specific word, what is the translation.
And that is what this specific color shows, right.
Uh, you can probably go ahead and see about this particular research paper, but I hope you got an
idea that how this entire things work.
Right?
I know there are multiple steps.
It is a very complicated neural, uh, attention mechanism.
But this is what is the base of what we will specifically use in transformers, right?
Which is very much important and which is the base of every generative AI models that we use currently.
Right.
So I hope you got this entire idea again.
Just go and see this particular research paper.
This is first of all, my bidirectional.
Then I went ahead and here you can see e of I j we can go and see from here e of I j is nothing but
this.
What we are basically doing where we are using this s of zero and h of I, we are passing it through
an an, and from the softmax activation function I'm getting a1 one a1 to a1 three.
This is basically saying that how much context of H1 I have to probably go ahead and take and probably
create my attention weights.
Right?
So, uh, I hope you like this particular video.
Uh, again, uh, in order to show you a good examples, I also have one more, uh, website.
Okay.
And probably I will also give you this particular link in the website.
Okay.
I will just reload this introduction to attention mechanism.
Let's say this is with respect to our encoder decoder.
So I'm passing this h1 h2 h3.
We go over here.
Then over here you will be able to see that I'm creating one S0 which is my hidden state and context
vector.
And this is basically getting passed.
And this is actually going and generating all the sentence text.
This was basically happening in uh, sequence to sequence encoder and decoder architecture right now
in the next step, how we are basically doing.
So here is your entire diagram how we are calculating this E three.
This s of zero is going over here.
Then you have the softmax then A11.
Then finally we see this multiply attention weights.
Then we create this particular weights of attention weights.
Then we go ahead and create our CT.
That is a context vector right with respect to timestamp C1.
And passing this information that we'll go ahead and create C2.
Here you can see SS1 is going in uh when we calculate the s two right over here.
Then here you can see y two is currently coming up.
Right.
So I will give you all this specific information and you can go ahead and check it out.
But again uh super thanks to this particular block uh, from which is published in 12th May 2021 International
Introduction to Attention Mechanism.
Right.
So yes, this was it for my side.
I hope you liked this particular video.
I will just go ahead and give you this link quickly somewhere.
I'll paste it.
Right.
And this is the link that you really need to refer.
Right.
So yes, this was it from my side with respect to attention mechanism.
I hope you liked this particular video.
I will see you in the next video.
Thank you.
Take care.
Bye bye.