Hello guys.
So we are going to continue the discussion with respect to our Transformers.
Already in our previous video we have discussed till Self-attention, and we have also noted down all
the steps that we specifically use in self-attention.
So first of all, we go ahead and calculate our key uh query key and value vector.
And this is from a learned weights that is w of q w of k and w of v.
Then we calculate the attention score.
Then we calculate our scale scores.
Uh, after doing that we apply a softmax activation function.
And finally we calculate the weighted sum of the values.
And this is just done by my multiplying with the vector v okay.
So overall uh you'll be able to see that initially when we are giving some kind of embedding vectors,
we should be able to get some kind of contextual vectors over here.
Right.
And this contextual vector is based on the context and the and the dependency of the other words that
are present in that sentence.
Okay.
So this is one of the blog, uh, that, uh, you know, has some amazing representation diagram.
Uh, and uh, the URL specifically, if I probably go ahead and talk about the URL, I will just also
show you the URL over here.
Again, I really want to give the credits.
So it is nothing but Jalal Ma.
So his this amazing diagrams.
He is basically covered with the working of the entire transformer.
So you can definitely check it out and you can have a look onto this.
Okay, now my idea is very much simple over here.
What I really want to do is that I want to again show, with the help of diagram what all things we
did.
Right.
So here you can see, uh, this is one of the diagram.
And here in this video I'm also going to discuss about something called as Multi-head attention.
Okay.
So this we are going to discuss about it Multi-head attention okay.
Now initially I have my words over here.
Let's say this is my word one.
What is the word like?
This can be the this can be cat right.
This will be first.
Uh, all my vectors will be available over here.
We are multiplying or we are doing a dot operation with w off key.
And this is how we get our query vectors.
We get our key vectors.
And remember, for each and every word that I have, I get a separate query and key value vectors.
Right.
And similarly separate value vectors.
We separately get all these things right.
And after we get this, what we do, we first of all do a multiplication operation with the query and
the key, right?
Then we divide it by root of d k where we are specifically doing scaling.
Then we apply a softmax activation function.
And finally we do a dot operation with our value vector.
And that is how we get our z value which is nothing, but it is my, uh, attention, uh, self-attention
for the word the right, the vectors that we are specifically getting.
now till here.
I think everybody has followed this.
And step by step, uh, I have actually explained it right.
And we can actually say this one as our attention head, attention head.
Now the idea with multiple multi-head attention, what exactly is that?
We will have a self-attention.
Self-attention with multiple heads.
With multi heads.
Okay.
Now this is fine.
Here you can see I got one kind of vector for the right.
Similarly, can't you think that I can probably go ahead and create multiple attention heads for the
same words.
Right?
So here you can see.
I have taken Thinking Machine as an example over here.
So this is my embedding vectors.
Initially we went and we initialized some other weights like w of o of q, w of o of k, w of o of v
right.
And we calculated this query q of o, k of o and v of O right.
Similarly, you know and after this what will happen is that again that same, same method, uh, it
will probably go to our, it will calculate the score, then it will go to the softmax.
Then we multiply by v and finally we get our attention head.
Attention head.
Right.
Or we get our final vectors after performing all this particular operation.
Right.
So we get our contextual vectors here.
We get it right.
Now this vectors may probably take up dependencies with respect to other words.
Let's say in this example thinking machine is there.
Or.
I can also take another example the cat sat mat.
Okay, if this three is there right now, finally you'll be able to see whatever vectors I get over
here.
This will be with respect to dependencies on this and this.
Right.
Let's let's consider that it has captured with respect to this and with respect to this.
And it has probably given me this.
Similarly, it can go and create another attention head where it will be able to capture some more words.
Importance.
Right.
And for this they will initialize another vectors, uh, another weights like w one of QW1 of KW1 of
v.
And similarly they will be getting vectors like q one, k one and v one.
So whatever vectors we are getting out of this, this may be different than this.
This may be different than this.
Here some more other context importance is basically getting captured here.
Some more other context.
Uh, the importance is basically getting captured.
So this can actually become my Z zero.
This can become my Z one.
So similarly I can go ahead and create multiple attention heads so that.
What will be the main aim of this particular multiple attention heads?
It will expand the model's ability to focus on different positions.
So here I will go ahead and write.
It expands the model's ability to focus on different position of words or different position of tokens.
This is what it says.
Yeah.
So here you can probably see right.
Z zero may contain a little bit more information of every other encoding, but it can be dominated by
one word like sat right here.
Uh z sorry, Z zero may be like that.
Z 1st May be like it may focus more on different words.
It can be dominated by word like mat.
So if we have all this kind of combination right then what will happen is that we will be able to get
more and more information to our models.
Right?
So here this attention layer gives uh provides multiple representation subspaces okay.
And um in case of multi-head attention, we have not only one but multiple set of key value pairs.
Multiple set of key value pairs.
Right.
And each of this set is randomly initialized initially.
Then after training is set, is used to project the input embeddings into a different representation
shape subspace okay.
That basically means it will take a vector and it will try to convert them.
So overall you will be able to see that by using this we may get this kind of diagram.
See.
So here.
With a different set of q k v we will be able to get this attention z zero with a different set of q
k v.
And along with that obviously w w of k w of q w of v, this weights will be also different.
Similarly we can go ahead and calculate this attention which will capture other important thing in the
token right with respect to the positional encoding.
And then here you will be able to see different different captures uh different different uh outputs.
Right.
So this is the entire information about Multi-head attention.
So here we are just not focusing on providing one set of vectors with respect to this.
But we'll have multiple set of vectors, and each one of the other vectors that we are getting right,
which we also call it as attention head.
It will have importance with respect to other other tokens that is available over here in the sentence.
Okay.
So yeah, this was it about Multi-head attention.
Uh, the concept is very similar.
Whatever steps you are doing, instead of just getting one one attention head, we will try to get multiple
attention head wherein we will try to train it with multiple key.
Q uh, that is queries key value pairs by initializing W1W uh w one of QW1 of KW1 of v right weights
of q, k, and v, and as many number of attention head will be there, that many number of weights
can be parallelly done right.
So this was uh more about the multi attention head.
So uh multi head attention and uh yeah in the next video.
Now here you can see that what all things we have specifically discussed.
Right.
If I go up.
Oh my God I have written so many things for you all.
So here you can see after performing the self attention, I need to provide all this information to
my feed forward, uh, neural network.
Right.
But here you can probably see now how my diagram will change by multi-head attention.
Once I give how right after this self-attention, I will be able to get like this.
Multiple heads of vectors.
Right.
So this will be my attention.
Head one zero.
Attention head one.
Attention head two.
Attention.
Head three.
Attention.
Head four.
Attention head five.
Right.
Similarly for this.
What?
I may get other attentions.
Now all this I need to pass it to my feedforward neural network.
So obviously the feedforward neural network will consider this as just one encoding.
So we need to combine this.
And that is what we are going to see that how we are just going to pass all these things to a feedforward
neural network, and how the forward and the backward propagation is going to take place.
That is what we are going to see in our next video.
So I hope you like this particular video.
I will see you all in the next video.
Thank you.