Hello guys.
So we are going to continue the discussion with respect to self-attention.
And uh, already we have seen the basic architecture of transformer.
Now we are going to deep dive into the self-attention layer and we'll try to understand how exactly
this works.
Now this session, in the upcoming session, we are going to probably discuss a lot about self-attention,
where we'll also be solving like how this entire self-attention layer actually works.
Uh, All the mathematical intuition behind it.
And uh, through this, I am definitely sure that you'll try to understand or you'll get to understand
how the contextual embedding is specifically created.
Okay.
So again, uh, the idea behind self-attention, what exactly it is.
So first of all, here you'll be able to see that let's say I have the self-attention layer okay.
Now with respect to this self-attention layer, whenever we give any vectors.
Right, let's say the word is like the cat sat, right.
Let's say if this is my sentence that I'm actually giving.
Okay.
Now with respect to this particular sentence, you will be able to see that when I am giving this specific
sentence, this words that you will be seeing one by one, which will I'm actually giving.
This will first of all get converted into vectors.
So let's say I'm converting this into vectors.
Vectors.
And here also we will convert this into vectors.
Let's say that right now the default vectors for the is like 100 for cat is 010, and for SAT is 001.
Okay.
Now as soon as we pass through this embedding layer or sorry, not embedding layer, it's the self-attention
layer.
Okay.
As soon as we pass through the self-attention layer, then from here, we should be getting the output.
Now, the output that we should definitely be getting over here will be another vector, will be another
vector.
And this vector that we specifically get.
this is our contextual embedding.
And why do we say this as contextual embedding.
Because.
Contextual embedding because see here we are going to take the importance of different tokens right
in the input sequence.
And we are going to create this particular vector.
So right now if I just consider this as an example, let's say this vectors gets converted as .8.3.4.
Then this particular vectors gets converted to like you can just say that as .4.6.2.
Okay, I'm just saying as an example.
And this particular vector gets converted as .3.4.8.
Now what is the importance of getting this specific vector.
It is very much uh, it is just simply you can consider that here, the entire uh, here the self-attention,
what it is basically doing is that it is also specifying some importance of the other tokens or other
words, and it is displaying or it is creating this particular vectors.
Okay.
So first of all, let's see this entire definition.
So here it shows self-attention is also known as scaled dot product.
Attention is a crucial mechanism in the transformer architecture that allows model to weigh the importance
of different tokens in the input sequence related to each other.
So here we are just not going to keep our embedding vectors fixed.
Instead, we are going to change all these embedding vectors into contextual embedding vectors based
on the importance of all the other words.
Right.
And this is what we are doing.
And why do we do this.
Because C with respect to different different sentences different different paragraphs.
Right.
All the paragraphs, all the sentences will have different context.
They may be importance of some other word.
One word will be dependent on the other word itself.
Right.
And because of this right you will be able to see that will be able to efficiently create applications
like, uh, language translation, right.
Language translation and.
All right, language Language translation or text summarization.
This kind of use cases, we will be specifically be able to create.
Now how to convert this to this.
Right.
And the entire self-attention works the on the concept of scaled dot product attention.
Right.
So this is the entire operation that it specifically takes.
Uh, here, one by one we will be discussing what is this key q k v.
Right.
These are some vectors.
Why do we use this specific vectors.
But at the end of the day this is the entire operation that we are doing now.
What we are going to do is that we are going to see that how the self-attention layer is going to convert
this vectors to this vectors, right, which is a contextual embedding.
But here, just to give an idea, this vectors has basically been changed because considering the importance
of different tokens.
Okay.
Now let's go ahead and let's talk about self-attention in detail.
So let's go ahead and discuss about self-attention in much more detail.
Right.
And uh uh, over here our main aim is to convert this particular vectors into contextual embedding vectors.
Right now.
See one thing converting this particular word into a vector right into a dense vector like this.
Here, for this particular scenario, we can use embedding layers, right?
So in embedding layers what happens based on the dimensions.
Right.
We will be able to get a fixed vector for each and every word.
Right.
Like for the the cat sat right.
We can actually get the fixed vector.
But getting this contextual embedding vector based on the context of other words, it depends on the
entire sentence, or it depends on the data set that we have.
Right?
So how with the help of this sentence or data set, we will be able to convert from this vector to this
vector.
Right.
So for this there are multiple steps that we really need to follow okay.
So let's talk about this specific step.
And that is where this entire scaled dot product attention will come into existence.
So the first important thing that we really need to do is that we need to create or we need to derive
three important vectors.
So I will first of all talk about this inputs.
So in the first step we need to derive our three important vectors.
One is queries.
Then I have keys.
And then we have something called as values.
Okay.
So queries keys and values.
Right now what is the importance of this query.
Keys and values that we will try to understand Okay.
For every word, if you are trying to convert that into a contextual vector for every token, we specifically
create a model.
Right.
And this model will be computing this three important vectors.
One is queries, one is keys and one is values vectors.
Okay so guys now before we go ahead let's go ahead and understand or see the basic definition of this.
Queries keys and values vector.
And what is the importance of this okay.
So here you'll be able to see first we will understand about query vector.
Now what exactly is a query vector?
A query vector represents the token for which we are calculating the attention.
Let's say for the first word that I'm actually passing the.
If I really want to probably calculate, uh, the attention for this, which is the output vectors,
right.
Which we say it as contextual embedding vectors also.
Right.
So in order to calculate this we will specifically specify this vector as our query vector okay.
Then they help to determine the importance of other token in the context of the current token, right?
So in this particular token, the importance of the other token will also specified the final vector
that will be able to get.
Now what is the importance over here.
Focus determination.
This is the first importance.
Queries help the model decide which part of the sequence to focus on for each specific token, because
there will be multiple token in a sentence.
By calculating the dot product between a query vector and all key vectors, the model assesses how much
attention to give to each token relative to the current token, right?
So what we do is that we multiply this query vectors or or we we do a dot product or not say multiply.
We do a dot product of query vectors to key vectors.
And this helps us to determine how much attention to give to each token related to the current token.
Right.
So if I have the if I have cat right.
If I have set, let's say this is my three words.
So this initially becomes my query vector for this particular.
If I'm calculating the attention for this, and the final vector that I'm actually going to get is that
when I multiply this query vector with the key vector, right, I will be.
The model will be able to assess how much attention to give with respect to all the other tokens that
are there relative to this particular token.
Right.
So that is what query vectors will talk about.
Right?
And as we see more examples, you will be seeing that I will be solving a step by step like how this
entire query vector will be calculated, key vectors will be calculated or not.
So second important thing is that with respect to contextual understanding, queries contribute to understanding
the relationship.
This is very important relationship between the current token and the rest of the sequence.
This is the most important thing.
Query vectors actually contribute to understanding the relationship between the current token and the
rest of the token, which is essential for capturing dependencies and context.
Okay.
Then coming to the key vectors, what exactly is key vectors?
Key vector represent all the token in the sequence and are used to compare with the query vector to
calculate the attention scores.
Now what exactly key vector is?
It will have all the tokens that is probably there in the sentence, right?
And it is basically a query will be compared to this key vectors, uh, to calculate the attention score,
what exactly attention score is, we will discuss about it and how to calculate it.
Also will discuss about it.
So let's talk about the importance of key vectors.
Keys are compared with queries to measure the relevance or compatibility of each token with the current
token.
This comparison helps in determining how much attention needs, how much attention each token should
receive.
Okay.
Information retrieval.
It also plays a critical role in retrieving the most relevant information from sequence by providing
a basis for attention.
Mechanism to compute similarity score.
So what we do is that we take a query vector.
We we we, we can directly query from this key vector itself.
And based on similarity scores we get the most relevant information based on the dependency of all the
other words also.
Okay.
And finally, uh, the value vectors, value vectors holds the actual information that will be aggregated
to form the output of the attention mechanism.
Okay.
So value contains the data that will be weighted by the attention scores, the weights, some of the
values from from the output of the self-attention mechanism.
So we will discuss about all these things uh, as we go ahead.
But I really wanted to put a basic definition of key, uh, query key and value vectors.
Okay.
Now let's take one specific example.
And by that you will be able to understand how we are going to create this query key and value vector
okay.
So let's say I have an input sequence.
So let's consider this is my input sequence.
And the input sequence is nothing.
But it is having this word the.
Cat sat.
Okay, so this is my input sequence right now.
Let's say, uh, we are just going to take the embedding dimension by the embedding layer.
So the embedding size I'm going to probably take it as four.
That basically means for every word we will convert this into four vectors.
Okay.
And uh uh, even for the q k v vectors that we are also going to compute, let's consider this also
as four dimension okay.
So I'm just going to consider all this as four dimension.
So here you will be able to see that how we will be able to generate this k q v.
We will try to see okay.
So first of all the first step over here is something called as token embeddings okay.
Token embeddings basically means whatever sentence we have we will try to convert that into some kind
of vectors.
So let's say I will be writing e of the the embedding of the.
Okay.
Let's say that here there are three words, but I am considering there may be four dimensions.
So I will just go ahead and mention hey though is represented like this.
Okay then I will be having e of cat.
Let's say cat is represented something like this.
And then I have E off set one comma, one comma, one.
Okay, so this is how the entire tokens is basically represented for this particular words.
And these are my fixed vectors okay.
Now the second important step is that see see guys one very simple thing that you really need to understand.
If I want to convert one vector into a context vector okay.
By using this self-attention.
See this is my one vector, right.
Let's say if this is 100I need to convert this into .8.2.3.
Right.
This vector is basically getting created based on the context of all the other words also.
Right.
So there is a huge dependency on the entire sentence.
Right.
Because with respect to different different data set there will be different different sentence.
And based on different different context different different dependencies.
This vector will keep on changing.
Right.
So we should definitely come up with a model wherein I will say this model as self-attention.
Self-attention where I give one kind of input and I should be able to get my contextual embedding based
on the sentence.
This will have all the information about all the other words also, and that is why the reason we are
using this query query key vectors value vectors.
That is the reason for that, so that it has some information about the other words and sentences also.
Okay.
So now once we have done the token embedding.
So that is what we are doing now we will try to create this entire model.
And I will also show you that how this model will be trained.
The second step is something called as linear transformation.
Now what this linear transformation basically does.
Okay.
So here you will be able to see we will create query key value vectors by multiplying.
See I am telling you we have to create a model when we create a model right.
Definitely a model needs to have some kind of weights, because if I just want to convert a word into
a fixed vector, we can directly use word two vec or some kind of embedding layer.
Right.
And uh uh, but here our scenario is something different.
We need to create a contextual embedding based on the other words.
And we need to do it in the runtime.
Right.
So we have to probably train a model which in which it will be able to take this particular fixed vector
and convert that into a uh, contextual embedded vector.
Okay.
okay.
So for that we perform in the second step that is called as linear transformation.
Here we create query key and values by multiplying by multiplying the embeddings by multiplying the
embeddings by learning by learned weights sorry by learned weights.
matrices.
I will try to denote this matrices as w of q, w of k and w of v okay.
Weights of queries.
Weights of keys and weights of values okay.
Now what does this basically mean.
So this is very simple.
Let's say if I have a vector let's say this is my word of cat.
Okay.
I will take this cat vectors.
Let's say it is 100.
I will probably take a weight.
Let's say this is my weight w of Q right.
If I take this vector and do a dot operation with this, I should be able to get my I should be able
to get my.
Key vector query vectors.
Okay.
I should be able to get my query vector.
So this will basically be my query vectors.
Okay.
So this is one cross three.
This is three cross three.
If I do a dot product it will be nothing but one cross three okay.
Similarly, if I consider about my another weights in order to compute the key value or the key vectors,
I will multiply with a with a weight matrix, or do a dot operation with respect to weight matrix for
this particular um, for this particular vector.
And I should be able to get my key vectors.
Okay, so this basically becomes my key vector.
And similarly if I go ahead and consider my another weights that is nothing but values w of v, I should
be able to get my v vectors.
Okay, so this is what we are doing.
So we create a q, k and v by multiplying the embeddings.
This embeddings will be done as a dot product.
Dot product with respect to this particular vector.
Then this vector and this vector okay.
And then we'll be able to get this k q v.
But understand here we have written learned weights.
That basically means initially we'll go ahead and initialize the weights.
And later on with the help of back propagation this needs to be learned to get the exact right kind
of key, uh, query key and value vectors.
Okay.
And that is what we are specifically doing.
Now let's if I really want to show you one practical application or example, let's go ahead and do
one thing.
Okay.
So here is my token embedding which I have actually considered.
Now let's consider my I have initialized w of q I have initialized w of q, w of k, and w of v is equal
to identity matrix.
So identity matrix.
In this particular case if I'm taking three cross three this will have all the diagonal elements as
one and remaining all as zeros okay.
So this can definitely be an identity matrix.
So let's take this identity matrix.
Let's probably initialize it now uh, you'll be able to see that if I really want to compute the query
of the right.
So the is nothing, but, uh, you'll be able to see that.
Uh, right.
So in this particular scenario, if I just take this, the the is nothing but this 1010.
Okay.
So here I'm going to basically write 1010.
If I do a dot operation with 100010001, right?
So in this scenario, if I do a dot operation with this entire operation, uh, then I know with respect
to the identity matrix I'm going to get the same value similarly, uh, key of the right.
If I really want to find out the vectors again, I will be able to get the same thing.
Then value of the again, I'm going to get the same thing right.
So in this scenario you will be able to see that with respect to the key query and value.
If I am multiplying this with my or if I am initializing and identity matrix, I'm actually going to
get the same value.
Okay.
So this is the first step right?
Similarly, if I go ahead and compute uh, so this is my first thing where I have actually computed
the key of the K of sorry queries of the key of the and V of the right.
It will be nothing, but it will be 1010.
And this how we have actually computed?
Very simple.
We have taken our vectors, the embedding vectors, and we have multiplied with the identity matrix.
Okay.
Now we have done the dot operation.
Now similarly for the other word let's say the other word is nothing but q of cat and k of cat and V
of cat.
It is not compulsory that you need to initialize it as an identity matrix.
You can initialize anything.
But just for my computation, I've made it like this right now.
Here you'll be able to see for Cat what exactly it is.
For cat it is 0101.
Okay.
So here I'm going to go ahead and write 0101.
Right.
So this basically becomes my query, uh, sorry query vectors, key vectors and value vectors.
Right.
For cat.
Similarly, the third one that we are basically going to compute is q of sat, k of SAT, and V of sat.
So in this case we'll be able to see that I'm going to get 1111 right.
So all this particular vectors is there.
Perfect.
Now once we have actually done this this is the first step, right?
Uh, the first step is basically to get the token embedding.
Then we create uh q k v by multiplying the embeddings by learned weight matrix.
Right now it is still not learned.
We have initialized it, but with the help of back propagation this will keep on getting learning okay.
Now the third step that we are going to basically do is compute attention scores.
We compute the attention scores.
Okay.
Now, as I told you, how do we compute the attention score?
It is very simple.
And over here we have told queries help the model to decide which part of the sequence to focus on for
each token.
By calculating the dot product between a query and all the key vectors, the model assesses how much
attention to give to each token.
For doing this, we need to do a dot product of q query vector and all key vectors, and that is how
we calculate our attention scores.
And why do we do that?
It is very much simple to provide.
See why do we do that.
Because the model assesses how much attention to give to each token relative to the current token.
So with relative to this particular token that is the how much importance I need to give to Cat and
SAT tokens.
Okay.
And that is how we basically go ahead and compute our score okay.
So for computing the score right.
So let's say for the the word if I go ahead and compute the score, the score will be nothing but query
of the and key of the.
So first of all we'll go ahead and compute this.
So I know my query is nothing but 1010.
And then if I want to do if I want to multiply with k right k is also 1010.
So I will probably take a transformation of this.
Right.
Because if I really want to do the dot operation, my if my q vector is like this 1010, I will probably
do a dot operation by taking the transformation of this.
And then, then and then only I will be able to do the dot operation.
Right.
So once I do this, I will be able to get the value as two right.
So one multiply by one plus zero plus one plus zero right.
So finally you will be able to get two.
So for the with respect to query of the and q a key of the when I do the dot operation I'm actually
going to get the value of two.
Similarly, if I go ahead and calculate the score with respect to query of the and key of Cat right.
If I do this particular dot operation, because we need to understand the context with respect to the
other words also.
Right.
And here is what I'm actually going to get it.
So I'll be writing 1010.
And I'm going to do a transform operation with what what is key a key of cat.
See what is key of Cat.
It is nothing but 0100101 and this will be the transformed value.
Now, if I try to go ahead and find out, I will be getting the output as zero, right?
Similarly, if I go ahead and calculate square off score off query of the with respect to key off set
right key vectors of sad.
When we do the dot operation of q off the and key key vectors of SAT here, what we are going to get.
again I'll go and write 1010.
And again what is key of SAT.
It is nothing but one comma, one comma, one.
This will be the transform value.
So here also we are going to get the value as two right.
So this is how we compute the score values right.
And why we are doing this.
Again why we are doing this.
It is very simple over here.
It is basically to assess how much attention to give each token related to the current token.
Okay, so we have got some kind of values over here.
And this will actually help us to understand that.
How much context how much uh, how much focus should I really need to give with respect to the other
token, with respect to the current token that I have actually considered?
Right.
So here I have actually got some values scores.
Right.
202 okay.
Similarly we go ahead and compute the for the token cat.
Now for the token cat we need to do the same thing right.
We need to do the same thing.
That is go ahead and compute the score.
Now the first score will be Q of Cat with respect to key of the right.
Now I have to probably consider whether there is a dependency, how much dependency I need to keep with
respect to this.
Right.
So here you'll be able to see that I will be able to compute it.
Now here I will have 0101.
Right.
And this time I'm going to do a transform operation.
With what dot transform operation with.
Basically I need to transform this and do a dot operation.
Right.
So here you'll be able to see if I do the computation.
Um I'll be getting the value as zero.
Similarly, if I go ahead and compute the score of q cat with cat.
Right.
So here it will be nothing but 0101, which will be a dot operation of 0101 transform.
Right.
And here I'm actually going to get the value as zero.
Then again if I go ahead and compute this q of cat with k off set right.
In short I'm just doing a dot operation right.
So here I will be having 0101.1111 transform.
And again I'm going to get the value as two okay.
Right.
So just by seeing this word you can just understand SAT is important for Cat.
So that is the reason why I'm getting equal values for this particular score for uh cat with cat and
cat with sat.
Right.
This kind of dependency is definitely there.
And I've definitely used identity matrix.
But this all weight parameters will also get trained right.
Then similarly for the token.
For the token set.
For the token set how do we go ahead and compute it.
So we will go ahead and write the score.
And here you'll be able to see Q of sat.
K of sat right.
So sorry K of the we need to do the dot operation with first of all the queries of the query vector
offset with the dot uh, with the key vectors of the right.
So here I will be getting one comma, one comma one, which is my query vectors of sat dot operation
of 1010, which is my transform operation.
And here I'm going to get two.
Similarly, if I go ahead and compute this with SAT and K of Cat, I will be getting two over here.
You can go ahead and do the computation.
And then here you will be able to see K of SAT.
I'm also going to get a four right.
So this is in turn the values that you will be able to see.
Right.
And this is my third step.
So initial step what we do is that first of all we go ahead and compute.
We take a token embeddings.
Then we do linear transformation and probably create this k q vectors by initializing by by a learned
weights matrices of w of q, w of k and w of v.
In this scenario, I've just considered this as identity matrix, so let's consider it.
Okay then I've shown you that.
How do we probably go ahead and compute this right when we do this particular dot operation, dot operation,
dot operation.
So I'll be getting the same values and this is what I get it finally.
Then we compute the attention scores.
And this attention score is for a different purpose just to how much focus I really need to give on
the other tokens.
Right.
So here I get all my token score values right then Then my next step after this is something called
as scaling.
Okay.
Now what exactly is scaling okay.
See whatever scores that we are getting right.
Whatever score that we are getting over here.
Right.
You'll be able to see that I'm getting some values like 2 to 4 right.
0 to 2 and 202.
Right.
And you can clearly see that.
Why uh why this scaling needs to be done.
Why why we we are specifically performing scaling.
Okay.
First of all I'll just make you understand what does exactly scaling mean okay.
So scaling what we do is that we take up we we take up this course.
And scale.
Scale down by dividing.
By dividing the scores.
By the square root of dimensions of key vectors.
Okay.
So in our case our dimension of the key vector is four.
So if I do square root of d of k it is basically going to become two.
So guys now let's understand why do we really need to consider scaling okay.
So scaling.
So let me just write it down.
Scaling in the attention mechanism.
In the attention mechanism is crucial to prevent.
The dot product.
Dot product from growing too large.
From growing too large.
Now when we say dot product.
Dot product between what?
Dot product between the queries and key vectors that we are specifically done with respect to this right
now.
Why see why we need to prevent this dot product from growing too large?
Okay.
And in most of the research paper it is basically given to ensure ensure stable gradients during training.
So let's go ahead and understand this with an example okay.
And uh if we don't scale then what kind of problems we will be facing.
And uh, as you know that if DK, dk basically means, uh, the dimension of the key vectors, if it
keeps on getting large here, you'll be able to see that we will be facing two different kind of problems.
Okay.
And as the dimension keeps on getting larger, if we don't try to scale with this value, and if the
dimension is getting larger and larger, right.
We face two different types of problem.
One is gradient exploding right here.
You'll be able to see that during the back propagation this gradient will become very, very large.
Uh, and because of that, your training will not be stable.
Right.
And the second one is something called as softmax saturation.
So when I say softmax saturation, this basically means, uh, the softmax function has become saturated
when most of the attention weight is assigned to us, just a single token and other tokens for other
tokens.
The weight will be assigned to near to zero.
The weight will be approximately equal to zero, so we will discuss about it.
Don't worry, I will try to explain you with an example.
Okay, so two problems we are going to face over here.
One is gradient exploding and the softmax saturation.
Okay.
Now let's uh see one example.
Let's say that I am going to consider three vectors.
One is k a q.
This is nothing but two three, four, one.
And let's say I have another vector KK1 right 1010.
And another key vector that is nothing but 0101.
Okay.
Now let's uh without scaling, let's do the dot product okay.
When we do the dot product.
So here what I will do I will be using k along with k one of transpose.
So if I do the dot operation how it will be two multiplied by one plus three multiplied by zero plus
four multiplied by one, four multiplied by zero.
So here overall I'm actually going to get this value okay okay.
Uh one multiplied by zero.
It should not be four multiplied by zero, but one multiply four multiplied by one and one multiplied
by zero.
So overall, if you see two plus four which will be six okay.
And similarly if I do the dot operation with the next key vector over here.
So this will be two multiplied by zero, three multiplied by one, four multiplied by zero, one multiplied
by one.
Okay.
And if you see over here zero plus three plus zero plus one, it is nothing but four.
So with respect to this two dot operation, you know, initially my key values were this much okay,
key vectors were this much, but now I'm after the dot operation I'm getting six and four.
Okay.
Now, you know, after after the scaling, let's say that I'm going to apply a softmax because that
is what happens here also.
So first of all we need to apply scaling.
And then after that we'll apply a softmax activation function to get the final contextual embedding
okay.
That is what happens in the next step.
So let's say here I have got two scores.
One is the score of six.
The other one is the score of four.
Okay.
Now let me do one thing here.
We are not going to apply any scaling.
Scaling I'll just go ahead and write.
You'll understand the problem.
What problem you are going to face.
Scaling not applied.
So if I don't apply the scaling you will be able to see over here what I will just go ahead and apply
my softmax.
Now softmax will get applied to both this number six comma four okay.
And uh here uh when we are specifically applying the softmax.
Okay.
So here you will be able to see that if you remember about the softmax formula, it is nothing but e,
let's say e to the power of six divided by e to the power of six plus e to the power of four.
Okay, comma.
E to the power of four divided by e to the power of six, plus e to the power of four.
When we specifically apply on this two.
Okay.
And.
Here if I just go ahead and see to this.
So let's say e to the power of six.
And here I will take E to the power of six as common.
And I'll write one plus e to the power of minus two.
This is how we can represent it.
Similarly, if I go ahead and write E to the power of four, and if I take common as e to the power
of four, I'm going to get E to the power of two plus one.
Okay, so from here, if we do the forward calculation, since we are applying a softmax activation
function, this will be nothing, but it will be.
Or let me do one thing.
Let me just go ahead and write one divided by e to the power of six, E to the power of six will go
off, so it will be e to the power of one, plus e to the power of minus two comma.
And this will be one plus e to the power of two plus one.
Write this e to the power of four, E to the power four will get deducted.
Now, once we are calculating this, uh, here, you will be able to see that, uh, the final output.
If I do the calculation with the help of calculator, it is nothing.
But it is approximately equal to 0.88.12.
Okay, so what has happened is that when we applied this softmax activation function on a score of six
comma four, which we got as a product of query vectors with the key one vector of this specific word
of any word.
So here we had this key one vector v.
Here we had this key two vectors right.
And when we do this when we did this dot product we got six form of four.
We calculated the score here.
We did not apply any scaling.
And finally you could see that we could get a soft when we applied a softmax activation function here
you got a value of 0.88 and 0.12 okay.
Now what does this value basically mean?
First of all this value actually means that.
most of the attention weight, most of the attention weight, okay, is assigned to the first key vector,
right?
Because the value is higher.
This is higher than this, right?
And very little.
Very little to the second.
Second vector.
Okay.
Now let's understand what does this sentence mean.
Most of the attention weight is assigned to point uh to the first key vector that is .88 and very little
to the second vector okay.
Okay.
And let's understand see one of the property I think you should be knowing if you have knowledge of
deep learning.
Right.
So one of the property of softmax is that whenever we apply a softmax to two numbers right.
Two scores are there.
Here you can see the difference between the score is very less six comma four.
But with respect to the output here you are able to see that there is a huge difference.
0.88 I'm getting to the score six and 0.12 I'm getting to the score four.
Right.
And similarly, if I try to apply a softmax activation function on a number like ten comma one.
Now in this case ten comma one, you'll be seeing that there is a huge difference.
Right.
And this values are basically coming from the dot product of q and k.
Right.
right.
And here you can see that I have not applied scaling.
Right.
So one of the property, if I do not apply scaling whenever I try to find out the softmax for this,
again, if I try to find out the output, you will be able to see that one value will be like 0.99 and
one value will be like 0.01.
Now in this scenario, again this value is very high.
This value is very less okay.
You should understand all the scores will be specifically used for updating that particular weights
that we are using, right?
The weights like w of q, w of k, w of v.
Now when we are doing back propagation right this small value, just imagine when this small score comes
into picture, obviously we are going to face a problem of gradient or sorry softmax saturation.
Right.
So there will be one problem that softmax saturation where my gradient is not going to get updated more
because the value that I'm actually going to get is becoming very small.
And during the back propagation, I can also see this term as vanishing gradient problem.
Right.
Vanishing gradient problem where my weights are not getting updated.
Vanishing gradient problem.
So this is the property with respect to softmax.
See right now I did not do any scaling.
And I got this kind of value like 0.88 .12.
Even though my value of scores was six comma four.
Not much difference, hardly a difference of two.
Similarly, if I do a dot product of query and key vectors, if I'm getting a very high value over here,
if I apply the softmax activation function, just try to apply the softmax activation function.
You'll be able to see that I'll be getting a very high value at one of the vector, and the other value
will be very, very slow.
Uh, very, very small.
Right.
So this scenario we usually face when continuously, you'll be able to see when your large, when your
dot product, when your dot product that you specifically do is coming as a large values.
Okay.
Now let me do one thing.
Let me just show this by using or by scaling this value okay.
Okay.
So now I will just go ahead and write with scaling.
With scaling will try to apply the softmax activation function.
Now here you can see with scaling.
First of all I will just go ahead and compute my scaled dot product.
Okay.
You know that compute scaled dot products right now obviously you know my score.
My score one.
Right?
My score one was six and four.
Right now for this six and four.
What we do first of all we if I really want to scale it right I need to divide it by root of d k right.
I need to divide this number by root of d k.
So I will take each and every number.
I'll divide by root of decay.
Route of decay is nothing, but it is root of four, which is nothing but root of two.
So if I divide six by two and four by two, I'm going to get what?
Three comma two.
Understand?
My scores were initially six comma four, but now I have scaled it with this value.
That is root of decay.
And then I'm actually getting three comma two.
Now if I go ahead and apply softmax on three comma two, you just go ahead and do this calculation how
much you will be getting okay.
So again I will do the calculation over here E to the e to the power of three e to the power of three
plus e to the power of two comma E to the power of three.
Sorry, e to the power of two.
And this is e to a power of three, plus e to the power of two.
Similarly, uh, if I go ahead and probably do this computation.
Okay, uh, you can take what all commons you specifically want, let's say here I'm going to take a
common off e to the power of three divided by e to the power of three.
I will take it as common.
This will be one plus e to the power of minus one.
Okay, comma, E to the power of two, e to the power of two.
I will take a common.
Then here I will write e to the power of one plus one.
Okay.
And this will deduct it.
This will get deducted.
So finally what I'm output I will be getting.
If I do the computation.
It is nothing but 0.73 comma 0.27 right now Before, when we did not apply the scaling, you could see
that I'm getting a value as 0.88.12.
The difference is very huge.
But now here you can see there is a less difference.
So when we are now back propagating right, we will not face that problem of vanishing gradient problem.
Right.
And here you will be able to see the output or this is also called as an attention weights.
This is my attention weights.
This Attention weights.
Let me write a point over here.
Here.
The attention weights.
Here are the attention weights are more balanced.
Are more balanced compared to the Unscaled case.
So if we are scaling now, we are getting a proper balance.
Before the difference was huge.
I know here also you can see the difference is there, but at least in the back propagation with respect
to the other vector that we have.
Right.
And when we try to update the weights, at least we won't face the vanishing gradient problem okay.
So this is the most important thing right?
And because of this, if I just try to write down the summary for you okay, let me just note down all
the summaries point that I really want to specify over here.
Because of this, there are two important things that you need to take a note of.
Okay.
So the first thing is that here you are stabilizing.
You are stabilizing the training.
Scaling prevents extremely large dot products, which helps in stabilizing the gradient during back
propagation, making the training process more stable and efficient.
And second, you are preventing saturation by scaling the dot product, the softmax produces more balanced
attention weights.
Now there is a more balanced attention weight.
This token is also having some importance.
This token is also having some kind of importance right before, even though we give any kind of values
over here, the gap was very high when we were trying to create the softmax.
And that is what is basically like creating a stable gradient, right?
We specifically divide by this number.
Now you may be thinking, Krish, how do it become for for this particular number.
Right.
Why we are specifically dividing by root of d of k.
So guys for this there is a there is a there is a one simple research with respect to variance.
Okay.
Whenever we do the dot product, you know as we keep on increasing the dimension the variance, the
dot product variance will also keep on increasing okay.
So as this was keeping uh, and there is a research for this guys uh, as this keeps on increasing Okay.
Here you will be able to see that if we divide by root of d k.
This is making sure that even though the dimension increases, even the dimension is two three, right?
The variance for that particular dot product will be almost same.
That is what this specific number is all about.
Right.
And this is a small research in variance.
Uh, I just have mentioned it so that you can refer it whenever you want.
But I think for our understanding, this much is more than sufficient.
Okay.
So because of this scaling, one more point that I can specify over here is that scaling ensures the
dot product are always kept within a range that allows softmax activation function to operate effectively,
providing a more balanced distribution.
Okay, so if I go again back with respect to all the steps.
So my fourth step was scaling.
And after scaling, uh, we will just try to understand what is the next step okay.
But here I've shown you multiple examples with respect to scaling.
Right.
So if I just go ahead and show you what all things we discussed till now, first of all we go ahead
and you know, do the token embedding.
And then with respect to the token embedding we do the linear transformation where we are going to find
out three important vectors query key and value pairs.
I've told you about the discussion with query key value pairs.
Why it is important.
Then we go ahead and compute this.
Then we compute the attention scores.
And this attention scores also play a very important role.
Then we probably do the scaling part with respect to this okay.
Now once this scaling is done right, once we specifically do the scaling, then we calculated the weighted
sum of all the values okay.
That is what the next step is all about right.
And I know this may be somewhat confusing, but it is always good that we try to understand things.
Okay.
So let me go back to my previous problem statement here.
If you remember, we had calculated this course, right?
All this course right.
For compute attention scores for the the score of query of the with respect to key of the was 202.
Like uh, this course we have actually computed.
Right.
So let me do one thing.
Let me just go ahead and apply softmax for before that we will do the scaling okay.
And then we will compute.
And I will take only one word for this okay.
Or one word I'll just try to show you with respect to one word.
And then you should be able to understand.
So the fourth step that we are specifically going to do is nothing but scaling.
I've shown you one example of scaling.
So scaling what we will do, we will try to divide by root of d of k okay.
In this case it is four.
So this will basically become two right.
Root of four which will become two right.
Then we are just going to calculate our scaled score.
So if I go ahead and calculate with the query off the width key of the right, we had this value initially
two.
And we are dividing this by two.
So I'm going to get one right.
Similarly if I go ahead and calculate our scaled score with q of the and k of cat.
So here we are just going to divide by number zero by two which is equal to zero.
Then I have my scaled score.
Then this is nothing but q of the comma K of SAT.
So initially this was two.
So I'm going to divide by two which is nothing but one.
Now once I do this division you'll be able to see that again.
This value has not decreased right.
So similarly all the scaling will be done.
Similarly you will be seeing that scaling will be done.
Similarly scaling will be done for all other tokens.
Perfect right.
Now after this, as I have already told you, the next operation will be specifically apply softmax.
Now, when I apply softmax here, you'll be able to see that.
And this softmax will be applied to the, to the, uh, scores that we have actually got the scaled
scores.
And then whenever we apply the softmax we basically create our attention weights.
So the attention weights is what we get.
And this attention waits is nothing but my.
So for the the word, the attention weight will be softmax applied on my values like one comma zero
comma one.
So here once I apply the softmax you can go and and check it out.
I've done the calculation.
It is nothing but 0.42 to 3.1554 .1554.4223.
Okay.
So here initially your value was this.
Right with respect to the the word.
What was your initial vectors.
See this uh your initial vectors was 1010 okay.
1010.
Okay.
I missed out one.
Oh.
No.
We we we got it exactly right.
So 1010 was your initial vectors and now I am able to get this as my contextual embedding vectors.
Similarly, when I go ahead and calculate for the attention weights of cat.
So this was the cat right here.
If I apply the softmax activation function and the softmax activation function is applied on zero comma
two comma two.
Then I'm going to get 0.155, 4.4223, and 0.4223.
Now, whatever vectors we are getting, this is considering the other vectors in context okay.
Similarly, when we go ahead and calculate the attention weights.
For sat sat word.
So again we have to go ahead and pass our softmax.
And this will go to our two comma two comma four okay.
And here we are specifically going to get .2119.2119.
And this is 0.562 right.
Right.
So this is how we go ahead and apply the softmax activation function.
And we get the attention weights.
Okay.
Now finally see this is just our attention weights.
Like what are the attention weights with respect to all the other values.
But you remember right initially we gave a four dimension vectors.
And finally the output also should be a four dimensional vector.
So the final step that we are specifically doing is weighted sum of weighted sum of values.
Now in the weighted sum of values what we do is that we multiply.
The attention weights.
Attention weights by the corresponding value vector, because somewhere that value vector needs to be
used, right?
Value vectors to get the final output.
So for the token.
For the token, the right.
If I want to go ahead and calculate the weighted sum.
So I will just go ahead and write.
The output of the will be nothing, but it will be.
What is the attention weight?
0.4223.
So first one I'm getting 0.4223.
This will be multiplied by value vector of the right.
Then the attention weight of cat right.
We will just go ahead and do this.
See the the the the the the the output.
See the the vector needs to be converted into some other vector.
Right.
And it should be converted with respect to the context of cat and sat.
So attention weight of cat.
So what we are going to do, we are going to basically use the next weight that is 0.1554.
So here I will go ahead and write 0.1554.
And this much attention needs to be given to the value vector of cat.
Similarly the last one is 0.24223.
This much attention needs to be given to the value of sat right value vector of SAT.
So 0.423 right.
Similarly, once I probably go and do the forward calculation.
So here you'll be able to see I will use for 2 to 3 value of the is what.
So if you remember what did we consider as the value of the.
If you see this.
So this is my q q k value.
So value is nothing but the initial value that we have used embeddings right.
So let's see the value.
So for the it is 1010.
For this it is 0101.
And for SAT it is 11112 okay.
So let's go ahead and write this I will do the calculation over here.
I know there are many steps guys but please focus I'm solving it completely step by step 1010.
Then we will multiply.
Sorry.
We will add
.1554 multiplied by v of cat.
Right.
So V of cat is nothing but 0101.
Then you have
0.42231111
Okay.
So now once we do this dot product what values we are going to get over here.
Very simple.
Just go ahead and do the computation.
So first of all uh when we do this particular dot product over here uh so the further calculation once
we do this particular dot operation here, we are going to get 0.42230.
Then again .42.
Two.
Three zero.
Right.
This will be my first vector.
Then I have this vector .1554.
Sorry, it will be zero.
Initially it will be zero because we are doing a dot operation with zero.
Then I have .1554, then again zero, then .1554.
Okay.
And similarly here I will be having
.4223.4223.4223.4223
okay.
Done.
So once I get this, it is nothing.
But finally I'll be able to get 1.2669 and point.
If I just do the, uh, plus operation with respect to all the numbers.
Okay.
.3491.2669.
You can validate it if I'm doing all the computation right.
If not, you can let me know in the comment section.
Okay.
So this is basically my output.
So initially you'll be able to see that when I gave the the keyword which was having an embedding vector
of 1010 after passing through the self-attention.
It is giving us this keyword that is nothing.
This vector that is contextual vector
1.2269.9991.2669.9999.
Okay, so this is what is my contextual vector.
Okay.
Contextual vector here what we did.
First we calculated our q k v values vectors.
For this we initialized w of q w of k w of v vectors.
And again this will be trained.
Then after this we went and calculated our attention score.
After calculating attention score we scaled the value.
Then after doing this, we passed it through an softmax activation function.
And finally we calculated the weighted sum of values sum of values.
And how do we calculate this?
It is very simple.
After performing the softmax, uh uh, whatever value we get, we have to multiply with V, right?
And that is how we specifically get all these values.
Okay.
I hope you are able to understand this.
And similarly for token cat and all we can also do the calculation I will live up to you like how you
can probably do the calculation.
But uh again for token cat you can follow the same mechanism.
So here you have .1554.
Then you multiply it by the vector of value of the then value vectors of the.
Then you have the second 1.4223 multiplied with the value of the.
And similarly when you do this you will be able to get all these things right.
So this, in short, was your entire self-attention uh, step by step.
What all things we did, how the calculation was everything I've explained to you This entire weights
will be trained it.
These are trained, learned weights.
I will say it as learned weights.
And we have to make it learn based on the back propagation, like how we specifically do for other neural
networks.
So I hope you are able to understand this particular video.
This is it from my side.
I will see you all in the next video.
Thank you.
Take care.