Hello guys.
So we are going to continue our discussion with respect to Transformers.
Until now we have discussed so many things.
That is self-attention layer.
We have discussed about what is multi-head attentions.
You know what all operations specifically happens inside the self-attention layer itself?
We have actually discussed okay.
Now in this video we are going to discuss about a very important topic which is called as positional
encoding.
That is nothing but representing order of sequence.
Okay.
Now see this order of sequence is a very important thing.
Okay.
One of the major advantage of using transformer is that okay, the major advantage, all this tokens.
Right.
All these words tokens, words, tokens.
It can process.
It can be processed.
Parallely.
Right.
This is the most important, most important advantage because at that time I can just directly go ahead
and pass my X1X2 word.
And all this operation that specifically happens in the self-attention layer.
Uh, it will just happen parallelly, right?
And if I consider because of this advantage, there is also a specific drawback.
So now if I talk about the drawback because of this advantage it lacks the sequential
sequential structure of the words.
Now, what does that basically mean, right?
Now, this sequential structure basically means the order, right?
Obviously, since all the words are specifically taken by the self-attention layer in a parallel mode,
all the word tokens are taken all at once, and all the processing is basically happening.
It will not have any idea whether x one word has come first or x two word has come first, right?
Let's say I take an example over here, right?
I say hey, lion eats tiger okay.
Or I can basically say lion kills tiger.
Okay.
So we can take this example.
So this is just one sentence.
The other sentence can be tiger kills lion.
Now because of this ordering of the sentence or sentence one and two, the meaning of the sentence completely
changes, right?
Anyhow, in both the sentences, all the three words are basically considered.
And at the end of the day, if I pass this through self-attention layer, you know, and since right
now my self-attention layer is also not taking care of the ordering part, both this sentence will be
returned with the same vectors, right?
With respect to all the words.
But as you all know, this entire sentence is completely different because of the ordering of the word.
So this is what right now my self-attention layer or this entire transformer architecture is missing,
right?
It is missing this information.
So in order to prevent this, what we specifically do is that we use a concept of positional encoding.
And this positional encoding will be responsible in representing the order of a sequence.
Okay.
And that is what we are going to learn about.
Uh, because without this positional encoding it will not know which word is probably coming first.
Okay, so the concept is very simple.
What we will do is that, uh, according to the research paper of attention is all you need, right?
Attention is all you need.
So what they have actually done is that they will see if this is my embedding vector.
Right.
For this particular word.
Similarly we will go ahead and create another one vector.
This vector and we will see how we can generate this particular vector.
And this vector will be nothing, but it will be the positional encoded vector.
This is nothing but the positional encoded vector.
And this positional encoded vector will be added will be added with this vector.
Similarly, this positional encoded vector will be added with this vector.
And this vector will be responsible in telling which words are in which position.
Exactly.
Okay.
And that is what is the entire idea over it.
Right.
So there are multiple ways how we can fix this particular problem.
One of the way is that why not with this embedding vectors that we are passing, let's say over here
I've just taken three dimension.
Why not just add one more dimension and probably talk about the position of the words that are coming?
Let's say for the first sentence, tiger comes first, so this position will be one.
Then the second vectors that we are specifically giving this position will be two, along with the vectors
that we are giving.
And the third word basically is about lion.
Let's say that we are giving one more word over here with the third index.
Right.
So once I'm sending this information here we are giving some kind of position.
Right.
But do you think right now for a smaller text it is fine.
But let's say if I have a book or if I have a journal, or if I have a novel right here, they may be
more than one lakh words.
Now, if there is a more than one lakh words, obviously we are not limiting how many number of positions
will keep on writing, right?
And when we have so many number of words and just imagine we are trying to add a vector with such a
big number.
So this usually causes a problem while we are doing the back propagation while updating weights.
And if we consider huge, huge data right, there is no limit of that particular word.
So that basically means the number right now with respect to the position, it is unbounded.
This position can have any number of words.
Right.
So this is one of the problem.
And we cannot directly use with this specific position.
You may also say hey we can probably apply some kind of mathematical rule and we can try to replace
it.
But if we go according to the research paper, that is attention is all you need here.
The idea is very simple.
Here we will try to add a positional encoded vector which will be of the same size.
And then we will try to add this particular vector to this embedding vector itself.
Right.
And that will actually give the uh, position information with respect to the word that comes in the
sentence.
But now the question comes right.
How do we how do we probably create this positional encoding vectors?
Okay.
So for this, uh, there are two different ways and out of which we will be discussing one of the way,
which is even spoken in the, uh, research paper or which is written in the, uh, research paper.
So types of positional encoding.
Okay.
So first one is nothing but sinusoidal.
Sinusoidal position encoding.
The second one is something called as learned positional encoding okay.
So out of this two okay I will give a detailed idea about what exactly sinusoidal positional encoding.
We'll try to understand this okay.
And uh sinusoidal.
Uh see.
So let's go with the first one.
That is nothing but sine you saw the positional encoding and what exactly this is, this is basically
a technique of creating that positional, uh, encoding vectors.
So here it uses sine and cosine function.
Sine and cosine functions of different frequencies.
We'll try to understand what exactly it is okay of different frequencies to create.
Positional encodings.
And we'll solve this entirely with a good example also.
Okay.
And now because of this positional encoding.
So sorry.
Because of this positional will not have this particular problem.
Also like it will not be like unlimited because the sinusoidal encoding all the values will be ranging
between minus one to plus one, and at any point of time with respect to any number of vectors.
Okay, you'll be able to see that either we'll be getting the values between minus one to plus one.
Okay.
Um, so let's see this how this exactly works.
And uh, then we'll try to understand with a basic example.
First of all, in order to compute the sinusoidal position encodings, the formula is very simple okay.
And here what we are doing basically whatever vectors we really want to create right.
Positional embedding vectors, all these values.
Let's say if I have a four dimension, all these values will be replaced, right?
All these values based on the position of the words will be replaced by this formula.
So this is positional encoding.
That is nothing but pos uh position comma two of I.
It is nothing but sign.
Position divided by 10,000 to the power of two of I.
D of model.
We will try to understand each and everything.
Okay.
This is for the sinusoidal uh, sine function.
One more positional encoding that we will try to calculate is nothing but position of two of I plus
one.
Sorry two of I plus one.
Yeah, this is nothing but cosine of same formula.
That is position of 10,000 square two of I divided by d of model.
Right.
So this is the formula.
Now here let us break down this particular formula.
Because see at the end of the day we need to create this position embedding vectors in such a way that
the dimensions that we specifically have with respect to the embedding vector, the same dimension,
it should happen.
So here we will go ahead and write where p is equal to sorry, where pos is equal to is nothing, but
it is the position.
Okay, I is the dimension.
Okay D of model.
Is the.
Let me just clearly write this down.
Okay.
So here I will just go ahead and write where POS is the position.
Everything will make sense once I solve a specific problem and show it to you.
And then here I is the dimension.
Okay.
And then you have this D of model.
It is the.
Dimensionality
of the embeddings.
Okay.
Now what exactly this is and how we will go ahead and compute it.
I will just show you this with a specific example.
Okay.
Let's say, uh, here I have an example which says the cat sat.
Okay.
Uh, first of all, we'll convert this into vectors.
Okay.
So let's say that this is my the the is represented by a vector like this .1.2 I'm just initializing
.3.4 okay.
Similarly I have this cat.
Cat is represented by
.5.6.7.8
okay.
And then you have the SAT.
SAT is nothing but it is point nine, 1.0, 1.1, 1.2.
So this is the embedding vectors that we have.
Now we will go ahead and create the positional encodings for this because before giving it to the self-attention
layer the same dimension this is just a four dimension.
So this entirely is like four dimension for each and every word.
Similarly, we have to go ahead and create our positional encoding with the same four dimension.
And we are going to specifically use this technique, which is called as positional encoding with the
sinusoidal learning.
Okay.
So here what we will do.
Again we'll go ahead and probably use the formula okay.
Uh with respect to the positional encoding.
So here I'll write P or let me use another color.
So here I will just go ahead and write p with respect to pos comma two of I is nothing but sine of.
Pos.
10,000 square I of d of model.
Okay, the same formula.
And here I'm just going to write p that is positional encoding of pos comma two I plus one is equal
to cos of pos.
10,002 of I divided by d of model okay.
Perfect.
Now understand over here okay I have to generate for positional encoding vectors.
Okay.
So here for our example.
Okay.
D of model basically means what dimension of model.
So dimension of model is four right.
Four four over here.
So I will just go ahead and write for position.
POS is equal to zero for this position.
For this position, how do I place a value?
Okay, so for this I will go ahead and calculate p.
My position is zero comma.
What is I over here.
What is I to I comma one to I plus one.
Okay so for the first word you'll be able to see right.
For the first word I have to represent this into for positional encodings.
Right.
So I will iterate through this specific I right.
So here what I will say for position zero.
If I want to calculate the positional encoding for zero comma zero it is nothing but sine of zero divided
by whatever.
Is there 10,000 square or sorry square?
Uh two of I.
So I is zero over here, right.
So this is zero, right.
So obviously this will entirely become zero.
That is nothing but, uh, zero itself.
So here I will just go ahead and write ten to the power of zero.
Right.
Because I is zero divided by d of model.
Again it is nothing but four.
So this entirely sign of zero it will become right.
So here I'm actually going to get zero itself.
Then.
Similarly if I go ahead and say positional encoding for zero position.
But first I like one ith position okay.
Ith position how many ith position I have to really do it is 0123.
Right.
So for four for the first, again I will go ahead and compute sign zero divided by zero divided by.
Anything over here will be zero sine of zero.
Again it will be zero or sorry should not be sine of zero.
Uh, just a second.
It should be cos.
Because for every position you'll be able to see that I will be applying this position and then I will
be applying this position.
Right.
So here you'll be able to see with P with positional encoding with zero comma one okay is nothing but
cos of whatever cost function we are specifically writing position is nothing but zero zero divided
by anything will be zero.
So cos zero is nothing but one.
Okay.
Similarly I will go ahead and compute for positional encoding for zero comma two.
Now zero comma two.
Again I will be using sign okay.
See uh you'll be able to see that if I have two embeddings, like if I have this four dimensions, whenever
I apply a sinusoidal wave.
Right.
Like this sinusoidal wave looks something like this.
And the, uh, cosine waves looks something like this.
So for each vector, we have to probably go ahead and find out the sinusoidal wave, uh, sinusoidal
values for that and the cosine values for that.
That basically means if I want to find out the positional encoding for this, for, uh, dimension vectors
here, this one will basically be calculated as a combination of two.
So let's say this is my first combination.
This is my second combination.
That is the reason we are using this formula.
This combination will be calculated one.
The first one will be calculated with this formula and the second one will be calculated with this formula.
And again when we go to the next set of two, this will calculate it by this formula.
And this will get calculated by this formula.
That is how it exactly works okay.
The reason is very simple why we are using in this way, because we should not be getting a similar
kind of encoded values.
If we get a similar kind of encoded values then we will miss the order of the will, miss the order
of the elements.
Right?
Will miss the order of the elements.
This is really important to specifically understand.
Let's say you say that hey Chris, why to use this cosine instead just try to use sine, okay?
The amazing thing about sine will be that all the values will be ranging between minus one to plus one.
Okay, but if I just use sign, there may be that for same two different values, I may get the same
sign value right for this.
Also, I may get the same sign value for this.
Also, I may get the same sign value.
So that is the reason we may miss the order of the elements or of the tokens.
And we don't want this to happen.
So what we have done is that we have combined this with another cosine equation.
Okay.
Now what will happen for every vectors?
First, if this is computed by sine the next vector will be computed by cosine.
So let's continue this particular uh findings with respect to the positional encoding.
So now my third one that I'm actually going to compute over here will be sine of zero comma two.
See I will be into uh 012.
At this position I you have to use sign here.
I have to use cosine.
Right.
So this will basically become sine of zero.
And then we are going to use 10,002 by four okay.
But again here I'm actually going to get sine of zero.
Then again this will be zero.
Then finally we'll go ahead and compute the positional encoding of zero comma three.
Again we are going to use cosine.
Again this entire value will become zero and this will be one.
So with respect to our first word right, what is the positional encoding that we are getting?
It is nothing but 0101 right.
Now similarly if we go with respect to the position one right position one is with respect to our second
word, right.
So if we go ahead and calculate it.
So let me just go ahead and write for position one.
So for position is equal to one.
Now I'm just going to use the same formula.
First of all I'll write position encoding for one comma zero.
One comma zero basically means my second vector.
So this is for this.
And this one will be my second vector which will be represented by another four dimension.
The positional encoding will be a positional encoding will be shown by.
And this is for this particular word.
The positional encoding is for this word.
Okay.
Again I have that same operation that I have to do.
And when I say position one, that basically means we are going for the first word, right?
And finally after that we'll go for the second word.
So positional one.
Then again we go ahead and compute it.
It will be sign again it will be one divided by or 10,000 okay Okay.
Zero divided by four.
Right?
Why zero divided by four?
Because I is zero.
So again I'm going to get sine of one.
Sine of one is nothing.
But it is approximately if you go ahead and calculate it it is 0.8415.
So similarly we will go ahead and calculate the post encoding of the position one.
So over here we will go ahead and compute.
Now the post encoding of positional one with the I value is equal to one.
So again here what formula we are going to use.
We are just going to write cos and let's see the formula.
So I will also make sure that I will try to specify this formula over here so that you will be able
to refer this.
Okay.
So here I'm just going to copy it.
I'm going to paste it over here.
The next formula again I will select this.
and I'll paste it over here.
Okay.
So once I'm calculating this okay.
Please try to understand over here.
So.
So I'll select this right now.
Okay.
So I'll keep it over here.
And similarly this I will keep it here for my reference okay.
So here when I am calculating with position one and I is equal to one.
So if I really want to use the cosine value.
So this uh I that you will be specifically getting right.
So here I will write position value one okay one divided by 10,000 okay.
To the power of two of I.
Right.
So two of I is nothing.
But uh, over here I will just go ahead and write two multiplied by one divided by four.
And this whatever approximate value will be getting will get.
It's 5403.
Similarly, you can go ahead and calculate the post encoding for one comma two.
Again we have to use the sine formula.
And with respect to the sine formula just go ahead and write over here one by 10,002 by four.
This will be approximately again equal to 0.01.
And again once we go ahead and compute it for one comma three.
again, have to use my cost value.
Just try to do the computation.
Okay I'll leave it up to you.
So here I will be getting my value as 0.995 okay.
So in short here what we have actually done is that the the word that we have specifically taken.
Right.
Cat.
First of all, I will just go ahead and write with respect to the word for for the the word.
So here you can see for the, the word which was given by the vectors.
Right.
Two, three for the vectors for the that we have actually used is .1.2.3.4.
So here if I go ahead and write this is .1.2.3.4.
For this we have converted this into some other encoding.
And here the positional encoding is nothing.
But, uh, this value that is 0101 okay.
So here we have specifically got as zero one and zero one.
So this is the positional encoding that we are able to find out for this particular vector.
Right.
For this vector.
Similarly for the word cat.
For the word cat, initially we had this vector.
And after performing the positional encoding you'll be able to see it was .5.6.7.8.
Right.
So here I will go ahead and write .5.6.7.8.
This we were able to get an encoding of the values that we got over here two, three, four, and the
values were nothing but these all values 0.8415 .84.
Then you got 0.54 and then 0.01 and 0.95 or 9995.
Okay.
Now this is how you have actually got the positional encoding.
And through this way what you are actually doing is that we have to probably sum up these values, right?
So finally, you'll be able to see whatever self-attention I will be giving right over here, whatever
vectors I will be giving in this self-attention layer, okay.
It will be the combination of your vectors that we are specifically getting with the positional encoding.
That is nothing but 0.5.
Let's say sorry 0.5 0.6, but instead I will write 0.840.
Let me just go ahead and remove this and let's use an positional encoding here.
It will be four dimension again based on the problem statement that we have done in uh, in the research
paper it is taken as 512 dimension.
Okay.
You can go ahead and check out the research paper.
So
.84.54.01.9999
okay.
And this we have to sum it up with the our vectors that we have actually calculated.
Right.
So for the cat word this will be the word that will be going on point five plus point six plus point
seven plus point eight.
Similarly for the other vector that we have right, the again the ordering can be placed over here.
Also, it can be placed over here, but we are handling it with the help of because all the words will
be going parallelly.
Right now, this positional encoding is making sure that whatever values we are putting, it knows the
order now, right?
So for this, let's say the value was .1.2.3.4.
So so for this you are actually getting nothing but 0101.
And this is also going to get summed up after the summing up is going to happen, then this entire value
is going to go to the self-attention.
And finally we are going to give entirely this to our feedforward neural network.
And here we are going to get our Z one.
Here we are going to get our z two considering the multi-head attention is applied.
But this values that we have actually got this values that we have actually got.
Sorry.
Here I just need to reverse this okay.
So this needs to go down and this needs to come up okay.
Because this is the positional encoding okay.
So this were my word vectors.
So if you probably go ahead and see with respect to the cat this is the vector that is getting created.
This is with respect to the.
This is my positional encoding.
Positional encoding to making sure that what word is gone in which order it is gone.
But in this case, uh, you know, the the has to become a cat has to go over there.
Right.
And similarly over here, this is my positional encoding.
Please don't get confused.
I've just changed the order here itself.
Okay, purposely.
But it won't matter a lot because anyhow, I'm giving the positional encoding information so that to
make sure that which word is coming after what?
Right.
And then we finally give it to the self-attention.
And here if multi-head attention is basically there, multi-head attention is there, you know what
is exactly happening in the self-attention then all these multi attention will give given to the neural
network.
Okay.
okay.
That is feed forward neural network.
And finally we get this that is z one and z two which will be our contextual vector.
So this was the overall idea about positional encoding.
But in short if you really want to know about positional encoding, it's it's more about, you know,
uh, it's more about making sure that the order of the words is taken.
The representation order of order or sequence is given to the self-attention layer in some way.
Right.
We need to make them know.
Okay, this is the first word.
This is the second word in the sentence.
Obviously this problem will not happen in RNN because our RNN we give one word at a time with respect
to timestamps, right.
So this was one of the most efficient technique that we used.
One more technique is something called as learn positioned encoding.
Now in the case of learn position encoding, what we do is that uh, in this case uh.
Position encodings.
Positional encodings are learned are learned during training okay.
During training.
Now when we say during training, it is learn.
That basically means we again we need to create a positional encoding matrix over here, and it needs
to get updated through back propagation.
Again, I don't want to discuss more about this because I think in in the research paper also it is
mentioned about sinusoidal positional encoding.
Right.
And this is nothing, but it is a combination of sin and cos where we keep on continue or we keep on
creating this such that no two values with respect to our positions, no two values should get the same
value over here.
Uh, with respect to the encoding.
That is the reason we specifically use this.
And this is how the entire calculation is basically done in order to calculate the positional encoding
of a specific word.
So I hope you like this particular video.
This was it about positional encoding.
Um, in the next video we are going to discuss about something called as layer normalization.
And that is what we are going to discuss in the next video.
So yeah, I will see you all in the next video.
Thank you.
Take care.