Hello guys.
So we are going to continue the discussion with respect to our Transformers.
Now in this video we are going to discuss about the entire encoder architecture.
Already we have discussed about self-attention.
We have discussed about multi-head attention, we have discussed about positional encoding and so many
different topics.
We have also discussed.
We also discussed about layer normalization and each and everything.
But now based on the research paper, we will just try to discuss how the entire architecture is because
this is just one encoder.
Right.
And this is just one decoder.
So right now first of all we'll focus on encoder.
But initially only I told you right.
Based on the research paper you will be seeing that there will be six encoders like this.
Right.
So there will be not one encoder.
But instead of this there will be six like one encoder two encoder 3456.
Similarly there are six decoders okay.
And with respect to any task like machine translation, you give the you give your input over here,
you get your output over here.
Right.
So this is the part that we are specifically going to discuss about.
Okay.
When I say this is just one encoder, that basically means this part can be mentioned in this way,
right.
If I probably expand this, it is nothing.
But it is basically this entire encoder part, right?
Similarly, you have other encoders also because after passing, after coming after the information
is basically coming out of this, it will go to the next encoder, then it will go to the next encoder,
then it will go to the next encoder.
And all right so based on the research paper let me just go ahead and write some very important information.
So first of all the input that you specifically take right.
You convert that into vectors.
Then you add a positional encoding on top of that.
Then you pass through a self-attention layer.
Right.
So uh, if I probably show you the flow, it looks something like this, right.
So I'll show you step by step, like how things basically happen.
And then we will just try to, uh, understand, uh, like how many embedding vectors is specifically
used and all.
Okay.
So let's say this is your input sequence.
So here is my input sequence Now, from this input sequence, what we specifically do.
We go to the next step.
In the next step we are specifically going to convert this into vectors.
That is through text embedding techniques.
It can be any embedding techniques.
Plus we add it with positional encoding okay.
So once we add this with positional encoding.
So this is what your input encoding plus positional encoding is basically going on right.
The input sequence when it is converted into text embedding.
So the number of dimensions that is specifically taken for the text embedding it is nothing but 512
okay.
That basically means every word is converted into a 512 dimension based on the research paper.
Okay.
So in the research paper it is mentioned when they try to entirely create this.
Initially they use this 512 uh vectors for every input sequence.
Right.
So positional encoding will also be same.
Right.
Then after we pass through this what we get we get our multi-head attention.
Right.
We get our multi-head attention.
And this is where your self-attention works, right?
Between this multi-head attention.
The multi-head Multi-head attention basically means how many different head attentions you have over
here.
So if I probably talk about here, you have eight head attentions like z two, z three, z four, z
five, z six, z seven, z eight.
So this is also based on the research paper.
So in research paper how many attentions had they specifically used.
Eight.
Okay then, before we go ahead and pass this information from Multi-head attention here, what we are
specifically doing here.
We are first of all, adding and then normalizing.
So that is the reason we say add and not write.
The normalization technique that is specifically used is layer normalization.
So here what we are doing before doing this we need to add this information that is probably coming
from this layer.
And this is what is called as residuals.
Okay.
This is basically called as residuals.
What is the importance of this.
Just imagine before I do the normalization the output of the multi head attention is also going.
And we are adding the text embedding and positional encoding information.
So that basically means we are giving some additional signals.
Right.
So we'll try to understand I'll write down point by point what is the importance of this.
So residuals is basically over here.
Then after doing the normalization we basically send it to the feed forward neural network.
Right.
So these are all the steps that we are specifically using in the feed forward neural network.
It will be something like this.
This will basically be my input.
The input will be based on all these dimensions that we are getting.
And then we have one hidden layer.
In this hidden layer we have somewhere around 5 to 12 hidden nodes.
Okay.
And then I have my output layer okay.
So this will basically give you the output based on the information that I have based on the number
of vectors.
That is probably coming up.
Okay.
So here that is what it is basically happening.
And this is the input that is coming from this.
Right.
All the input that we are specifically passing and it will get trained.
Once I do all these things, then it is sent to the next encoder.
And the next encoder will have all the information with respect to all these vectors that I have.
Right.
But understand when I say five to l, this is with respect to every word.
Every word is basically converted into five to l dimension of vectors.
Okay.
Now this is what the basic information, but they are still more information if I go inside self head
uh, self uh attention right self uh self attention.
So what are the parameters that is specifically used.
Right.
That 3 to 4 important parameters over there.
Also one is your Q query key value.
So what dimension this query key value is right.
Based on the research paper this is 64 6464.
So you'll be able to see inside self-attention when I do the product of query into K right.
Later on when I normalize and when I have to divide right, I divide by root of 64, which is nothing
but eight.
Okay, this is what we are specifically doing, right?
We are dividing by 64.
And the number of attention of head is basically 68.
Right.
So all these information I have actually mentioned.
So these are some of the basic parameters that are specifically used in research paper.
Now the question is coming right.
Crush.
Why 52.
Well why so many number of encoders.
Because the understand guys the kind of task that you do is kind of sequence to sequence task sequence
to sequence task are very complex task.
They are very very complex task.
Right.
Let's if I probably say language translation, it is a very complex.
Right.
You have one language.
Then we're trying to convert that into other language.
They'll be dialects.
There'll be so many different things.
Now, if I really want to convert this kind of task just with one encoder, you'll not be able to get
a very good accuracy.
So we definitely need to use many encoders over here.
According to a research paper they have used six encoders and they were able to get good results.
Okay.
So this is one of the very important thing that you really need to understand.
Okay.
The second thing, uh, that usually comes in your mind, right.
And obviously everybody's mind first of all why residuals okay.
Why you need to probably, uh, add residuals.
Why you need to be probably provide some information over here.
Uh, and probably give it to the additional and normalization layer.
Right.
This is the next question that many people will specifically ask.
Right.
And the third question that probably comes is why to use feed forward neural network.
Right.
So that is also there in your mind because feed forward neural network.
Uh, here also you have used one kind of layer.
Even though we are doing the back propagation over here, we have updating the weights of w q w k and.
All right.
So we'll try to answer step by step all these things.
And uh again uh after thorough research, what are the points?
What are the, uh, things that have basically been there from the researchers?
Uh, will try to note it down.
Okay.
And as I have already shown you, self-attention, this is what is the operation that is specifically
happening.
And this is just a two example like encoder one and encoder two.
You pass your self-attention.
You pass this particular information from your input embedding vectors.
So this is basically a phi two L vectors.
Right.
So this is nothing but 512.
This is also 512 okay.
Then positional encoding will also be 512.
Then you have to add this.
Then what you are doing you are passing this information to the next layer.
That is additional normalization.
Then this gets passed to the self-attention.
Then after that a feed forward neural network.
Then for every uh combination of self-attention feed forward, you are doing normalization and why normalization
specifically used?
I've already spoken about it.
Right.
Your input distribution may be different here.
And after doing all these operations, your distribution may become different, right?
So what we do we basically do a layer normalization which we have discussed in our previous video.
Right.
So let's talk about the residual right.
What is residual over here.
Any information that I'm passing it to the next layer here.
Also I'm passing here also I'm passing here also I'm passing.
And if you see in decoder also here also you are passing.
We'll discuss about decoder later on.
But let's focus over here in the encoder okay.
So residual connection.
What it is it is also called as skip connections.
Okay.
That are used in neural networks okay.
Now let's understand why residual connection is basically required okay.
First of all, just imagine okay guys I'm passing this information to the next layer.
I'm skipping the self-attention.
I'm also passing what was the input.
So if I'm passing additional information if I'm normalizing it, don't you think this layer will be
having more information about the input?
Right.
Even though we are able to probably do all the we are able to convert that into a textual contextual
vector by using self-attention, right?
But based on the research paper, there are multiple reasons why it is basically used.
So first one is addressing the vanishing gradient problem.
Now why do we say vanishing gradient problem.
Because see guys here we have somewhere around six layers right.
Six layers of encoders and six layers of decoder During this particular case, you will be able to see
that as the number of layers increases, there are chances the gradient of loss function with respect
to the weights can be very small, and this phenomenon is basically called as vanishing gradient problem.
Okay.
Now how does the residual help you in this okay.
So I will just go ahead and write.
How does residuals help you in this.
Or skip connection help you in this.
It is very simple okay.
Here residual connection.
Create a.
Short paths.
Very important point.
Gradients to flow.
To flow directly through the network.
Okay.
Because of this, the gradient remains
sufficiently large.
And this even in deep neural networks.
Now see what is happening over here.
I'm passing this information from here to here.
I'm skipping this entire operation of self-attention.
Right.
And I'm passing some additional information.
So when this entire information is basically passed, you'll be able to see that we are creating a short
path because see, in back propagation what will happen over here, all the weights will get updated.
And here you are passing some additional information.
Right.
So because of this it creates a short path for gradient to flow directly through the network.
Gradient remains sufficiently large okay.
Now because of this what will happen?
The second very important point that I really want to mention it improves gradient flow.
When I say gradient flow, that basically means my convergence will be faster.
My convergence will be faster.
Okay.
Now if I don't do this, I may face some vanishing or vanishing gradient problem.
Okay.
It may also be uh, exploding gradient problem.
Because if let's say if my weights is initialized in a higher way, right.
In a very large values, okay.
So when convergence is faster, then obviously your training will be also smoother, right?
Now coming to the third point, this actually enables training of deeper networks.
Deeper networks.
Now here in the encoder you can see that I'm just not using one encoder.
I'm using six encoders.
So this becomes a very large network right.
So very deep neural network right.
So here because of this, at every part of this encoder we are doing this particular step that is called
as residuals.
Right.
So this will basically enable training of deep neural networks okay.
So you can basically say that residual connections allow for the effective training of much deep networks
by addressing the gradient flow issues and making it easier for the network to learn identity mappings.
This enables the model to be very expressive and capable of learning more complex functions.
Right.
So this was about, uh, you know, the residual connection that we have specifically discussed.
Okay.
Now let's talk about over here why we are using this feed forward neural network.
Okay.
This also you really need to understand why feed forward neural network.
What is the importance of feed forward neural network.
Because, uh, I've just given one example with respect to feed forward neural network.
Obviously, if you know about N, then n is nothing, but it is a feed forward neural network.
So here why feed forward neural network?
Again, based on the research paper, I will try to give you the definition with respect to some advantages.
Okay.
First of all, see if you have created anytime an Ann.
Right.
Let's say this is an Ann.
And in the Ann let's say in the output layer you don't have any activation function.
So this basically solves a linear problem right.
Sorry.
This solves a linear function problem.
Problem.
But the main aim of feedforward neural network is basically to capture the nonlinear function.
It is to solve the nonlinear function.
Right?
Now in the nonlinear function how do we solve this and all.
Obviously we we have seen right in every layer we apply an activation function.
Right.
And based on that it is able to capture the non-linear functions, the most complex properties within
a specific data set.
Right the behavior.
And all right.
So similarly feed forward neural network will also do that same thing obviously before the self-attention
is able to do good task.
But to capture more information for this complex task we specifically use this feed forward neural network.
So what it does over here, I will just add one point that it is going to add non-linearity.
Okay.
Second information and probably um, this is even not mentioned in the research paper.
But I was reading an article and there I specifically understood about this.
Okay.
It is nothing but processing each position independently.
Okay, now let's talk about this.
Why it is important.
Now you know that self-attention mechanism captures the relationship between tokens, right?
What does self-attention basically do?
It captures the relationship.
It captures the relationship between the token.
It processes the relationship in such a way that each token can attend to every other token, right?
That is how it basically captures the relation, right?
And that is the reason you don't require further transformation or feature extraction on the representation
obtained from the self-attention.
Okay.
Now how does feedforward neural network help this feedforward neural network?
Based on the tokens, whatever context you are specifically able to get, it processes each token representation
independently right before what it needs to do.
We are sending all the information with the positional encoding.
That is how your self-attention will understand which words are probably coming first.
Which words are probably coming second, right?
But if I talk about the feed forward neural network, this process is each token representation.
Independently.
Now you may be thinking what is so important about me?
This means that for each token representation resulting from the self-attention mechanism, the feed
forward neural network applies the same two layer neural network.
As I said, there will be a two neural network, and this helps since we are taking each and every token
and we are trying to retrieve more specific information out of it.
This helps in transforming this representation.
Further.
And allows the model.
Allows the model to learn.
To learn.
Richer representation.
With the self-attention, we are definitely able to understand some of the information based on the
contextual information.
But by using feed forward neural network, again, we are taking each and every vector for that specific
information or for that contextual embedding.
And then we are allowing the model to learn more representation from that particular thing.
Right.
It's just like more trying to explore more of this specific vectors that are created by the contextual
embeddings.
Right.
So this is why we specifically use feedforward neural network.
And if you know about Ann, which is the kind of feedforward neural network, you can just understand
with the help of forward and backward propagation, what all information will be able to explore and
get.
Okay.
Um, again, uh, the third important point about feed forward neural network is that already our network
is really deep, right?
So feed forward neural network helps your model, uh, helps your neural network to become much more
deeper.
Right.
So it basically adds depth to the model.
And one important property about depth is that if you're handling all those vanishing gradient problem
at all, as there is more depth, more learnings will be able to get from the data, more learnings
you'll be able to get from the data.
So that basically means the neural network will be able to capture more and more information from that
particular data.
Okay.
Now it's not like that.
You can probably use just ten layers of feed forward neural network.
But this is doing one thing because the two layers is specifically able to do much more things over
there, right?
And again, uh, when you say one more very important property with respect to feed forward neural network,
it also increases the model parameters, okay.
And because of which it will be able to help us to generalize well with the training data and with the
test data, which is unseen data.
Okay.
So this is more about understanding why feed forward neural network, why this and all.
But if I just go with the flow, I have each and every word that is passing to self-attention.
Add normalization, feedforward neural network, then again passing through the next encoder.
Now again you may also have a question Krish why so many number of encoders.
Right.
Again as I said synchronous sequence to sequence task are more complex.
So as we are able to extract more and more information, more better, it is right.
Tomorrow, if you are planning to create another kind of LM models or any other models, you can also
change this.
You can use for encoders if you want.
And all right.
So after all this information is probably passed then from here it goes to the decoder.
Now in my next video I'll try to make you understand what is exactly about the decoders.
But in this video I told you about the parameters.
So let me revise what all parameters are specifically used.
One is for the embedding vectors.
Embedding vectors is nothing but 512 dimension.
Your q k, v is nothing but 64 your multi-head attentions.
How many multi-head attentions you have?
It is nothing but eight, right?
And then along with this positional encoding will also be the same thing over here.
And all the remaining information I have discussed.
Right.
So yes, this was it for my side.
I hope you liked this particular video.
Uh, this was about the encoder architecture.
I'll see you in the next video.
Thank you.
Take care.