Hello guys.
So we are going to continue the discussion with respect to Transformers.
Already till now we have discussed about many things, right?
We have covered something called as self-attention.
Then after covering self-attention layer and how it exactly works, we have also understood about Multi-head
attention.
Then after covering Multi-head attention, we saw the problem with respect to the sequence of word,
and for this we also understood about what is the importance of positional encoding.
Right.
And we have actually seen a lot of examples I have personally solved done all the calculations how with
the help of self-attention layer, how we can convert that fixed vectors that we specifically had from
the inputs to contextual vectors.
Right.
So all those things we have specifically covered.
So in this video we are going to go ahead and understand about something called as layer normalization.
Now near normalization is one of the component.
Now if you see the architecture of the transformers and this architecture is from the research paper
here, you will be able to see that initially I have my inputs.
We convert this into some input embeddings.
Then we add it with the positional encoding.
And after this we go ahead and create our multi-head attention.
So here we specifically pass through our self-attention layer right.
Now when we are creating this multi-head attention now you can see the next step.
What exactly after this it is right.
It is basically called as add and normalize.
Okay.
Add and normalize.
And this is what we are going to talk about right.
When we say add and normalize, the normalization technique that is specifically applied is called as
layer normalization.
So we are going to understand the mathematical intuition.
Along with this there is also one more topic that we really need to understand which is called as residuals.
Now here you can see that before going to the multi-head attention, whatever vectors that I get after
adding the positional encoding to the input embedding embedding that is also passed to this add and
normalize.
So this are called as residuals okay.
Residuals.
And this provides some additional signals okay.
Additional signals to the ad and to the layer normalization.
Right.
Layer normalization.
It's just like this line is basically providing the information of this input embedding plus positional
encoding vectors to the additional and normalization layer.
Right.
So in short, uh, that is the reason we basically say here we are saying add and normalize.
So we are going to probably add this previous encodings.
That is probably coming from the input embedding and positional encoding.
And then we are basically going to normalize right.
So that is basically the residuals that is getting added.
So this is the information.
Whatever pre hand we are able to get that we are passing it to the next stage okay.
And that is what residuals is all about.
But again uh don't worry we will try to understand it.
What exactly is normalized and how this addition and normalization takes place.
Okay.
Now before we go ahead, uh, let's understand where this addition and normalization will get added.
Okay.
So here uh, again I've taken this particular diagram from the blog.
Right.
So here you initially have your input embeddings okay.
Layer by layer.
It will be there.
This is for my word one word two.
Right.
Then for each of these words, based on the number of attentions, I will be initializing different
different weights.
Right?
Then we probably go ahead and find our query key and value vectors for each and every attention.
And finally we get all the attention itself right z zero, z one and z seven.
Now see one thing over here.
Where does this see.
Because this is the multi-head attention, right?
And this multi head attention before we pass it to the feed forward neural network.
Right.
So here we will go ahead and apply our next step.
That is nothing but.
Add and normalize.
So let me write it in another color add and normalize okay.
And this is where your layer normalization basically happens.
Layer normalization basically happens.
Now what exactly is this layer normalization.
We'll discuss more about it.
But with respect to the kind of working that you'll be able to see that over here add and normalize
is specifically happening okay.
And here you can see once we also provide this before we go ahead we also need to add our positional
encoding.
Okay.
And this positional encoding will be like with the same shape right.
So let's say if this is my positional encoding.
So here 1234.
So this will be my positional encoding that will get added.
And it will be sent over here.
Along with this.
What happens is that this positional encoding information is also sent over here, and it is added before
the normalization takes place.
Okay.
And the reason over here why we do this.
Because we are providing the signals this this entire information information to this layer okay.
Some more additional signals.
So that is what residuals is all about.
Now let's go ahead and understand about the layer normalization.
So first of all we will just get to know an idea about what exactly is normalization.
And specifically with respect to deep learning we use normalization in two different ways.
One is batch normalization, one is batch normalization.
And the other one is something called as layer normalization.
Batch normalization and layer normalization.
Now let's discuss about this batch and layer normalization.
So for this I will take an example.
Let's say I have a neural network with two inputs.
Okay.
And let's say here in my hidden layer I have two hidden neurons.
And then finally I have one right.
So whenever this input is basically connected to the hidden layer right in this way.
And then finally I'm able to get my output over here.
So this is my output.
Now what exactly is normalization.
See guys let's say over here I am having two features.
Let's say my feature is like house size right.
This is my first feature.
Let's say I will go ahead and right area of the house.
Okay.
House size and area.
I'll say number of rooms in my second feature number of rooms.
And I need to probably predict the price of the house.
Okay, so this is my F1 feature.
This is my F2 feature and this is what F1 feature goes over here.
F2 goes over here.
Now before we usually send our F1 and F2 input to this particular.
And that is artificial neural network.
Or I can also say it as feed forward neural network.
Right here I will be having some information and one by one I will probably send this information for
training the neural network.
Now what usually happens is that the first step and you know that right?
I will be having multiple values over here.
Let's say I will just go ahead and write what is the house size?
Let's say it is 1200 square feet okay.
Number of rooms are two.
And let's say the price is 45 lakhs okay.
And uh, I will go ahead and write, hey, my house size is another house.
Data that we have is 1500 square feet.
The number of rooms are three and this is 70 lakhs.
Okay.
And similarly here you have 2000ft².
You have 3.5, uh, BC rooms.
And here let's say the price is 80 lakhs.
So all this information you specifically have.
Now, the first thing that we basically do before we provide all these inputs, F1 and F2 for the training,
right.
What do we do?
We do something called as normalization.
One of the most common normalization we do.
Right.
That is nothing but standard scaling.
Now what does standard scaling basically mean?
With the help of standard scaling, right.
What we do is that we take each of this each of this features.
Right.
Each of these features, even the output feature if we want and we apply a general standard scaling
formula and that we basically say it as z score.
Now nine z score.
What happens is that we the formula is very simple.
We take every input, we subtract it with mean and we divide by standard deviation.
Now once we do this, if we are considering any feature, let's say if I'm considering house size and
if I apply this formula, all these values, all these values, you will be able to see that we will
be getting in such a way where my mean will be zero and my standard deviation will be equal to one.
Okay, so what is basically happening?
Let's say if f one is one of my distribution, after applying the standard scaling this will get converted
into f one dash.
And inside this f one dash here you will be able to see my mean will be zero and my standard deviation
will be one.
Okay.
Now one thing that you really need to understand, right?
Whenever we are applying this formula, this basically gets applied to this entire column okay.
To entire column.
So first of all, what we need to do in order to apply this formula we need to calculate the mean.
For this we need to calculate the standard deviation for this.
And then for every number we will go ahead and apply standard scaling.
This is what happens when we try to provide our input over here, right?
When we provide our input over here.
Now this scaling is fine okay.
And usually we do this.
And what usually happens because of this I will tell you there are some very good advantages of using
this.
And similarly in deep learning if I talk about deep learning right.
In deep learning, uh, let's say my input is in the form of images.
So there also we do some kind of scaling.
And that scaling that we specifically do for the data is something called as min max scaler.
Now in the case of min max scaler you know that all your images.
Right.
Let's say I have some color images of this of some four cross four matrix.
And you know, all the values will be between 0 to 255.
So what we do after applying this min max scaler.
We convert all these dimensions, all these dimensions between 0 to 1 right, 0 to 1, 0 to 1, all
these pixels that we have.
Right.
If it is four cross four pixels.
So we try to convert this into.
We try to convert this into 0 to 1.
So this usually we do for the input data whatever input data we specifically have.
And what are the usual advantages for doing this.
Scaling techniques which we also see it as normalization.
First let me just go ahead and write some of the advantages.
The first advantage that I would like to write is about improved.
Training.
Stability.
Okay, now what does this improved training stability basically means, right?
See, before before we are passing it to our and write all the information.
We know that my data may be following a different kind of distribution.
It can be this distribution, it can be this distribution.
It can be probably, uh, this kind of distribution, different, different distribution.
It may follow.
Right.
And when we actually convert this distribution into a distribution where all my information is centered
near to zero.
Right.
Because here, what is happening here we are able to convert into data where our mean will be zero and
it will be standard deviation equal to one.
Right.
So and all the values that you will be seeing let's say initially when I probably check the distribution,
let's say that I'm checking the distribution between my f one and f two feature.
Now you know my distribution is coming over here.
Let's say it is coming over here.
Now, what we are doing after applying scaling, it is very simple.
We are taking this and this is my f one and f two.
Now here my magnitude was I'm trying to bring all the magnitude over here right.
So here what is the main property over here.
Your mean will be equal to zero and a standard deviation will be equal to one.
This entire distribution will have this kind of distribution itself.
Now, because of this, what will happen is that when we are doing the back propagation, okay, when
we are doing the back propagation, we will definitely not face any kind of problems related to vanishing
and exploding.
Gradient problem.
This is one of the properties that we usually get after doing the scaling.
And right now all the scaling has been done in the center right.
If we had this huge values, there may be scenario that whenever we do, we are doing some kind of multiplication
with respect to the weights.
And all these values will be quite huge.
Okay.
And because of this, what will happen is that the second important thing that may happen is that we
will be able to get faster convergence.
We will be able to get faster convergence because obviously these values are very slow.
Uh, less right.
So if you remember the gradient descent with respect to our weights, uh, we try to find out our loss
right now since our values are already centered completely.
Right.
This is zero centered.
All the data over here is like zero centered, right?
You'll be able to see that initially, if I compare with respect to our weights and loss, our loss
when we are doing this, when we are probably doing the scaling initially, it will be very, very small.
If you don't do this, there may be scenario that there may be a huge loss over here.
And then our main aim is basically to come to the global minima, right?
It can be from here.
It can be from here, it can be from here.
It can be from different different position.
Right.
So because of this we are able to do the faster convergence.
And the best thing will be that when we apply back propagation.
Back propagation there will be a stable update.
Stable updates okay.
Perfect then.
one more important information that I really want to talk about.
Right.
And, uh, that that point I will discuss later on.
But first of all, this is one of the way how we do the scaling.
But again, if I take this particular neural network, okay, let's say this is my neural network.
This is my input layer, this is my hidden layer and this is my output layer.
Now, you know that this is connected to this.
This is connected to this.
And I have this right now here you will be able to see that if I am doing all the normalization with
respect to F1 and F2.
Now, you know over here we have w one, we have w two, we have w three, we have w four right now
in the hidden layer.
If I just consider as an example let's let's take this as an example.
Let's say I will just go ahead and take some house size right area, uh, rooms.
How many number of rooms are there?
And this will basically be my price.
Okay.
So let's consider this three features I have.
Okay.
The first feature with respect to the house size is let's say 1200 square feet.
Now after I probably do the scaling, let's say I have some values like one, I'll just put some smaller
value okay.
Let's say 0.45 okay.
Rooms I will just convert this into 0.55 and price I will keep it like that.
Only 45 lakhs okay.
Then again this can be 0.60.
This can be um 0.20.
This can be some other value.
Similarly, I'm just putting some values over here okay.
After doing the scaling.
And this is my f one, this is my f two I'm sending this value okay.
Now you know that when I'm sending F1 and F2 right.
So first for a first feature I have given over here as 0.45.
And this is my 0.55, right.
Now what is basically happening over here.
The first what I will do for this particular node.
You know, we will go ahead and see that, uh, the first Z one that we will go ahead and compute.
And here you know, that a bias is added B one.
And here bias two will be added.
And top of that activation function will appear applied.
So inside this z one we will try to apply an activation function.
Let's say I'm going to apply a sigmoid activation function.
So what is my first feature.
One value.
It is 0.45 multiplied by okay w one okay.
So this basically becomes my w one.
Then 0.55 multiplied by what is this W three, right.
Once we do this, then we are adding a bias, right?
We add a bias that is b one, right?
Now once we do this entire calculation we pass it to the sigmoid activation function.
We get some value.
We will get some value.
Over here you can go ahead and calculate the value if you want.
Okay.
Well you can initialize any weights and all okay.
Now similarly I will go ahead and do it for z two.
Right.
Z two is the output of this.
So z one is here and z two is over here right.
So in the case of z two again I will go ahead and apply sigmoid activation function with respect to
that whatever value I'll get here I'll add a bias and this will be my another value okay.
So here I'm getting my value one.
Here I'm getting my value two.
Okay.
if I probably consider this in the form of matrix, uh, it is very simple.
How many weights are basically getting created?
Two.
Cross two.
So here the output that I'm actually going to get is two cross two.
Right.
So here you can see two two right.
So two cross two will be basically my output okay.
Two inputs I have.
So this is one cross two.
Then um uh with respect to this you'll be able to see that I'm passing it through all the weights over
here so it will become a two cross, two matrix.
Okay, so this is fine.
Let's discuss more about this okay.
Now let's observe this.
So this is my value one.
This is my value one.
This is my value two.
Now when I have this value one and value two initially when I gave this F1 and f2, obviously we have
done the normalization.
but after doing all this kind of multiplication, don't you think this value one and value two value
will change when compared to the distribution of f1 and f2?
Right.
Whatever multiplications that we are doing over here.
Does this distribution may change.
So here let me just go ahead and write it over here.
Z one is nothing but some value.
Z two is some value.
Okay.
Similarly for all the other inputs I'll be getting some value.
So whatever inputs, whatever calculation that we are getting over here, don't you think this will
be.
This may have a different distribution when compared to this feature one and feature two right.
It may have a different distribution.
And that is what is the problem.
Right.
Because the distribution is getting changed.
Right.
It may it may have any kind of distribution over here.
This and this distribution will definitely get changed.
Right.
But and for most of the problem statement we feel that okay this distribution not be should not be getting
changed when compared to your input features distribution.
Right.
And it is always good to have a normal distribution at each and every place.
But here what is happening because of so many different types of operations that you have done, the
entire distribution is getting changed over here.
Now, because of this, what we have to do is that in order to make sure that the distribution does
not change very much in a, in a, in a way that it impacts the entire training of the artificial neural
network or any neural network.
What we do is that we further go ahead and perform normalization on top of Z1 and Z2.
So here we go ahead and provide, uh, perform normalization.
Normalization in every in every output z one and z two.
And this normalization is basically called as batch normalization.
So in the case of batch normalization we take this entire values of z one.
We normalize it.
How to normalize it I will talk more about it I will just show you the formula over here okay.
And this Z two.
Um so for this Z one if I really want to normalize it, what I have to do, I have to go ahead and compute
the mu, the standard deviation, let's say mu one and standard deviation.
For this.
And for every number we will go ahead and apply the formula okay.
And similarly for z two if I really want to normalize it again I will go ahead and compute the mu two
and sigma two uh standard deviation of two for that particular z two.
And for every number I will go ahead and apply my z score.
Right.
And that is how the entire normalization probably happens, right.
By by doing the normalization again, our distribution will start matching this particular distribution.
Because again, what we are doing when we do some kind of normalization over here, with the help of
some z score value, you'll be able to see that my mean will again be coming to zero, and my standard
deviation will be equal to one or near to one.
And when we are doing that, that distribution will start matching with your features one and feature
two.
Okay.
So this is batch normalization.
Now what is the difference between batch normalization and layer normalization I will talk about it
okay.
So here let's say I will also go ahead and talk about batch normalization versus layer normalization.
Because in if I probably consider my neural network okay.
Because if I consider transformer.
So in transformer specifically we apply layer normalization okay.
So let's say over here I have my feature one my feature two okay.
And this was my Z one output.
This was my Z two output.
Right.
I had some values over here.
Right.
So I had some values over here.
So batch normalization says hey go ahead and normalize each and every value like this.
Right.
This was batch normalization.
Layer normalization says hey, instead of taking the complete Z one, what you do is that you take layer
by layer, you take layer by layer, and then you normalize them.
So you normalize in this way.
For this you go ahead and compute your mu one sigma one and apply your z score with respect to x of
I minus mu one divided by sigma one x of one will be this X1X2 will be this.
Similarly for this you go ahead and calculate your mu two sigma two.
Then you go ahead and calculate your z score.
That is nothing but x of I mu two by sigma two.
And similarly for this you go ahead and compute mu mu mu three and sigma three.
And for this you go ahead and apply your z score x of I mu three divided by sigma uh, divided by sigma
three.
Okay.
And this way you specifically normalize n layer normalization.
This is the basic difference between batch and layer normalization.
So guys I hope you got a basic idea with respect to batch normalization versus layer normalization.
Now let me talk about two important learnable parameters that is called as gamma and beta okay.
These learnable parameters are also used while we are training our models okay.
Now what are these learnable parameters.
White is specifically used.
We will discuss about it.
But before that what is the use of layer normalization that will try to see let's say over here I have
z one, I have z two okay.
And this z one and z two is what it is basically the output from here.
Right.
So here we are basically talking about right.
And like that we can have any number of hidden layer for every hidden layer.
Also it is a good idea that we actually perform batch normalization okay.
Now let's say with respect to z one and Z2I have different different values.
And with respect to this let's say I'm also having some zero values okay.
Let's say over here also I'm having zero values okay.
Now for this values that I have.
What is the use of doing layer normalization.
See in batch normalization what we do we take this entire z one output z two output.
When I say z one output.
Uh see I'm basically considering the output of both of them okay.
Okay.
Both of them?
In short, uh, sorry, in the form of this particular matrix.
So Z one has one side and Z two has the one side.
Right.
And it is in the form of matrix itself okay.
Now understand one thing over here okay.
Very simple.
If I have many number of zeros over here if this are zeros basically the output is not coming up anything.
Right.
When we have this kind of normals uh, with when we have this kind of numbers.
So here we are definitely performing batch normalization.
Right.
We are taking this entire batch and we are normalizing it okay.
Now once we are doing this you know that most of the values are zero.
But for all these zeros values also we are still normalizing it normalizing it.
Right.
And we are normalizing how by calculating your mean value sorry your mean value for this.
Okay.
And then along with this you are calculating your standard deviation.
And then you are applying z score for each and every number.
Now when you apply the z score for each and every number, this values are also going to change right.
Some other values will definitely come over here.
Similarly over here you have this zeros.
Now what is basically happening.
This zeros is impacting the entire distribution.
So this entire distribution will get impacted because of this normalization.
Right.
So usually in an and all we definitely apply batch normalization.
But in transformers that we have seen we specifically go with layer normalization because in transformer
we have this MLP kind of use cases where every word is specifically important considering the context
of all the other words.
Right.
So what we do in layer normalization we go ahead and take each and every combination.
We basically take each and every layer's combination.
And then we go ahead and normalize it.
Now let's see if I have zero comma zero.
Even though we go ahead and normalize this I know I'm going to get the mean is equal to zero for this.
If I go ahead and calculate standard deviation is also zero.
So no much impact no values is going to change.
Similarly over here.
Also no values is going to change.
Similarly if I have zero over here no values is going to change, right?
This is the most important thing and the basic difference between layer and batch normalization.
Now one more question that comes in mind Chris.
Is it compulsory that we always have to do this normalization?
What if the final output of z one and z two that we are getting and this distribution may be important,
may be important for predicting the output and that point of time.
What if I do not want to do the normalization?
Okay, because I'm considering hey for my problem statement whatever distribution I'm getting in z one
and z two, that is perfect for my execution.
Let's say for my solving my problem statement.
So if I do not want to normalize this.
So for that we will be using two learnable parameters that is gamma and beta.
And this learnable parameter will be specifically learnt when we are training the entire neural network.
Right.
So now let me just go ahead and write again.
So Z one if you are calculating it will be some function.
Let's say I'm going to go ahead and apply my sigmoid over here.
Right.
And this will be w transpose x whatever transpose we are specifically doing with B1 right, or whatever
B1 weights we are specifically getting after this step, what happens is that we assign learnable parameters
okay.
And this learnable parameters, I'll just write it as y, y will be my output.
And here I will be using gamma okay gamma.
And then I will go ahead and use this x uh Z1Z1 okay.
Minus mu one.
Z one basically means all the numbers over here.
We are going to subtract with minus mu one, which we are going to calculate it.
And then along with this gamma parameter we are also going to use one beta parameter.
And these are the learnable parameters.
These are the learnable parameters.
Now you can see that right learnable parameters.
Let's say if our distribution is not important over here and we do not need to normalize it right then
based on this gamma and beta value, we can make sure that we can tell, hey, do not normalize in this
scenario, or do not or normalize in this scenario based on the data that I have.
Right.
So that is the reason we use this gamma and beta value.
I will be showing you this entire example by solving a simple problem also as we go ahead.
But I hope you got an idea with respect to your layer normalization and batch normalization and how
this entire layer normalization is probably taking place.
And we specifically use this two important parameter that is gamma and beta okay.
Now I hope you have understood till here.
Now we can go back over here and we can see why add and normalize is basically coming over here.
Because after this particular output that I get, okay, it is really important to understand, okay,
after this particular output that I am actually getting, once we combine this entire entire output
with respect to the sequence length that we are getting right, this output is going to get added with
this plus this, which I will mention it as x dash.
So x dash is going to get added with this entire values right.
And then we are going to apply our layer normalization.
Right.
So here we are going to specifically apply our layer normalization okay.
Perfect.
So that is what is all about layer normalization.
Now what I will do I will go ahead and solve a basic problem, okay.
Just by uh, putting some numbers.
And when we use this gamma and beta this are also called as scale and shift parameters.
So this gamma and this beta is also called as scale and shift parameters.
Okay.
Now let us go ahead and let us take a simple problem.
Okay?
So I will take a simple problem where I will take an example of Cat as a token.
Let's say I will denote this cat as token embeddings like 2.0, 4.0, 6.0, 8.0.
Let's consider that this is my four dimension and the next parameters that I'm actually going to define
is my gamma and beta.
Because these are the learnable parameters.
Initially this will be initialized.
Later on we'll train them.
We learn them 1.01.0.
So let's go ahead and initialize this to 1.0 and beta.
I will go ahead and initialize to 0.00.00.00..
The reason is very simple.
I have seen the research paper and there also they have initialized this.
Okay.
Now we will go ahead and perform step by step calculation, like how specifically it happens with respect
to the transformer and how this entire process basically happens.
Okay.
Um, so we will just go ahead and apply this normalization.
We will try to calculate.
We will try to find out the normalized vector.
Then we'll do the scale and shift.
And then we will finally get our y value okay.
So this entire computation that will do, uh, remember one thing over here.
I'm not taken w transpose x.
Instead I'm directly taking this particular vectors and how one vector gets normalized.
That is what I am actually planning to show you how this vectors will get normalized.
Okay, normalized.
That is what I'm actually going to show you over here.
So let's see this in the next video.