So guys, now let's go ahead and, uh, let me show you an example with respect to all the steps that
you have discussed in layer normalization.
Okay.
So for this example, as I've already mentioned, uh, we will be taking a single token in sequence
going through a layer normalization process within a transformer layer.
Okay.
So the initial step we have initialized some values over here okay.
Token embeddings.
Let's say this is the token for cat.
It is represented by this vectors.
We have initialized some parameters okay.
And this uh parameter is basically when I probably say right, this is uh, this gamma is your learned
scale parameter.
Okay.
And beta is basically your shift parameter.
So that is the reason we say it as s scale and shift.
Right.
This combinedly is basically called as scale and shift patterns.
Now let's go ahead with the step by step calculation.
So for the first vector first of all we will go ahead and compute the mean.
Because if you remember the z score formula it is nothing but z score is equal to x of I minus mu divided
by standard deviation okay.
So here if I go ahead and compute the mean.
so it will be nothing but one by four.
Let us take all the vectors values 2.4, 4.4, 6.0, 8.0.
And let's go ahead and compute this by 20.0 divided by four, which is nothing but 5.0.
Okay.
Next, uh, let's go ahead and compute the variance.
Compute the variance.
So variance is nothing but your sigma square.
So for that how do we compute it.
It is very simple.
Sigma square is equal to one by four.
Then we take up each and every numbers okay.
And we subtract it.
Let's say the first number is 2.0 -5.0 which is my mean plus 4.0, -5.0, whole square plus 6.0, -5.0
whole square and plus 8.0, -5.0 whole square.
So this is how you calculate the variance.
So overall you will be able to get your variance as five okay.
So now here you can see obviously your mean and your variance looks five five okay.
So now in the third step we are going to normalize all our inputs okay.
Third step.
Let's go ahead and normalize the inputs.
Now in the case of normalizing input it is x I hat which is nothing but x of I minus mean divided by
uh just dividing by this route or see uh, what we will be basically doing is that we will also be dividing
this by standard deviation.
But instead of just writing standard deviation, I will just go ahead and give a new representation
square root of variance plus epsilon value okay.
Now this epsilon value we will be using a very small value.
Let's say one e to the power of minus five okay.
And this is used to avoid division by zero okay.
Okay.
This is basically used to avoid division by zero.
So here if I just go ahead and see and calculate this, it is nothing but root of 5.0 plus one e minus
five, which is equal to nothing but root of 5.0001, which is nothing but 2.236.
Okay, so this is my 2.236 over here.
And this is what is my below value that is coming.
Now we will go ahead and normalize for each and every values with respect to the numbers.
So the first normalized value will be x I bar okay let's say and I will write 2.5 uh 2.0 -5.0 which
is my mean.
And here we are just going to divide by 2.236.
This is approximately equal to 1.34.
Similarly x two bar vector that we have I will subtract 4.05.0 divided by 2.236, which will be approximately
equal to -0.45.
And similarly coming to the next vector that is x three bar third vector.
Here again we will go ahead and write 6.0 -5.0 divided by 2.236, which will be approximately equal
to 0.45 and x4 bar will be nothing but 8.05.02.236, which will be approximately again equal to 1.34.
Okay, so these are my outputs.
And this is my normalized vector.
So here I have my normalized vector.
But still we need to add something very important.
So normalized vector my x bar will be nothing but -1.34, -0.45
.451.34.
Okay.
Finally let's go ahead and apply our fourth thing.
Still we have not applied the beta and gamma parameters, which is like scale and shift.
So here we will go ahead and apply scale and shift.
Now with respect to scale and shift, I will go ahead and write y of I gamma phi x of I plus beta of
I.
Right.
You know, your gamma uh, gamma is nothing but 1.0, 1.0, 1.0.
So we are just going to scale and shift.
Okay, beta is nothing but 0.00.00.00.0.
Okay.
And again, in order to just do the scaling and shifting, obviously just by seeing this particular
values, you are going to get the same x bar or final y output, which is nothing but -1.34, -0.4,
5.45 and 1.34.
So in this scenario, when we initialize this particular value and this is what I explained.
If I say, hey this distribution is important, I will just this value will be remaining like this itself
and we will be getting the same value.
Let's say if you want to change this distribution, some more values will be separated present over
here.
And based on that we can multiply and add that particular values.
Right.
So this was the entire understanding about how a layer normalization work.
And this we have specifically applied to a specific vectors I know on this vector we apply positional
encodings.
Then along with that we go ahead with self attention multi head attention and all.
And then uh finally whatever output we are basically getting we add it right.
Uh at the end of the day from this particular architecture that we see.
Now similarly, once we do this normalization, you can see over here we are passing this information
to feed forward neural network.
But before that again we are going to pass this entire signal back to the next add and normalization
where I will be adding this output again back to my output of the feed forward neural network.
And again we will be performing normalization.
So after doing this, this entire information will probably go to the decoder which we'll discuss in
the later stages.
But I hope you got an idea with respect to the entire working with respect to add a normalization Multi-head
attention, positional encoding, input encoding, almost each and everything has been explained.
Now in the upcoming video, we are going to discuss about the encoder architecture, what is remaining,
and how the entire other operation is basically performed that I will be discussing in my next video.
Thank you.