Hello guys!
So finally we are into the last stage of understanding about Transformers.
I know this entire series now probably is more than five hours of content, but you can just understand
why this entire architecture is so amazing, right?
It looks complex, but if you break down this entire architecture, you will be able to understand each
and every thing.
And that is what, while explaining all these topics I have actually done right.
So from starting from positional encoding to multi-head attention to feed forward neural network here
also we discussed about masked multi-head attention.
We went to Multi-head attention.
We also covered about feed forward neural network.
Now it's time to understand about the final layer of the decoder.
Right?
That is nothing but linear and softmax layer.
And through this we basically go ahead and calculate the output probabilities.
Now I will try to just make you understand again, this entire diagram is from that amazing blog right
about Transformers.
And what I will do is that I will also go ahead and share that link with you all.
Okay, you can go ahead and refer it.
So till here, uh, what we have actually seen, we have seen all the kind of operation related to q,
q, k, v here, q and k is basically going V is basically coming from here.
And again we also have lot of multi skip connections like from here to here.
We are passing uh some information regarding positional encoding then other information.
I'm passing it over here.
Now let's finally do one thing.
See guys after this decoder is completed, right at the end of the day, you are just going to get vectors
right now, how do we convert this vector into a word?
That is what we are going to see in this session, right?
So our main aim of this particular session is that how we will be able to convert our vectors into our
output word.
And that is what I am going to discuss step by step.
Now, first of all, here you will be able to see there is something called a linear and softmax.
First of all, we'll try to understand what exactly is this linear.
So to begin with this linear or I will just go ahead and write the linear layer.
Is a simply is a simple fully connected neural network.
That projects the vector produced by the stack of decoders.
Okay.
Stack of decoders.
And it basically generates a very large vectors, which we also called as as log vectors.
Okay.
Logits vectors.
Now I'll talk about what is the importance of this logit vectors.
And this is what a linear layer basically does.
Okay.
Linear layer basically does.
So let me go ahead and explain a linear layer is a simply, fully simple, fully connected neural network
that projects the vector produced by the stack of decoders.
Right.
And it basically generates this into a large vector which is called as logit vectors.
Now what exactly is this logic vectors.
Okay.
So we'll try to understand this.
Let's say if our model if our we have a specific model, if let's say, uh, our model knows more than
10,000 words, right.
Let's say that there are 10,000 unique words.
And this is my entire vocabulary.
Right.
So in this case my logit vectors will be 10,000.
Logit vectors will be 10,000 cells wide.
So here, instead of just having this many vocab size, we'll be having this 10,000 vocab size okay.
Or logit size.
Now understand each and every block that you see in this logit vectors.
It corresponds to a score of a unique word.
So here this specifically corresponds to a score of a unique word.
What kind of score?
We will talk about it, right?
It's just like mapping a vector to words.
Right?
So for this, once I give once, I probably get all the vectors from the stack of decoder.
I'm going to pass through this through the linear layer.
It is a it is a fully connected neural network layer.
And once we pass through this, this is going to give us an output of this much nodes.
Right.
This many number of outputs of a vocab size.
So number of words vocab right.
Uh so again uh over here you'll be able to see it will take this particular vector and it will convert
this into logit vectors okay.
Uh, then if you go to the next step it is a softmax layer.
Now you know why softmax layer is used.
It is specifically used for multi-class classification.
Multi-Class classification.
Now in the case of multi-class classification, what will happen as soon as I pass this particular vectors?
Every vector is basically going to give us our log probabilities.
Right now this probabilities means what?
Let's say if this vector has the highest probability right?
Or if this word has the highest probability, then my output will be this specific word.
If this word has the highest probability, then the output will be this specific word okay.
So here in the second step what we are basically doing is that the softmax layer turns those scores
into probabilities.
Now, since it is turning this into probabilities, if I probably do the summation, it will all add
up to one, right?
That I hope everybody knows about because that is the property of softmax layer.
Then the cell with the highest probability is chosen as the output is chosen.
Right.
And the word.
Associated.
With it is produced as the output.
As the output.
Right.
For that specific timestamp.
Okay.
Or for for that specific timestamp.
Now I hope you are able to understand it okay.
So whatever output I get from the decoder that is passed to the linear layer, which is a fully connected
layer, and this output will be based on a vocab size that many number of cells.
And once this is passed to the softmax, I will be getting a probability, and whichever has the highest
probability, that will be my output word.
Okay, uh, when we say vocab size, vocabsize basically means if there are 10,000 words for every index,
there will be one word.
Okay, so more to talk about.
Let's go through the entire training process now.
So here I will talk about the recap of training.
And here also I will take multiple examples.
Let's say in my entire vocab size I have this many number of vocabs.
Let's say my this many number of words are there.
So in my vocabulary I have a am I thank student iOS.
Okay, so that is the reason I've given this as indexing.
Okay.
Now the first for the first instance, what we do, uh, if I don't just use a simple embedding technique.
Okay, I will just go ahead and use this simple embedding technique.
So for every word, uh, what we will do is that we will try to use one hot encoding to represent it.
So let's say this is my one hot encoding okay.
This is my one hot encoding for the word m okay.
So wherever m is present since the vocabulary size is six.
So I will write 0.0 where m is present, it will become 1.0, 0.0, 0.0 and 0.0 okay.
Now So during the loss function, what will happen if I have started training the word okay if I have
started training the word.
And similarly like if I want to have thanks, then this will become one and remaining all will become
zero.
Let's say that I have a problem statement where I need to convert this mercy, right?
Mercy.
I think it is in French.
We need to convert this into thanks.
Right.
And vocabulary size is over here for me.
Okay.
So what will happen if I have an untrained model, right.
My initial output may be something like this.
See, I need to get this word right.
Thanks.
I need to get this value as 1.0.
But if we see with respect to the untrained model initially, you'll be able to see these are different,
different values.
I may get different different values for each and everything.
Okay.
So this is how my output should come.
But this is how I'm getting the output right.
So after this what we do we basically go ahead and calculate the loss function okay.
Loss function.
Now this loss function is really important.
That is where your back propagation will specifically happen okay.
And just to make you understand uh, let's say I want to completely frame this particular sentence okay.
So just uh, I'll take an example.
Okay.
I have a task where I give this words and it should probably form a sentence for me.
Okay, so here I have a m, I thank student if I really want to know the output, it should be like
I am.
I am a I am a student and thanks.
We can basically write right?
I can basically say thanks, okay?
Or let's say I don't want to use thanks or even because thanks will not be a very good, uh, thing
in that particular sentence.
Let's say if I give this particular word, I need to frame a better sentence.
So here I can frame one kind of sentence like this I am a student, and then we should basically get
our us right.
So in this particular scenario what will happen is that for positions also right for position one,
what will be my target model output.
And also the for for the real output.
My target output in my output feature will be something like this.
Right?
So my input is something like this.
Let's see I will just show you over here okay.
Let's say my input over here in my data set.
And this is my output in my input I have words like am I.
Thanks okay not thanks student.
Let's say I will go ahead and write.
Student okay.
Now my output is I will go ahead and write.
I am a student Okay, let's say this kind of output I really need to get now here based on this particular
actual output I should be getting.
Like if I consider my vocabulary size as six, I should be getting something like this right?
My position one should be I.
Position two should be I uh m position three should be uh, position four should be student.
And then finally end of statement.
But what happens is that while we are training, you know, with this particular data set, after some
epochs during initial training, you will be seeing that, hey, uh, my output will not be that good.
But after training for some time, let me just go and show you.
After training for some time, you will be able to see over here.
I will be able to get something like this.
I will become 0.93.
Then this will become .8.9.94.98.
Right.
And this is all because of loss function in epochs.
And this all keeps on getting updated.
Okay.
And that is what I really wanted to show you as an example.
But at the end of the day, you'll be able to see that we will initially be doing encoding for this.
Here we have just used one hot encoding.
Different encoding can also be used, right.
Untrained model may give you some kind of output.
But again we need to go ahead and compute the loss.
And then we will do the back propagation.
And then we will go ahead and compute all the weights.
Right with respect to all your training data.
And all right.
So back propagation the main aim is to reduce the loss.
Right.
So after training from some specific outputs initially my training model output will be like this.
Right.
If I convert each and every word into one hot encoding, one hot encoding, one hot encoding, one hot
encoding, right.
One hot encoding.
And this is with respect to position also in position one.
I is basically having 1.2.
So this is what is the one hot encoding for this.
Then you have this then you have this then you have this right.
So once you do this this is what is my output that I need to get.
But my model when it initially generates it can generate something else.
Right?
Then a loss will be calculated between both of them and it will be reduced till we get this kind of
values.
And, uh, once I get this particular model out and this is after the, uh, you can probably say with
respect to your fully connected layer right over here.
So the linear layer, that is what I'm actually talking about.
Right.
And then once you probably compute the loss you will again be doing back propagation.
So this was uh overall the final linear and softmax layer.
But I hope I tried my level best to whatever was possible.
But just imagine guys 34 pages have actually written just to explain this research paper, broken down
each and every thing.
Uh, tried my level best.
Okay, but yes, there may be again, some things that you really need to revise from the research paper,
but I tried my level best.
Whatever was possible from my side.
Okay.
But I hope, uh, you're quite happy with this explanation.
I hope you are able to understand each and everything.
So, yes, this was it from my side.
I will see you all in the next video.
Thank you.