Hello guys.
So we are going to continue our discussion with respect to the Transformers.
So this is the first topic that we are going to pick up and we are going to understand.
So here uh I've also given the definition.
So that is the reason I've written what and why in Transformers.
So if you really just want to understand a simple definition about this particular architecture, it
is nothing.
But in transformers and natural language processing are a type of deep learning model that uses self-attention
mechanism.
Please keep a note of this because this is really important to analyze and process natural language
data.
They are encoder decoder model that can be used for many application including machine learning translation.
Now if I talk about machine learning translation, this is specifically something called as sequence
to sequence task.
Right now when we say sequence to sequence tasks, uh, let's say I want to solve a problem statement.
Uh, one of the example that we can probably take is like language translation, right?
Language translation.
And if you probably go ahead and see an example of Google Translate, right.
This is a task of sequence to sequence uh, sequence to sequence task.
It is a sequence to sequence task.
Because if I really want to convert from English to French, write English to French.
So in this scenario you will be able to see that my input, uh, with respect to English will be many
words.
I will be having many inputs over here, it will be many.
And my output will also be consisting of many words.
So this basically becomes like a many to many, uh, sequence to sequence task right Right.
Now when we have the sequence to sequence task, and obviously length of the sentences is also a meaningful
thing over here.
Right.
Length of the sentence.
Now, as the length of the sentences increases, we should be able to solve this particular problem
again with a very good accuracy.
So in this kind of task usually transformers is used.
Okay.
But you may be thinking, Krish.
Fine.
You have also told that, uh, in the previous architecture of encoder decoder, encoder decoder, you
told that this is also used for sequential sequence task.
And if I talk about encoder and decoder like they were, uh, if I just probably go ahead with this
particular diagram.
So I had an encoder over here.
Right.
And I had a decoder over here.
Right.
So in encoder and decoder, what we used to do, we, we had our LSTM right over here.
And based on this LSTM when we used to go ahead and pass this entire sentences okay.
Please remember we do not take the output in this.
Right.
So if I just go ahead with a basic uh, architecture of encoder decoder, we'll be able to see that
if I used to give my words over here, let's say X1X2X3.
And this words we used to give based on timestamps t is equal to one, t is equal to two, t is equal
to three.
So let's say if this is my sentence one which is having this particular words.
And again we used to convert this entire words by using some embedding layer okay.
Convert this into vectors okay.
And then pass it to the entire, uh, LSTM.
So this was my LSTM over here.
Right.
And this is my encoder and this is my decoder.
Right.
So after doing this finally you should see that we were able to generate one C vector that is context
vector right.
So here I will just go ahead and generate this context vector.
And based on this context vector we used to provide it to our next decoder layer.
Right.
Next decoder layer which used to have this LSTM.
Okay, so here is my decoder completely I will pass it over here.
I'll pass it over here.
And then uh, with respect to this particular decoder, I used to also get my output.
And here we used to use some kind of softmax.
Right.
Now this was the basic architecture of the encoder decoder.
Right.
And this is what we used to do.
This context usually used to define when we are passing every words based on the timestamp, the final
context that we used to get at the end of this particular sentence used to represent this entire sentence
itself, right?
Which is consisting of this entire words.
And this context was further sent to our decoder model.
And this decoder model would use to do the prediction based on this context that we have.
And along with that, we used to also calculate the loss.
Everything was discussed right, based on the encoder architecture.
But one problem we understood from the encoder decoder was that this context was not sufficient to represent
this entire sentence.
If the sentence length increases, if the sentence length increases, if it.
This increases.
The sea context was not at all sufficient, right?
And this was the problem with respect to the encoder decoder architecture.
And that is the reason you could see that our blue score was decreasing as the length of the sentences
was increasing.
So this I have covered in my previous video.
I would suggest if you want to quickly revise it, you can go ahead and do the revision.
Okay, now for this.
We also had seen an architecture.
So here was my sequence to sequence learning.
This was the entire research paper of this.
You know how the entire working actually happens.
Each and every thing was explained clearly over here.
Right.
So after this, what we did is that in order to solve this problem of the context that we had, like
our main idea was that, okay, we should not create the entire context at once and probably send it
to the decoder model instead.
What we can actually do is that we can use something called as attention mechanism, and that is where
this research paper came.
Right.
So based on this research paper, what we exactly did was that, uh, we the plan was very simple.
Instead of just giving a single context, we have to actually provide an additional context to our decoder.
And this was the architecture we specifically discussed about.
And if you remember in our previous videos, we have discussed the entire mechanism of how this attention
mechanism actually works.
Right?
And we discussed about this entire research paper over here.
So here was the entire working, wherein along with a single context, we also have to provide additional
context, create our alignment scores, create our attention weights and then pass it to the decoder.
Okay.
And that is what we specifically solved it with the help of attention mechanism.
So let me do one thing.
Let me just take this screenshot okay.
So that you will be able to see this reference over here.
So here I will just put this reference.
And here also I have added this additional reference.
So this was all possible because of your um you know the attention mechanism.
Right.
So this is the entire working, right?
And with the help of this, what we were able to do is that we are able to provide additional context,
additional context to the decoders.
And then this decoders, what they were able to do is that they were able to do the prediction.
And the problem with respect to the long sentences.
right.
The problem of accuracy that we are facing, it started increasing, right, because of this research
paper.
But still, uh, let's understand with this attention mechanism.
Okay.
So here we are using bi uh bi directional LSTM RNN here.
Also we are using LSTM itself.
One problem that we see with this attention mechanism or encoder decoder okay, attention mechanism
or encoder decoder is that we pass every words based on timestamp, right.
So here parallelly, parallelly we cannot send all the words in a sentence.
right?
And again, because of this, because of this C over here, you'll be able to see in even in encoder
decoder, we use to send the words based on timestamp.
Right now when we are sending the based on timestamp at t is equal to one, I'm sending one word, t
is equal to two.
I'm sending another word right.
Similarly over here in this bidirectional LSTM.
Also I'm sending one word at a various timestamp.
We won't be able to do the entire execution or this entire training parallelly, right?
And because of this, yet your attention model is not scalable.
When I say scalable, that basically means if my data set is huge, right?
Still the encoder decoder attention mechanism will not be scalable with respect to training, with respect
to training.
And this is really important to understand because the new models that we'll be seeing with respect
to Transformers.
Transformers uses something called as they don't use they don't use this.
They they never use this LSTM RNN in encoders or decoder.
Right.
What they use is that they use this something called as self-attention self-attention module.
And because of this self-attention module, you'll be able to see that all the words, all the words
will be parallelly sent, parallelly sent sent to the encoder for further processing.
Right.
And that is what this architecture comes into existence right over here in the input embeddings.
Right?
You're sending all the input all at once.
So it is supporting this entire parallel execution.
And that is where because of this we'll also be learning about one more topic which is called as positional
encoding.
Positional encoding.
We'll discuss about this as we go ahead.
Right.
And this will also play a very important role for this when we are sending all these words parallelly
and how each and every word vectors is going to get computed, we'll discuss about this, what exactly
this self-attention module will be doing.
We'll be discussing about it right now.
You should definitely understand over here.
Is that in attention mechanism encoder decoder?
We are not able to do this, hence the word scalable.
Now when we say scalable, that basically means why this transformers are really performing well is
that as we keep on increasing the data set, you will be able to see that we are able to get some amazing
models, amazing state of the art models.
I say Sota models, right?
So state of art models we are able to get specifically with respect to NLP task.
Now this is not just restricted to NLP task.
Um, with the help of transfer learning, right?
With the help of transfer learning, now transfer uh, transformers are even used in multi-modal okay.
They're used in multi-modal tasks.
Multi-Modal tasks basically means task.
Uh, that has both NLP plus image, you know, so they're here also they are able to perform really
really well okay.
How it is able to perform and all we will understand once we understand the entire architecture.
But this is the major, major important thing that you really need to understand why Transformers what
was the problem that we had in our previous models like encoder, decoder and all?
And this is one of the very important problems that we have.
Right.
right?
Uh, so just defining all these things, uh, since it solves all these problems, you'll be able to
see now, everywhere.
Everywhere.
This entire Transformers.
Have really changed the AI space.
Now you'll be able to see a lot of Sota models.
Right.
And this Sota models is nothing like, let's say, some of the transformer models that we have, which
is already trained in huge data set, is like Bert GPT right now, what companies are doing, if they
really want to create their own model, they don't have to train it from scratch.
And these models are trained with huge data, trained with huge data, right?
They can directly just use transfer learning and with the help of this transfer learning they are creating
this amazing sort of models.
Right.
State of the art models.
Right.
And the architecture this is completely based on Transformers itself.
And as I said, multimodal task.
The same architecture is also used with respect to images.
So if you see in OpenAI some of the application like Dall-E, right, they actually just based on a
text, they are able to generate the images right entirely.
It is completely based on transformers.
So just to give you an idea, uh, this will also be very much important, uh, because based on this
architectures, various LLM models are also used in generative AI.
Okay.
Generative AI Li LM lm basically means large language models.
So I hope you got an idea.
Just take this into consideration.
This is a very important thing that we have discussed parallelly.
We cannot send all the words in a sentence in attention mechanism or encoder decoder, but with the
help of transformers we are not at all using LSTM RNN.
Instead we are using self-attention module and this self-attention module functionality will be that
we will be able to send all the words right Parallely uh, for the further processing, what exactly
happens in the self-attention module?
We'll discuss more about it.
Okay.
Uh, so I hope you got an idea about this.
One more very important problem that we see in, uh, Transformers.
That is with respect to contextual embedding.
Contextual embeddings.
Right.
Now what exactly happens in contextual embedding?
We will discuss about this okay.
Now with respect to the contextual embedding, let me just give you an example.
In order to make you understand this okay.
Let's say I have a sentence.
Uh, my name is Krish and I play cricket.
Okay, now let's say over here when I pass this entire information right in my embedding layer.
Let's say this is my embedding layer.
Now, you know, in embedding layer when we pass it, our main task will be that from this embedding
layer for every word, we should be able to get our vectors right.
Some vectors.
So if I pass my then name I should be getting another vector is I should be getting another vector crush,
I should be getting another vector, and I should be getting another vector based on timestamps.
I will be getting this kind of vectors and understand if you are specifically using embedding layer.
And let's say if this embedding layer is using some some embedding word embedding techniques like word
two vec for every word we are going to get a fixed vector.
Okay.
So this if my is there then I will be getting a fixed vector of some dimensions.
Similarly name is there.
I will be getting a fixed vector.
Okay is is there?
I will be getting a fixed vector.
Krrish is there.
I'll be getting a fixed vector and is there?
I'll be getting a fixed vector.
I will is there.
I'll be getting a fixed vector right.
And similarly play is there.
I'll be getting a kind of vectors itself right.
All these words will be represented in some kind of vectors.
Now I need to make you understand about contextual vectors.
What exactly is this contextual vectors.
See, it is always a good idea that if you are able to get a vector, it is fine, right for every word.
But we should always try to get a vector whenever we have a longer sentences based on the relationship
with other words.
Now here you can see my name is Krish and I play cricket, okay?
I is obviously related to Krish.
There are some kind of relationship, right?
So if I probably give this same word.
On my contextual vector embedding.
Contextual embed vector embedding.
Then all the vectors that I will be getting.
This vector should be having some kind of relationship with respect to the other words like Christian,
I hear there is a very strong correlation, right?
So some context will be there.
I play cricket, so cricket is also there.
So this cricket vector, I should not be getting a fixed vector itself.
I should be getting a vector which should be related to crush.
Based on this particular relation, there should be some changes in this particular vector which suggest
that hey, there is some contextual dependency in this particular word.
right?
And this entire problem is solved by our self-attention.
And because of this, you'll be able to see that our transformers will be -- more accurate.
Right.
And we'll be talking more about the self-attention module in the upcoming video.
But if I talk with respect to the most two overall problems, why Transformers are specifically used.
One is here in Transformers Parallel, you can send all the words for processing, because of which
it makes the entire model scalable with respect to training with huge data sets.
Okay.
The second thing is that it has this contextual embedding thing, right?
Which was basically missing in encoder and decoder because the encoder and decoder, we just used to
use an embedding layer used to pass each and every word, get a vector, and further do the processing
with respect to this.
But now, with the help of the self-attention, you'll be seeing that we'll be able to even create contextual
vector embedding.
Now in our next video, we are going to deep dive more into this and try to understand how this entire
architecture basically works.
And step by step, we'll try to get each and everything as we go ahead.
Right.
So yes, this was it for my side.
I will see you all in the next video.
Thank you.