Hello guys!
Before we start implementing some of the amazing generative AI applications with the help of large language
models or multimodal, it is very much necessary that you really need to understand how these LLM models
are specifically trained.
What is the step by step mechanism to properly train this particular model completely from scratch?
Yes, with respect to practical implementation, it will not be possible because you definitely require
a huge amount of resource data and many more things.
But in this video, what I'm actually going to do is that I'll be taking some of the LM models, like,
uh, OpenAI ChatGPT models.
I'll be talking about meta, uh, meta llama three models and.
All right.
And when I read this research papers of all the specific models, they usually follow a specific pattern
for training all these LM models from scratch.
And in this video I will be talking about this entire steps.
Uh, step by step will try to understand it completely.
Right again, uh, the main aim, the main goal of this specific video is just to make you understand
how this, uh, you know, LM models are basically trained, and this is just a theoretical intuition.
Practical intuition will not be definitely possible where you will be training your model from scratch.
It definitely requires a lot of resources.
Right.
So, yes, uh, let's go ahead and enjoy this particular session.
And please make sure that you watch this session till the end because it is important.
Thank you.
Here I have actually created all this diagrams to just make you understand.
But, uh, just to understand how ChatGPT was trained, you know, I have I have explored lot of research
papers from a past month, you know, a lot of different resources, materials that are available in
the internet about ChatGPT blogs or even I have explored the ChatGPT websites.
The research paper that was basically generated for this, everything I explored and I found out multiple
things to probably explain about this.
I'll try to break this down, but before I go ahead, there is an amazing article written by Pradeep
menon.
You should definitely check out this.
I will be providing you the link in the description of this particular video and it looks completely
amazing.
He has explained in an amazing way.
And this last two diagrams that you will be seeing over here, right?
If you probably explore this, this last two diagrams have also copied and pasted.
So definitely a lot of credit goes to uh, to this specific article, uh, which is written by, um,
you know, Pradeep.
So I will just try to explain you again, just but not, uh, if you probably read this, you may get
some amount of understanding, but, uh, currently I will.
What I will do is that I'll give a lot of examples over here, right.
So that you will be able to relate it like how chat GPT is trained.
Now to start with chat, GPT entire model is basically trained in three stages.
Okay.
The first stage we basically say it as generative pre-training.
Okay.
And before you probably really want to understand about ChatGPT, you should also check out my video
in YouTube regarding Transformers.
Okay, because this is the base.
So if you search for Transformers, right?
So here you will be seeing that I have created a live session over here.
This specific video should definitely refer to, because this is the video that I probably uploaded
two years back.
And if I talk about models like ChatGPT or Bart, you know they are specifically using transformer architecture,
which has an encoder and decoder.
Okay, so definitely check out this particular video if you really want to know about Transformers.
Okay.
So in the stage one you probably train a generative pre-trained model which through which you get a
base GPT model.
Then in the stage two you do supervised fine tuning.
I will be talking about each and every thing of this, like how this generative pre-training is basically
happening, supervised fine tuning is happening.
And then finally in the stage three, you basically do the reinforcement learning uh, by using the
human inputs.
Right?
So that kind of uh, not human input, but I'll say say human feedback.
Right?
So that is the reason it is written as reinforcement learning, human feedback.
And finally you get a ChatGPT model.
Now this first step that you probably see the stage one, right.
Generative pre-training.
What kind of data is basically required from the entire internet data.
Right.
It can be website articles, books, public forums, uh, like uh, a website, which is probably having
tutorials, a lot of things.
Right.
So those are all internet data is basically taken to train the ChatGPT.
So let's go ahead and deep dive and talk about this okay.
But let me just give you a basic example like what a learner model looks like.
Let's say that I am a person over here and I am really much interested to know about dogs.
Okay, so what I did is that I have explored five, six different types of this big, huge 500 pages
of books regarding dogs.
And I have learned I have probably read it about it.
So I know many things about the dog.
Now you can ask me any questions, so I will try to answer any new thing about that specific dog that
is only present in the book.
Okay, that is only present in the book.
So this gives a brief idea about a model where you are able to answer something after probably getting
trained or reading it from multiple books.
Okay, so this is just an example.
But when we talk about the stage one generative pre-training model.
So what exactly is basically happening over here.
So first of all let me make this as a full screen here.
You have a internet huge input data okay.
So here you basically have an internet huge input text data.
You pass this data to the transformers okay.
Transformers, when you pass this huge amount of data to the transformers, in transformers you have
encoder decoder.
I've explained about transformers a lot in my live session.
Please do make sure that you watch that particular video after you after this particular data.
This huge data is basically trained with the transformers.
It basically creates a base GPT model.
Now, as you know what all tasks a transformer will be able to do it transformer can uh, like they
are able to do this kind of task like language translation, text summarization, text completion,
sentiment analysis.
So this is the kind of task that Transformers can easily do.
And just by training this huge amount of data, we are able to get this kind of task and we are able
to solve this problems.
Right?
Even, uh, in Transformers there is a concept called as attention is all you need, right.
And based on that, all this task can be easily implemented.
And I've also shown a practical implementation with respect to that particular live session also.
Right.
Uh, just in, uh, Google Colab notebook.
Right.
If you have good amount of data, you will also be able to implement this.
Okay.
So once this task is implemented, but our main aim is basically to use this ChatGPT model for conversation.
Right.
What we want we want basically a conversation chat bot.
Right.
So we want this functionality where we are giving a request and a chat bot is giving the response.
That is the functionality we want.
We don't want this independent functionality.
This functionality can be combined in the response part.
Right.
So in short, when we probably create a generative pre-training model, what we are specifically doing
over here is that we are able to get this sub task kind of task over here, right?
All this language translation, text summarization and all.
Now we need to convert this task in the form of request and response, right?
So that is the reason why three tasks are basically required.
Right?
So now what we do is that we go to the next step.
That is next stage supervised fine tuning.
Now what exactly this supervised fine tuning is with respect to safety.
It is also called as safety.
Now in safety what happens is that in one side a human being will be there.
Okay.
let's say this is a human being over here.
Let's say I am sitting over here, right?
And I'll talk about a very important role nowadays, which is very much famous, which is called as
prompt engineering role.
Okay.
We'll also talk about that.
Okay.
So here on the left hand side, one human will be there on the right hand side, another human will
be there.
And this human will be acting like a chat bot agent.
Okay.
So what happens is that whenever this human being sends a request, let's say it asks a question like,
hello, how are you?
So the another human will say that.
Yeah, I'm very good.
I'm fine.
Okay.
Something some response will be there.
Then again, this human will send another request.
Then again, the another human will probably send the response.
So like this it will be having the request and response continuously.
So these are some real conversations.
And these all conversations will be getting captured.
So guys now this kind of real conversation will be converted into an sfti training data corpus.
Right now this SFD training data corpus will basically be in the form of request and response.
Request will be your input and response will be your output.
Okay.
So like this lot of request lot of different different responses.
It can be that for a similar kind of request they can be multiple responses also.
So they will try to first of all create this kind of training data corpus.
And it will just not be 1 or 2 records but millions of records, right?
Millions of records.
Now once this is basically getting created you can see request is conversation history and response
is the best ideal response itself.
Right.
So this this format of training data corpus will basically get created.
And then it will then be sent to the base GPT model right for the training purpose.
Now once it is sent to the base GPT model for the training purpose, then Then over here, the optimizers.
According to the research paper that is basically been used, uh, the optimizer that is used over here
is called as stochastic gradient descent.
Right.
And from this you are basically getting an SftP chatbot model.
What is SFD?
SFD I have already written over here.
It is a supervised fine tuning model with respect to ChatGPT.
Right.
So SFD ChatGPT model I will be able to get after I probably do the stage two or supervised fine tuning
now.
Still, there are a lot of problems with this.
Obviously it will be able to give you the answers right.
But this model, this SFD ChatGPT model will be able to give you the output based on the data it is
basically trained with.
If I probably ask some other questions to this particular ChatGPT model that may not be there in this
particular training data.
Then it will start giving you some awkward answers which you may have not been seen also.
Right?
So this ChatGPT will start behaving in a way you'll also not know what exactly it is trying to say.
And this all problems was faced also by the researchers when they were actually creating this.
And that is the reason they came up with the stage three, which is called as reinforcement learning
through Human Feedback.
Now, because of this step, the model that was now created was called as ChatGPT model.
And right now, whatever things you're using with respect to ChatGPT 3.5 or 4 is basically using this
reinforcement learning through this human feedback.
Now let's understand what exactly is happening over here.
And here.
I'm also going to give you a lot of amazing examples to just make you understand, because the most
complex thing is not this.
See, data creation is always a task that we probably do as a data scientist.
Not this is also not a very, uh, very difficult step because there are transformers.
We are using the same architecture.
We are just taking huge amount of data and we are training with it.
Right.
The most important step is this.
Because of this, the accuracy of the ChatGPT has been increased in a tremendous way.
Right now, what exactly is this reinforcement learning through human feedback.
So over here, when we have this safety trained model, let's say a human agent gives a request, then
Sfti ChatGPT will give some kind of response.
Now similarly for this kind of request for this kind of request, we may have multiple response also.
Right.
We may have different different response.
Now this is based on this particular response.
Over here you can see these are all my alternative response.
Now when I say through human feedback where this new human has been put up over here in this right.
So once we probably get multiple different responses, now this human agent, what it will do is that
it will try to rank all this response saying that which is the most suitable response, right?
Which is the most suitable response, or which is the best response based on that ranking will get assigned.
So here you can see that response B is the best.
Then response A is the best.
Then response D is the best, then response C is the best.
We are ranking all the responses now based on this response is ranking.
What we do is that we create a reward model.
Basically, the researcher created a reward model wherein based on every response, they probably provide
a score right for every response they provide.
Provide a score.
And this score will be based on probability.
So it becomes a binary classification if the probability is high, right?
If the probability is very, very high, probability ranges between 0 to 1, the probability is high.
That basically means that particular response is a very good response.
The probability is low.
That basically means the the that for that particular response the score is less.
Right now this reward model.
Now obviously if I'm explaining you like this, it is very difficult to just to understand.
So let me give you an example over here.
Let's say that there is a chef.
Okay.
Now this chef knows how to cook any kind of food okay.
Any kind of food.
Now suddenly in a restaurant there is a request saying that from a customer, hey, I want to have a
very good non vegetarian food, okay?
It can be chicken right.
Something I'm just giving some things right.
Like chicken.
Right.
And I want to have it and dinner right now if probably the chef gets this kind of request, chef will
not initially know.
Like, what food should I create?
It depends.
Right?
If the chef is from India or he's from some other foreign countries, they'll try to create that kind
of things.
That is actually likable by the chef.
Right.
So what chef will do now?
It will first of all ask from many, many people like what kind of food would you like.
So this is the initial response that is taken.
So these are all my response that is basically coming up.
Right.
So these are all my response.
You will be able to see that I'm picking it up right.
So chef initially will put up all the responses.
Right.
And it will also ask that okay fine.
Then what will happen based on this response.
It will try to rank it right.
How many people have told that similar kind of responses.
Right.
If many people have told, okay, I like this specific food, obviously it can try to rank it, right.
This can be greater than this.
This can be greater than this, this can be greater than this, this can be less than this.
Now once we try to rank this specific response now, chef, what it will do is that it will try to create
a reward model.
And this reward model will be very, very simple.
It will be a binary classification.
Over here cross entropy is also used okay.
Cross entropy is used now based on this.
What happens is that whenever the chef gives any response right, it should be able to consider that
whether we should go ahead with this particular output or not, or whether should I cook this particular
food or not like that.
Right.
So this is what is the reward model that is basically getting created.
And this is based on the feedbacks.
The feedback is probably coming from the human beings over here.
Right?
So I hope you got this specific idea about it.
Now once the reward model is basically created, then reinforcement is basically applied by a technique
which is called as proximal policy optimization.
Now based on this proximal policy optimization, all the things that is basically happening is that
reward model, first of all, updates the reward based on the response that is probably coming from
the ChatGPT model, right?
And then it will also make sure that it will update the specific rewards by using this proximal policy
optimization technique.
Again, in this technique, uh, if I, uh, probably I'll not explain explain about proximal policy
optimization right now, but I'll make a dedicated video about this.
But again, understand that this is a reinforcement technique wherein we will be able to improve the
ChatGPT response, and based on that, whatever human feedback or response is coming up, it will try
to make sure that it will try to increase the reward if the response is properly correct.
Right.
So that is the thing that is basically happening over here.
And finally we basically get a ChatGPT model and this reward updation and and the policy model that
is probably getting updated using the proximal policy optimization will happen continuously as the conversation
is basically happening.
Right.
And this is how the entire process of reinforcement learning happens through human feedback.
See guys I know from this, whatever things you know, you are very much familiar with stage one and
stage two.
You may also know that how you can probably create this particular data set, right?
That can be a manual approach, but yes, you can definitely do it right.
Only the thing, the most complex thing that we usually happens is over here.
But understand if you are able to understand things right, writing a code for this is also very, very
easy.
Probably you can do it, but it will not just be possible by any companies to do this because they definitely
require a huge amount of data set, right.