In this video.
In this video we are going to talk about quantization.
Exactly.
Quantization.
We're going to discuss you know, uh, we are going to discuss about full precision half precision.
And this is something related to data types like how the data is stored in the memory.
When I specifically say data in LM models, I will talk about weights and parameters.
Right.
Because at the end of the day, LM is also deep learning neural networks in the form of Transformers
or Bert.
Right.
Then we are going to discuss about what what exactly is calibration?
Uh, this is also called as uh, like calibration in model quantization.
Right.
We are going to also make sure that we are going to see some problems.
Right.
How we can actually do calibration.
Then there is different different modes of quantization.
Right.
Uh, first of all I will explain you the definition.
Then only you will be able to understand in modes of quantization we are going to discuss about two
types.
One is post-training quantization and quantization aware training.
Right.
So these all are very important in terms of fine tuning techniques.
Now let's go ahead and talk about quantization.
And we will try to see the definition right.
Quantization okay.
Now if you want to really understand the meaning of quantization it is better to write a simple definition
for it.
Okay.
So quantization basically means conversion from.
Higher memory format.
To a lower memory format.
Right.
Now, I have written a very generic definition.
What exactly quantisation mean?
It is nothing but conversion from a higher memory format to a lower memory format.
Now, when I say higher memory format, let's let's consider, uh, any data.
Right.
And if I probably consider any neural network okay.
So let's say if I have neural network.
Right.
And when we train this neural network, right, all these neural network are interconnected.
At the end of the day, what are the parameters that is probably involved over here?
It is nothing but weights, right?
We specifically have weights.
Now.
Weights are usually in the form of matrix.
Right.
Let's say that I have a three cross three weight okay.
I'm just taking as an example in one of the layer I have three cross three weights.
And over here every value right is probably stored in the memory in the form of 32 bits, right, 32
bits.
We also say this bits as we also we also denote it as something like something like FP 32.
Now what exactly is FP 32?
So FP 32 basically means I can also consider it as c fp.
Full form is not floating point.
Okay, but we are.
I'm just writing floating point 32 bits right?
When I say FP.
FP basically means it is nothing but full precision.
Right?
Full precision or single precision.
Okay, so this is the definition that is probably given, right.
But in short this is like a floating point number okay.
So over here, let's say my number is somewhere around 7.23.
Now this number is stored based on 32 bits in the memory right.
Now understand when you have a very big neural network or you have LM models, right.
As you see different different animal models.
Right.
Parameters keeps on increasing.
Right.
Some may have 70 billion parameters.
If I probably consider llama two with 70 billion parameters, that basically means it has 70 billion
parameters in terms of weights and bias.
Okay.
Now it is not possible for me to let's say I want to use this particular model, and I want to probably
do some fine tuning with respect to the normal GPU that I have.
Right.
And let's say I have a very limited Ram in my system.
Let's say the Ram I have is somewhere around 32 GB, right?
I cannot directly download this specific model and put it in my memory in my Ram itself, right.
Let's say or load it in my Vram that is available in the GPU because GPU also has some limited Ram,
right.
And it is not possible you cannot directly download it.
Obviously it will require you space the other way is that yes, I can probably take a cloud space somewhere.
Let's say in AWS, I can create my instance.
I can say, hey, give me this, this much Ram, 64 GB Ram, and I probably want this much GPU.
And then I will try to load the model over there that you can do it.
But over there what is basically happening, lot of cost is involved, right?
Based on the resources, the cost is involved.
Right.
right.
So in this scenario and why this 70 billion parameter is happening, why this model has a 70 billion
parameter.
Because every weights or every bias that is available in this, you know, this may be getting stored
in 32 bits.
So what we can specifically do is that we can convert this 32 bits into, as I said over here C conversion
from a higher memory format to a lower memory format.
Let's say I can probably convert this 32 bit into int eight and then download the model, or then use
the model right after doing this, what will happen within my system?
I will be able to influence it, right?
Obviously for fine tuning, if I want to fine tune it, the new data set I will obviously require GPU,
but if I consider with respect to inferencing, it becomes quite easy because now all my values that
are stored in the form of 32 bits, it will be stored in the form of eight bits.
So what we are specifically doing over here, we are converting from a high memory format to a low memory
format.
And this is what is called as quantization quantization.
A very important thing.
Now why quantization is important because you will be able to influence it quickly.
See, inferencing basically means what if I have an LM model?
If I give any input to that, I should be able to get any output right.
I should be able to get response right.
Now, when I give any input, all the calculation with respect to different different weights will happen,
right?
And obviously if I have a bigger GPU, this inferencing will quickly happen, right?
But if I have a GPU with less less cores, let's say, then what will happen?
This calculation will take time, but if I convert my 32 bit to eight bits right, every weights are
basically converted into eight bits.
Now just imagine the calculation.
Will there be a difference?
Yes it will happen little bit much more quicker.
So quantization is very much important for inferencing.
And some of the examples that I can probably talk about is that and obviously you may have heard about
this.
It's not like only an LM model.
We specifically do write in different computer vision models.
In NLP models also very thing that there is a lot of weights that is involved and all these weights,
if I want to quantize it right, I can actually do it right now.
This inferencing, let's say I want to use a specific deep learning model in my mobile phone.
So in my mobile phone, if I want to use it right in in a specific app, then what I will do, I will
try to quantize, right?
Whatever deep learning model I have created from 32 bits to eight bit, and then I will try to deploy
it in my mobile phone or any edge device.
Any edge device, Right.
It is not possible that you can probably deploy this big model over there.
Right.
It is not possible.
So and so many parameters.
So what we do we basically perform quantization.
So I hope you are able to understand over here quantization is nothing.
But we are trying to lower down the memory right with respect to any weights that we have like from
32 to probably 60.
uh, int eight or let's say FP 16.
We can also say FP 16, right.
Let's say if I have an FP 32 bit that is specifically required to store any information in my memory,
I can also convert this into FP 16 bit.
This is also quantization only, right?
Usually all these values are stored in floating point, right.
We specifically say this FP 32 bit.
We say single precision or full precision.
We uh for FP 16 bit.
If I'm trying to convert like this, it is basically called as half precision.
Right.
So you should be able to understand all this technical terms.
Right.
And in short this is are nothing.
But these are floating point numbers right now.
Similarly in TensorFlow also you'll be able to see when we probably work with TensorFlow you'll find
TF 32 bit, right.
The data types the numbers are stored in this particular format.
Right.
And it is important okay.
This is all our terminologies that are super important.
But I hope you got an idea why.
What is the main aim?
What is the main motivation out of quantization is that if I have a bigger model, I should be able
to quantize it and make it as a smaller model so that I can use it for my faster inferencing purpose,
both in mobile phones and the edge devices, let's say in even watches smartwatches.
I want to use it over here.
I can actually do that right now if I talk with respect to LM model, also with the help of quantization.
See, once we compress this particular model right later on, we can also perform fine tuning.
Right.
Fine tuning.
But here there is one disadvantage when we quantize.
Right.
When we perform this quantization, since we are converting from 32 bits to int8, let's say as an example,
there is some loss of information also.
And because of this there will be some loss of accuracy.
Now how to overcome this?
We will talk about it.
There are different different techniques how we can specifically overcome it.
But I hope you got an example.
What exactly is quantization?
What is full precision.
Half precision, half precision?
Example is something like this.
Now let's talk about what exactly is calibration.
Now calibration basically means how we will be able to convert this 32 bit into int eight.
Like what is the formula.
What is the mathematical intuition that is specifically required.
Let's go ahead and discuss that.
So guys now let's go ahead and try to understand how to perform quantization.
And this is super important in terms of mathematical concept that I'm probably going to talk about because
with with the help of TensorFlow, just by writing four lines of code, you know, I will be able to
perform quantization, but it is important.
You should know that how you can actually do it manually.
Whenever I talk in terms of what are the types of quantization that we have.
So we have two different types of quantization.
One is symmetric quantization and one is called as asymmetric quantization.
Now just by showing you an example, you will be able to understand what is the exact difference between
them okay.
Let's say I have a task, and this first task that I am probably going to talk is with respect to symmetric
and understand.
I hope in deep learning you have heard of something called as batch normalization.
So if you have heard about this batch normalization.
So batch normalization is a technique of symmetric quantization.
Right.
So every time you'll be able to see that whenever we do forward propagation backward propagation between
all the layers, we apply batch normalization so that all our all our weights are zero centered, that
is near the zero.
And the entire distribution of the weights will be centered near zero.
Okay.
So batch normalization is one technique uh of symmetric quantization.
So let's go ahead and see one example.
So this will be my first example over here I will go ahead and write it down.
And now you'll be able to understand it how symmetric quantization is basically performed.
Now what is symmetric quantization you have understood from higher memory format to lower memory format
will try to convert.
Okay.
So here we are trying to understand the mathematical intuition.
So let's go ahead and talk about one technique which is called as symmetric unsigned unsigned int unsigned
int eight Okay quantization.
So we will see this technique first.
Now here what is our main aim.
Let's say I have a floating point number okay.
So let's go ahead and write it down.
Let's say I have a floating point number between 0 to 1000.
Now just imagine that these are my weights right.
Whatever matrix mates I have my values ranges between 0 to 1000.
Let's consider in this particular way.
Right.
And this is let's say these are the weights for my larger model okay.
Larger model okay.
Now one very important thing that you really need to understand, right.
When I talk about any larger model let's consider any LM model okay.
So in a linear model you have lot of parameters.
Let's consider this is one kind of LM model.
Now when I say all this weights are there this may be getting stored in 32 bits okay.
Usually what will happen guys the weights will not be in this range also okay.
It will be in the minimalistic range.
So I will just consider this as some numbers okay.
So that you don't get confused with respect to these are some numbers.
I will also not consider LM model over here.
And let's say this this this numbers are stored in the form of 32 bits okay.
Now my main aim is to convert this into unsigned int eight.
That basically means eight bit right.
So eight bit basically means what two raise to eight is how much two raise to eight.
But when we say you unsigned that basically means my value will be ranging between 0 to 255.
So I want to quantize from this values to this value.
Okay.
What is my aim?
I want to quantize my range of values between 0 to 1000 to 0 to 255.
Okay, this is what is my aim with respect to this okay.
So this is what is my target now.
Let's see okay.
So if I probably draw just a real points and the same thing we will do with the weights over there,
whatever quantization process we are specifically doing, let's say I have values between zero and this
is basically 1000.
Okay.
Then what I will do over here I want to convert this into again 0 to 255 okay.
now let me talk about one very important thing.
Guys.
Whenever we have any let's let's consider this one.
If we have this single precision, single precision floating point 32.
Right.
If we have this number, how this number is stored, do you know that the one bit will specifically
be used for sign or unsigned values?
Right.
Let's say positive or negative.
If positive is there, then this will be plus one.
If negative is there, it will be minus one.
So all the values are basically saved between 0 to 1 right.
Uh it can be 0 or 1 okay.
Then the next eight bits are stored for exponent okay 12345678.
Okay.
So this is for sine this all numbers that you see it is basically stored for exponent.
This is how it is stored inside the memory and remaining 23 numbers.
All this 23 bits will basically be saved for mantissa.
This is specifically for the fraction.
So if I have a number which looks like 7.32 now 7.32 is the number, it is a positive number.
So for my sign bit there will be a positive value let's say one over here.
Then the seven will be probably put up in this eight bit.
And remaining 0.32 will be put up in this mantissa.
Right.
So this is how the numbers are basically stored in the memory, right.
If I consider an example with respect to Fp16 right half precision floating point 16 bit, then you'll
be able to see there will be one bit for the sign number.
There will be five bits for the exponent.
We basically say exponent 12345.
Let's consider this five.
Let's let's draw till here okay.
So this let's let's let's say that this is 512345 okay.
So this will be five.
And remaining remaining ten bits will be saved with respect to mantissa.
Mantissa.
So we basically say this as a fraction right.
Anything that comes after the decimal.
Right.
And this is how you will be able to see it will take this will take less memory.
This will take high memory.
Right now what is our main aim over here?
I already have a 32 bit number.
I need to probably convert this into a range of unsigned int8.
Unsigned int end basically means I will not take any negative numbers.
It will be between 0 to 255.
Right?
This is what I really want to do okay.
Okay.
Now for this.
What will be the equation?
And I hope you have heard about something called as min max scaler.
I've repeated so many times this in my machine learning sections also.
So any number that is over here, how will I be able to convert from this floating point to this unit
eight unit Int8 right.
Unsigned indent right?
How I will be able to do it.
Now for this, let's go ahead and calculate it.
And what equation is specifically required.
Okay.
So that basically means over here what we are going to do 0.0 will be converted into a quantized value
of zero okay 0.0 it will be zero.
And similarly thousand should be converted to a quantized value of 255.
At the end of the day the bits are decreasing.
So quantization is basically happening.
But we have to probably come up with a scale factor.
Now what exactly is a scale factor.
So let me define a scale over here.
So here the scale formula will be x max divided by x mean.
And then they will be divided by q max minus q mean.
This q is nothing but quantization.
Now what is the x max over here 1000 right.
So this I will consider as my x.
This I will consider as my Q right.
I'm showing you how quantization happens in a symmetric distribution.
Symmetric basically means all the data is evenly distributed okay.
And we really need to convert this based on this itself.
These are evenly distributed.
Now what is x max x max is nothing but 1000 -0.
Then q max q max basically means 255 -0.
Right.
So if I probably go ahead with this specific division right.
Then what will be the value that I will be having.
Right.
1000 divided by 255 it is nothing but 3.92.
So this is nothing.
But this is called as a scale factor.
Right.
Scale factor.
So any number that I have over here, if I want to convert from this FP 32 bit to you into it, I just
need to use this scale along with one formula which is called as round.
Okay.
So I will apply round with respect to any number.
Let's say if I consider 500 divided by 3.92 or let's let's just consider 250 divided by 3.92.
So if I want to see what will happen to 250, right.
What will be the number in this you int eight.
So I can probably go ahead and divide it.
It will be nothing but 250 divided by 3.92.
So if I go and calculate it, it is nothing but 63.77.
So if I do the rounding that basically means this will be 64.
So in short, any number that is over here, let's say it is 250 over here this will get converted to
a quantized value to 64.
Right.
So the same thing the code will also be doing.
And this is for symmetric unsigned int8.
Okay.
Quantization if I want a quantization some for some other factor, let's let's talk about this okay.
So let's say uh I have another kind of distribution.
And this is time that it is asymmetric.
And I want this as you want it.
So if I want to perform this quantization so what will happen now in this particular case let's say
if I have a values between -20.0 to 1000.
Okay.
So these are my floating point.
Now I want to perform quantization and convert this into 0 to 255.
Right now in case of asymmetric what will happen is that in my real number section right.
This numbers are not symmetrically distributed.
It may be right skewed.
It may be left skewed.
Okay.
So in this scenario you will be able to see that my values are ranging between -20 to 1000.
I want to convert this into this.
Now in this scenario if I apply the same formula x max minus x mean.
So how it will be 1000 -20.
So minus of -20 is nothing plus plus 20 divided by 255.
So if I probably do the calculation then you will be able to see that I will be getting somewhere around
4.0.
Okay.
Now very important thing that basically means this 20.0.
If I quantize it right.
If I quantize it, it will get converted to something like 4.00.
Sorry.
This this is the my scale factor okay.
Scale factor.
Now if I take any number and try to convert it let's consider -20.
If I try to convert it by dividing by 4.0.
Right.
And if I do the round.
So what it will become.
Minus five.
Right.
Minus five of round.
This much I will probably get minus five.
Right.
Now you can understand that this -20.0 is getting converted to -5.0.
But you can see over here my distribution starts from 0 to 255.
So how can I forcefully make this -20.0 to 0?
All you have to do is that go ahead and add the same number in a positive way.
So in this case the number that you see this five right.
This is basically called as zero point right.
So there are two important parameters that we specifically talk with respect to quantization.
One is 0.4 for the above one.
Since we have a symmetrical distribution here, the zero point was zero only and the scale was 3.92.
In this particular case, since it is a symmetrical distribution, here we have a zero point as nothing
but five.
But scale is 4.0.
So this two parameters we usually require to perform quantization.
Okay.
And these are some of the examples that I have shown you.
To just give you an idea like how quantization basically happens.
And super important in terms of understanding the simple equations.
You'll be able to understand how things are basically working right at the end of the day.
Understand quantization is a simple process of converting that high, uh, full single precision of
full precision floating point 32 bits into small bits.
You know, it can be, uh, unsigned integer eight.
It can be signed integer eight.
If we say signed integer eight, then what will happen?
It is that it will be ranging between -128 to 127.
And based on that, you can specifically apply the formula.
Right now let's go ahead.
See we had already discussed about this two topics.
One is this.
And second one we wanted to discuss about calibration.
Now this squeezing that you could see right from here to here to here, here to here we are squeezing
it right.
This squeezing process is basically called as calibration.
Whatever process we are basically applying in this quantization process, it is nothing.
But it is called as calibration because we are squeezing those values from a higher format to a lower
format.
Okay.
So that is nothing but calibration.
So we have completed both this thing.
Okay.
Now let's see what are the different modes of quantization.
One is called as post-training quantization and quantization aware training.
Right.
I will talk about this.
Why it is super important both this technique okay.
So you will get an idea about it over here.
So first of all we will go ahead and say post training quantization.
So what exactly is post-training quantization.
Here we already have a pre-trained model.
So we already have a pre-trained model.
Now if I want to use this pre-trained model obviously the weights are very high.
Here we apply.
Calibration.
Right.
When I say calibration that basically means squeezing the value from high format to a lower format,
right?
And then after performing this particular calibration, we take what kind of data we take that weights
data, whatever weights data is basically there in this particular pre-trained model.
And then we convert this into a quantized model.
Okay.
So once we apply this process then only we will be able to get the quantized model.
And then we can use this entire model for any use cases okay.
For any use cases right.
This is a simple mechanism with respect to post-training quantization.
See understand Post-training basically means I already have a pre-trained model where my weights are
fixed.
I don't need to change those weights.
I will just take or download those weights.
I will take this weights data, apply the calibration and then convert this into a quantized model.
Right.
The second technique that we have written over here is quantization aware training okay.
Quantization aware training.
So let's talk about this.
Quantization aware training.
This is also called as cued.
So this is also called as cue 80.
Okay.
So we can write it as Q 80.
Okay.
Quantization aware technique.
This is basically called as p t q p t q okay.
So over here what is the exact difference.
Will try to see between this two okay.
Now in quantization aware training what happens.
See over here.
What is the problem if I probably perform calibration and if I create a quantized model there is a loss
of data.
And because of this what will happen is that the accuracy will also decrease okay for any use cases.
But in the case of quantization aware training okay, you will be able to see that we will be taking
our trained model.
Whatever trained model is there.
Trained model is there.
Okay.
Let me just go ahead and write it down.
Train model is there.
Then we perform quantization.
Again quantization is what same the calibration process will apply over there.
We will probably do all these things okay.
And then once we do this, the next step is that we will go ahead and perform fine tuning.
See, we know that.
We know that with the help of PDQ, you'll be seeing over here on the top some loss of data and accuracy
is there.
But here, with respect to fine tuning, we will take new training data.
New training data.
Now, once we specifically take new training data, we will be fine tuning this model.
And then we create a quantized model.
Quantized model.
So with respect to any fine tuning technique that you will be seeing we don't use post-training quantization.
We specifically use quantization aware training technique.
So that basically means we are just not losing accuracy or data over here because we are in turn adding
more data for the training purpose.
And through this we will be fine tuning our data and then we create our quantized models.
Right.
So this is the basic difference.
So all the fine tuning technique that I will probably show you in the future will be of this type that
is quantization aware training so that we do not lose much data or accuracy.
So I hope you got an idea with respect to all these three techniques guys going ahead, there are two
important techniques that we really need to understand.
One is chlora and one is Laura.
So this techniques specifically will be also understanding with respect to fine tuning.
So I hope you got an idea.
Just get to know about all these things.
Guys.
This is important because someone if someone asks in the interview what exactly it is, then you'll
be able to understand it very much easily.
And again, explaining all these things will be important if you are really interested, because in
January we have what I feel is the most important thing is with respect to the fine tuning things.
So I hope you like this particular video.
Uh, this was it for my side.
I'll see you on the next video.
Have a great day.
Thank you all.
Take care.
Bye bye.