Hello guys.
So we are going to continue the discussion with respect to natural language processing.
In our previous video we have already seen what is n grams.
Now what we are going to do is that we are going to see one more efficient way of converting a words
into vectors, and we specifically say it as tf IDF.
Now what exactly tf IDF is?
It is nothing but term frequency and inverse document frequency.
So we will try to understand how, with the help of tf IDF, we will be converting all these particular
sentences into vectors.
Okay, and I've taken the same example what we specifically did with bag of words like good boy.
And this is after lowering all the, uh, cases or character cases along with that, after performing
or after removing the stopwords.
So I have sentence one as good boy, sentence two as good girl, sentence three as boy girl good.
Okay.
And this is the same thing like, uh, from the same materials, if you will be able to see I've done
the same thing.
I've taken the same thing over here.
Okay.
Now there are two components in TF-IDF.
One is term frequency and the other one is something called as inverse document frequency.
So whenever I talk about term frequency, term frequency definition or the formula how we calculate
it, it is given by number of repetition of words in sentence divided by number of words in sentence.
Okay, I'll try to show you completely.
Taking this as an example, how we can calculate term frequency.
And over here inverse document frequency formula is very simple.
We basically calculate it by inverse document frequency is nothing but log to the base e number of sentences
divided by number of sentences containing the word.
Now this is super important and probably don't get confused with the formula right now.
I will try to explain you each and every thing.
Okay, so let's go step by step and let's see how we can calculate the term frequency.
Now first thing that I am going to make sure that what I calculate is nothing.
But I will be using this term frequency.
Now with respect to this term frequency, you know that how many vocabulary of words are there?
Okay.
So first of all, I will just try to show you in a different way by creating a table.
So I have s one sentence, I have s two sentence and I have s three sentence.
Okay.
And then with respect to my vocabulary of words I have something like good okay.
Then I have boy and then I have girl.
And already we know that only three words are basically present in the vocabulary I'm trying to show
you, show you with a simple example so that you will be able to understand how TF-IDF will work.
Okay.
Now the first thing let's go back to the definition.
Term frequency is nothing but number of repetition of words in a sentence divided by number of words
in a sentence.
So suppose if I take s one and with respect to s one.
If I really want to find out the term frequency of this particular word, that is good.
How do I calculate?
I just need to see how many number of times this particular word is repeated in the sentence.
So here you know that it is repeated just one time, and then I will be dividing by number of words
in that specific sentence.
So it will become one by two, because I have two words.
Now let's go to the next word that is boy.
So boy is also present.
How many number of times one.
And this will be divided by two.
Okay, I'll tell you why we are doing this.
Because when we understand the advantages and disadvantages, you will get a clear idea about why TF-IDF
will better play, perform, or will perform better than compared to bag of words.
Okay then I have the word girl.
So girl, I know that over here it is not present right in the sentence one.
So it will be zero by zero by two.
So which is nothing but zero okay.
Similarly with respect to s two here you will be able to see how many time good is present only one
time and the total number of words is two.
So one by two boy is present zero time.
So it will be zero by two, which will be nothing but zero.
And girl is basically present over here one time.
So again I'm going to write one by two.
Now let's go with respect to the sentence three.
So sentence three how many times good is present one time and the total number of words now is three
right?
And then if I go next, uh, to see the boy.
Word.
How many times boy is present?
Only one time.
So this will also be one by three, and girl will also be present one by three, because the total number
of words are three.
Okay, so this is how simple we are able to calculate the term frequency okay.
Now let's go ahead and let's try to find out the inverse document frequency.
So I'm just going to write over here as IDF.
Now with respect to IDF also we will be creating two fields, very simple fields.
And remember uh, this will basically be my IDF.
And this is with respect to my words over here.
So my words are nothing but good boy or let me write it down in a better way so that in the it should
look in the same order.
So here will be my good.
Here will be my boy.
And here the next word is something like girl.
Now, in order to calculate the inverse document frequency, it is very much simple.
Now all I have to apply this log base E.
Right.
How many number of sentences are there with respect to good right.
With respect to good.
Uh, suppose if I really want to calculate the inverse document frequency of good.
Okay.
So here what I'm writing I'll write log base e multiplied by number of sentences.
How many sentences are there.
There are three sentences.
So three divided by number of sentences containing the word.
So good is present in all these three sentences 123.
Right.
So I will basically be writing log to the base three divided by three okay.
And if I basically calculate this, if you try to calculate it it is nothing.
But I will be getting a zero okay.
And you can basically do with the calculator by over here.
Again log to the base e number of sentences are three.
And how many times boy is present.
How many times boy is present in uh how many sentences?
Boy is basically present.
It is present one two right.
Sentence one and sentence three.
So I will be writing log base uh, base e multiplied by three by two.
Similarly, girl will also be present same number of time if you probably see how many sentences girl
is basically present.
Okay, so I have independently calculated term frequency and I have independently calculated inverse
document frequency.
This is perfect right now whenever we say tf IDF.
In short what I'm actually doing I'm multiplying this two okay term frequency and inverse document.
I'm taking the combination of this two.
Now let me go ahead and write it down in a better way.
Still, uh, in the way that we specifically want.
And finally how our vectors will look like.
So this is my vocabulary.
Good boy.
Girl.
And this is the final tip.
Uh tf IDF okay, so final tf IDF based on this calculation.
And this will differ based on dataset to dataset.
Okay, so first of all, with respect to sentence one, whenever I see the combination of TF-IDF with
respect to good, all I have to do is that multiply one by two with this zero okay.
So I'm going to multiply this with this.
So in sentence one.
So see this is the sentence one right.
This is this entire thing is the sentence one.
Right.
So this is my sentence one okay.
So I will be taking this combination and multiply with this.
Right.
So sentence one good one by two multiplied by zero.
It is nothing but zero with respect to boy one by two multiplied by log base e one by two multiplied
by log base E three by two will be the value that I'll be getting in sentence one.
And with respect to girl zero multiplied by this it will be zero.
Now let's go to the sentence two.
In sentence two, I will go ahead and look for this.
Now again, one by two, multiplied by zero.
Again good will be zero.
And this boy is nothing but zero by zero.
So this zero multiplied by zero is nothing but zero.
And here I have one by two multiplied by log base e three by two.
I will tell you the exact thing.
What we really need to know why we are doing this specific thing.
Everything will make sense.
Uh, and it will make sense.
And I'll make you definitely understand all these things.
Okay, so coming to the next one with respect to sentence three, in sentence three, I will do this
multiplication with this.
Right.
So one by three multiplied by zero.
Again it will be zero one by three multiplied by log base e three by two and one by three multiplied
by log base E three by two.
Perfect.
So we have got all this calculation.
And this is how my vectors will look like.
So here you will be seeing that for sentence one.
We converted this entire sentence into vectors which looks like this.
Right.
So this is my sentence one vector.
This is my sentence two vector.
And this is my sentence three vector.
And obviously I'll be having a output with respect to any kind of classification that I want to do.
And then I will train my model by passing my sentence one.
So in short, if you see Good Boy is basically converted into a vector which will look like this.
Okay, this entire sentence one is getting a converted into a vector zero this and this zero.
Right.
So again you can calculate this with the help of calculator.
But this is what is the way that we have actually done.
We have converted all our sentences into vectors.
And in TF-IDF, this is the phenomenon that is used in converting the words into vectors.
Now you may be thinking, Krish, what is so special about this?
We have got some values.
Okay, that is fine.
And that is what I am going to talk about in my next video about the advantages and disadvantages of
TF-IDF.
At the end of the day, the best one is nothing but word two vec.
But I really need to show you this also because majority of the use cases.
Also you can basically solve with this technique that is TF-IDF technique.
Okay.
So yes, uh, this was it uh, for this particular video.
I hope you like it.
Uh, again, you can take different, different sentences and just try to apply this and try to see
whether you are able to find out TF-IDF or not.
So yes, I will see you all in the next video where I'll be discussing about the advantages and disadvantages.
Thank you.