Hello guys.
So we are going to continue the discussion with respect to tf IDF.
And already I've shown you how we can what is the formula of tf IDF?
That is term frequency and IDF that is inverse document frequency.
And I've also shown you an example over here.
Right.
So till here that is everything is fine.
Now let's talk about the most important thing about advantages and disadvantages and why this is probably
better than bag of words okay.
So first of all, uh, the basic advantage that we have again, this is quite intuitive.
Uh, the implementation is also quite intuitive.
Uh, coming to the second advantage.
Okay.
Like bag of words.
Uh, here also our inputs are basically fixed size.
And this is based on the vocab size.
Right.
And this advantage is also present with respect to bag of words that is also there.
But the third advantage that I'm actually going to talk about see in bag of words.
Also we had fixed size right.
But this third advantage is a major advantage.
Now let's talk about the third advantage okay.
So the third advantage is that the word importance is getting captured.
I'll explain you what exactly this is.
Word importance is getting captured.
Super important point.
And probably they may also ask you this specific thing in interviews.
Now if I probably go and see my entire paragraph, let's say this is my paragraph.
Good boy, good girl, boy girl.
Good, right?
I'm getting a TF-IDF of this number.
Right?
And over here I.
I have also written this with the help of bag of words and bag of words.
I used to get either 1 or 0.
Wherever that word is present, that is coming as one, otherwise it is zero if it is not present in
the sentence.
But now the word importance is getting captured over here, equal importance is given to both the word
like good and boy, right, because it is present in the sentences.
But here it does not work like that.
Here, considering the entire paragraph, what it is happening is that we are focusing on two things
term frequency.
Inverse document frequency.
If if a word is present in all the sentences, it should be given less importance.
Understand this okay.
If a word is present in all the sentences in that paragraph, it should be given less importance.
Why?
Because all the all the sentences having that specific word.
So it is not playing that amazing or important role.
Word importance needs to be captured from every sentence.
That is what we specifically want.
Now over here you can see that boy is there right over here.
Girl is there.
Now boy and girl are getting repeated in 1 or 2 sentences.
not in every sentences.
So if it is not repeated in every sentences, we need to value this particular word in every sentences
as such.
So if I probably take an example of good, good is present in all these three sentences.
So we calculate TF-IDF.
Here you will be seeing that all zeros we are getting over here.
Major major issue right.
So not a issue but it is a good thing we are ignoring the good word because it is present in all the
sentence.
Now if I consider with respect to boy so sentence one boy will play a very important role now, right?
So with respect to boy here, you will be seeing that I am getting some values right.
I'm getting some values.
Now in the second sentence, obviously boy was not there, so it became zero.
But if I consider girl in the second sentence.
So here you will be seeing that I'm getting some value.
That basically means in this particular sentence, the girl word is super important and the context
is based on that specific word that we are having a value of tf IDF.
Right?
So in short, what is happening is that word importance is getting captured.
And in the third sentence it is talking about both boy and girl.
So you'll be seeing that both this boy and girl has some values.
So in short, we are capturing some word importance over here based on the context.
Right.
Super important point.
And by this our machine learning model will be able to understand that, okay, something specific we
are basically talking about.
And that way the mathematical models will be able to find out what kind of predictions it actually wants.
And through this the accuracy increases.
Now let's talk about the disadvantages.
Obviously in this particular case also you have lot number of zeros.
So sparsity still exists okay.
Sparsity still exists over here.
Uh and again we will try to see how we can solve sparsity using word two vec.
The second thing is that what we specifically discuss about is something called as oov.
Out of vocabulary.
Now here also if I probably add any more words over here with respect to the test data, that is going
to get ignored because uh, over here also all my features is basically made based on our training vocabulary
size.
Right.
So this is basically the advantages and disadvantages with respect to uh, TF-IDF, but definitely just
by seeing the advantages and disadvantages, we can definitely know that tf IDF performs better than
bag of words right now.
Uh, in the next video, we'll try to see some practical, uh, implementation with the help of NLTK
and Python.
And again, guys, you really need to practice this considering different, different data sets.
We will try to provide you more assignments as possible so that you can practice these things also.
So yes, this was it from my side.
I will see you all in the next video.
Thank you.