Hello guys.
So we are going to continue the discussion with respect to TF-IDF.
And now we are going to basically implement it with the help of Python.
And we'll try to see that how easily we can implement TF-IDF with respect to all the concepts that we
have learned in our previous video.
I'm going to take the same example with respect to um, PD dot read underscore CSV with respect to the
data set that we specifically have over here.
Spam collection.
Right.
So this is the data set that we specifically have.
So I am just going to rename this over here just a second.
Uh, spam classifier.
Yeah.
So this is my data set.
Uh, like how we did it for the bag of words.
Uh.
And all the things.
And again, this is the normal cleaning process that we specifically did.
I think, uh, all these things are repeated, right?
Porter Stemmer and all and all.
And here also, you'll be able to see that, uh, we are basically cleaning the data, lowering it,
splitting it, and applying stopwords and then stemming it.
Okay.
Uh, so just to make some little bit changes, what I'll do instead of doing stemming, right, I'll
do Lemmatization.
So already I've shown you how to perform Lemmatization.
And for that we import word let Lemmatizer.
Right.
So I'm just going to go over here, copy and paste it over here.
And instead of using Porter Stemmer I'm just going to use word net Lemmatizer just to give you an added
flavor.
So I will write word lemmatize and I'll execute this okay.
And then instead of writing fw dot stem I will write word lemmatize dot lemmatize.
Okay, so I think this is almost repeated.
I've done it again and again.
Now I think you'll be able to understand.
Now this will probably take some time based on the size of the data set we have.
Um, and again, yeah, obviously it is going to take time.
So we'll just wait.
Uh, but uh, let me repeat what I did instead of doing stemming.
I'm actually doing word net lemmatizer.
Right.
And this is also one of the things that you should always try.
Different, different variety of things.
Right?
So finally it is done.
And this is my corpus.
Now with respect to the corpus you will already be able to get correct words over here.
Right.
And these are all my sentences input sentences and which I will probably be using to further create
TF-IDF.
Now to create TF-IDF.
I have already opened this tf IDF Vectorizer.
It is again present in sklearn.
Uh, it is again a technique.
Whatever things we explained you, I explained you with respect to tf IDF similar all the things it
will be done, but it is almost like a bag of words.
All the features are like bag of words.
Here.
Also you have lowercase, you have tokenizer, you have uh, analyzer and all these things.
And you also have n gram.
Right.
So you can also perform n grams over here.
Now let me go ahead and let me try to implement and show you how you can implement Tfidfvectorizer.
So I will say from sklearn dot so feature extraction or dot text import or tf IDF Vectorizer okay.
So I'm going to basically use this and let me make my mic a little bit straight.
Yeah.
So over here, uh, this is done.
Now all I have to do is that initialize Tfidfvectorizer provide, uh, after initializing it.
Right.
Like how we did it for bag of words in the bag of words.
If you see in our previous statement we imported this and then we initialize with max features this.
So what I'm going to do I'm just going to initialize this with the same thing okay.
And over here binary will not be required because binary is not at all required because binary.
It will not be working with respect to that.
Like how we used to work in bag of words.
Right?
So that's it.
See?
Same thing.
I've done it over here.
Now what I'll do, I will just say TF-IDF as my variable.
And then let's say I'm going to take the top 100 most occurring words.
Right.
So after this I will write TF-IDF dot fit underscore transform fit underscore transform on my corpus.
Right.
Like how we did it in the bag of words right fit underscore transform on corpus.
And then I'll finally I'll convert it this into an array.
Right.
So two array.
And I will store this in my variable x.
So once I do it now you'll be able to see x over here.
And for that I'm also importing this numpy so that I can see all my values.
Again, not a big issue because just I'm saying I'm trying to display all the arrays uh, clearly over
here.
Right.
So if I probably go and click on X.
So now you will be able to see I'm getting decimal values like 0.43, 4.46, 1.554.
Similarly you'll be able to see these are nothing but word importance importance of the word that is
given in a specific sentence.
So you'll be seeing different different values like 0.3650 .384.
Now here you're not getting ones and zeros.
So definitely some importance to the word is definitely given over here.
Okay.
So this is what simply so easily you can actually do with the help of tf IDF Vectorizer.
Now what I will do along with this, I will try to again go ahead and show you in n in n grams.
Also, you can also create words and let's say here I'm just writing n gram range.
Let's say I want to go ahead and try with two comma, 222 words okay.
And once I execute it this is entirely executed.
But before that let me show you the vocabulary.
Okay so sorry tf IDF.
TF IDF dot vocab.
So if I probably go and see the vocab.
So these are all the words that has been selected as the features I gave two comma two.
You can also give one comma two uh and based on that you will be getting the top 100 most frequent occurring
combination of words with respect to two comma two.
Right.
Same thing.
No changes, nothing as such.
So that is why I like this.
Because you don't have to do anything.
You're just trying to understand which is the better one, what advantage it basically has right now.
Finally, if I probably go and see the x value with respect to this here also, you'll be able to see
some of the other updates, which is not probably if you explore it some or the other way, you'll be
able to see it.
See over here 0.688 .726.
So all these combinations are already available 0.588 .56 so this is one sentence vector.
You know, this is the another sentence vector.
And this size will be 100.
Why.
Because your vocab size is basically 100.
You have taken the top 100 occurring words that are there with respect to frequency.
So yes, this was about the tf IDF practicals.
Uh, in the next step, all we have to do is that take these vectors, apply with any machine learning
algorithm, and you will be able to get the output.
But before that you really need to do train, test split and all and all.
Anyhow, I'll be showing you some uh, project implementations also, and we'll also be doing some end
to end projects.
Right.
So yes, this was it.
Uh, I hope you have understood this.
I will see you all in the next video.
Thank you.