Hello guys.
We are going to continue our discussion with respect to natural language processing.
In our previous video, we have already seen Ngrams intuition.
Now we will try to implement it with the help of NLTK.
We are going to take the same problem statement what we have taken for the spam classification.
So this is how we had actually created a bag of words, right where I took maximum features as 100.
Like most occurring, frequent occurring words, uh, I have taken that as top 100 words.
Now, if you really want to find out what are the top 100 words.
So you basically need to after you do the fit transform, right?
You just need to take the CV and you can just write dot vocabulary.
And this part I had actually missed while explaining bag of words.
So these are my top 100 words.
Go great.
Go out what okay free win text txt say already I think so here you can see these all are indexes of
the columns right.
So if I probably say which is the first first column that you'll be able to find, it is nothing but
ask, right.
Which is the second column.
It is something like babe, okay.
Uh, third column like that.
So total you will be able to see it will be having the indexes from, uh, 1 to 100.
Right.
Something like that.
So these are my top 100 words okay.
So this is with respect to CV dot vocabulary.
If you are able to find it out you'll be able to if you're just executing this you'll be able to do
it.
Now let's go back to this sklearn feature extraction.
Now this is super important.
Now I'm talking about this n gram underscore range okay.
Now see this.
What will happen once I just copy and paste the same thing.
So let's say I'm going to copy this and I'm going to paste it over here okay.
I'm just going to paste it over here.
Now this is super important okay.
So I'm just going to execute this and I'm going to paste it over here.
Here I'm just going to say n gram okay.
Now you know that there is a parameter which is called as n gram range.
Okay.
Now once I do one comma one nothing is going to change.
Now see this okay.
Nothing is going to change.
And I'm just going to do the same fit transform on the corpus okay.
And I'm going to save it in a x okay.
We need to do fit underscore transform.
So once I execute this here if I go and write CV dot vocabulary, and now if I show you, it will be
the same word.
See.
So these are the hundred most frequently frequent occurring words.
And here I have given n ng gram underscore range as one comma one.
That basically means all my features will be one.
There will not be any combination of two words like bi gram or tri gram.
Now see this?
Okay, what I will do is that I will change this to one comma two.
Now once I write one comma two, now it is going to be the combination of unigram and bigram.
Okay.
Unigram and bigram.
Now see this okay.
And I'm also going to increase the number of max features.
Let's say I'm going to make it as 200 okay.
Now let me just go ahead and execute.
And now let me see the vocabulary.
Now vocabulary over here you can see uh all these words are there single single words are there.
Now as you go ahead, right there will be two words.
See?
Over here.
Please call.
Okay.
So here you'll be able to see.
Please call and you won't be able to see the other words because that is not the maximum occurring words.
Let's say I want to increase this number.
Let's say I'm going to basically write it as uh 500.
Now the number of words will keep on increasing.
So yeah here you can see that let know is uh in the is in the 2228 index of the column.
Right.
Still you have how many please call is also there, which is present in the 325 index.
And that index number is basically given based on the maximum number of occurring frequency.
Right.
So uh, more words.
Are there something called as customer service.
Right.
And if I probably go and explore more price guarantee guarantee call try contact.
See all these words are definitely there.
There is obviously some spelling mistakes but it's fine.
So over here empty decimal decimal g g t something something is there now you see.
Good morn.
Right.
So morning.
Something like that.
Good night.
Right.
So this is the basically the combination of this.
Now what I can also do is that if I really want to see the maximum occurring, uh, just a bigram kind
of words, I can just write two comma two.
That basically means I'm ignoring I mean, ignoring unigram.
So now here you will be seeing that free entry claim call call claw claim free call chance when every
combination of the possible words are basically there along with the column index.
Now, I hope you are able to get an idea what is the basic difference between unigram bigram trigram.
Right?
So suppose if you also want trigram with respect to this one I can basically write one comma three.
Now let me do one thing.
I'll also take a combination of first three comma three.
Let's see Uh, and let's execute this.
Now here you can see call claim code.
So this is the sixth index.
That basically means this kind of message are occurring more in the and this is basically in the column
sixth index.
Right.
Like l t g t sorry.
Call letter please call custom call custom service.
Right.
All these things are there.
Now you can see this is the combination of three words.
Right now if I want to use the combination of two and three I can basically use two and three bigram
and trigram.
So here you can see call customer service.
Then you can see let know.
Then you can basically see game call.
So you can basically make the combination of various unigram bigram and trigram.
Usually if you are not getting a better accuracy first of all start with one comma one.
Okay, I'll tell you the simple phenomena.
Just try with one comma one with respect to one comma one.
You will be getting a vocabulary word which is the most frequent words, right?
Let's say with respect to this you are not getting a good accuracy.
Then what you can do, you can increase one comma two and go ahead and execute it.
Right.
Suppose let's say it is not increasing.
Then what you can do try one comma three.
Now if even one comma three is not working you can play with this max feature also.
And these are called a hyper parameter.
You can play with different different max features if that combination is also not working.
Use the combination of tri, bigram and trigram and this will definitely give you a better answer,
right?
So definitely my suggestion would be that try to check each and every parameters of this and how we
can basically use it.
N-gram underscore range plays a very important role.
And this was with respect to the practical.
Have already shown you that how you can play with this and how you can actually use it.
Right.
So, uh, at the end of the day, after you do this now, you will be able to see that once I make this
combination.
Right.
And this will basically be my vector of x.
So here, if I probably go ahead and execute.
So this is how my vector of x is there.
You will also be having ones and zeros since binary in true and all the remaining things are actually
there.
So I hope you got an idea about how to implement n gram.
Uh, yes, this was it.
Uh, in the next video we'll try to see with respect to TF-IDF intuition and uh, we'll also try to
solve it with the help of NLTK.
So yes, this was it.
I will see you all in the next video.
Thank you.