Hello guys.
We are going to continue our discussion with respect to natural language processing.
In this video we are going to discuss about a very important topic which is called as N-grams.
Now why N-grams is used and it can also be used with Bag of words and TF-IDF.
But till now we have just discussed about Bag of Words.
Let me take a very, very good example.
Suppose I have two sentences and this particular sentence is something like this.
The food is good, okay.
Food is good and the food is not good.
Okay.
Now here you will be able to see that both these words are completely different.
That is completely exactly opposite of each other, right?
Both the sentences.
This is my sentence one and this is my sentence two.
So I cannot say that this two sentences are same.
Okay.
Now what I will do is that I and obviously over here.
Right.
Let's consider that this are my vocabulary.
The food is good okay.
Food is good.
Or let me just add this vocabulary food is not good.
Okay, these are my vocabulary with respect to feature one.
Feature two, feature three, feature four.
Because these are the possible vocabulary that I have.
I'm not considering the because there can be a stop word.
I can also remove is let's let's remove these.
Okay, so what are the possible vocabulary that is basically present?
I'll basically having food.
I'll be having this not keyword because not keyword will distinguish right whether the sentence is positive
or negative.
So this is super important.
So I may have this all possible options with respect to vocabulary.
Now for the sentence one I may get a a vector of something like this 101 right.
Because not is not present over here.
So obviously it is going to be zero for the sentence.
Two I will be getting something like one, one one right.
Understand that why the n is is not present?
Because we have removed this with the help of stop words Okay.
We have removed this with the help of Stopwords.
Not a problem.
You may be thinking Krish not can also be there in the stopword.
It can be.
But we are not going to add it.
Because this word is basically distinguishing the entire sentence.
Now you need to understand that with respect to this two vectors, you will be seeing that these two
vectors are almost similar.
Only one change is there like zero and one.
In real world scenario, I will say that these two sentences are completely opposite of each other.
But when I'm trying to convert this into vectors, it is just almost similar with only one change,
right?
And this is a bigger issue, right?
Now what we'll do is that we'll try to solve this problem with the help of n grams.
Now see this is super important.
And you try to understand this.
Now you know that these are my possible options.
These are my possible options with respect to words.
Right?
So I have my first word like food not good right.
So this is my initial vocabulary.
Now what I'll do is that I will make a combination of this.
Let's say from the sentence one, I'm going to make a combination of something like this food good.
Right.
So my next word that you will be having is something called as foot.
Good.
And let's say that I am going to just perform by gram not by gram.
Also by gram basically means that I will have the combination of two words, right?
So I'll be having foot.
Good.
Then similarly over here you'll be able to see I will also be having something like this foot not right.
So this is one combination.
One more combination I have over here.
Foot not okay.
And then the next combination will be not good right.
So here I will be having my next word which looks something like this.
Not good right now.
See this.
Now if this becomes a possible vectors then we will be able to clearly distinguish much more better
when compared to this.
And here I have implemented by gram.
By gram basically means combination of two words.
I can also take bigram and trigram.
I will take the combination of three words like food, not good right?
So that would be one combination right.
Something like that.
So let's understand with respect to bigram.
Now how will the vectors get updated for sentence one.
Wherever there is a food I will put it as one.
Not is not present.
I will put it as zero good is present.
I will put it as one.
Now wherever food good is present, I will put it as one.
Wherever food not is present, I will put it as zero.
So food not is not present over here, right?
If food not is present over here, I will put it as one.
But right now it is not present.
Not good.
Over here with respect to sentence one is also not present.
So I will make it as zero.
Now if I go with respect to sentence two.
Okay.
Now see the difference between the vectors will be very much there.
Okay so food is there.
I will put it as one not is there.
I will put it as one.
Good is there, I will put it as one food, good combination.
You can see that continuous combination is not there.
Let's say I'm going to put it as zero food.
Not is there food not is there?
Obviously I'm going to put it as one.
And food not good is also there.
I'm going to put it as one.
Now here you can see that if I try to compare these two vectors now, they are not exactly the same.
And there is a huge difference because I have one difference over here, two difference over here,
three difference over here, four difference over here.
Now my model, my model will be able to clearly distinguish these two sentences because I have implemented
something called as Ngrams.
Right.
Now if I probably consider trigrams, that basically means I'm going to make a combination of three
words, right?
So in sklearn here in sklearn you have a very important parameter which is called as n-grams okay.
And you can make a combination of something like this.
If I make a combination of one comma one, that basically means I'm just going to implement unigram.
Unigram basically means only one word vocabulary okay.
If I write the combination like one comma two, then what it is going to do it is going to consider
something like unigram and bigram, like how I have explained you in this example.
This is a combination of unigram and bigram.
Okay.
And if I probably consider like this one comma three.
Then what I'm actually going to do I'm going to get a combination of unigram bigram and trigram.
So trigram basically means three three words.
Write all the possible combination with respect to this.
And then suppose let's say if I'm writing two comma three then it is not going to take unigram.
It is going to take the combination of bigram and trigram.
And this is what I'm going to show you probably in the next video.
But understand why we are specifically using N-grams over here.
Right?
Super important topic.
Since we are going to use the combination of all these words, it will give us a very good contextual
information about the sentence and wherever.
Do you feel that the sentence is completely opposite?
But still we are getting the similar kind of vectors.
This can be very, very handy.
So I hope you are able to understand about N-grams.
And in the next video we will try to do some practical implementation with respect to this, which will
be superbly important.
Okay, so let's go ahead and I'll see you all in the next video.