Hello guys.
So we are going to continue our discussion with respect to natural language processing.
And now we are going to move towards some more techniques with respect to text pre-processing.
Already in our previous video we have seen tokenization.
We have seen that how we can convert a paragraph into sentences and then probably a paragraph into words,
or converting a sentences into words.
Right?
So in short, we have seen that how we can actually do tokenization with the help of NLTK.
So in this video we are going to focus on something called as stemming, which is a very important process
altogether.
And uh, what exactly is stemming?
I'll also provide you the definition.
We'll also see a lot of examples with respect to that.
And we'll also try to see the different types of stemming.
So I have opened a file over here.
So here you can see regarding the stemming.
And I have also given the definition.
Now let's understand what exactly this is with some good examples.
Now first of all we'll see the definition over here.
It shows that stemming and this definition is taken from the Wikipedia.
So stemming is the process of reducing a word to its word stem okay.
This is super important guys to its word stem that affixes or suffixes or prefixes or to the roots of
the word known as lemma okay.
Stemming is important in natural language understanding and natural language processing.
Now what exactly this is?
I'll tell you some examples.
Let's say I want to just create some cell below okay.
Over here if I have a lot of text.
Right.
Let's say that I'm trying to solve a sentiment or I'm just trying to solve a classification problem.
Classification problem.
And the classification problem is very simple.
We basically need to find out whether the comments on the product is a positive review or negative review.
Right.
So when we are solving this kind of problem statement.
So in this what we'll be having in our data set, we'll be having the comments or reviews I can say
I'll be having reviews and based on this particular reviews and this reviews will obviously be some
kind of text data and we need to basically create a model where then we can basically classify whether
it is a positive review or negative review.
That is very simple.
Now usually in this reviews let's say that I have some of the words like eating okay.
Or it can be eat right or it can be like eaten right.
It can be different kind of words, but at the end of the day, it actually represents the same thing
regarding eating.
Right.
So this is basically eat.
Eat is the root word.
Or I can also say it is the stem word, word stem of this all the words.
Right.
Because eat is very much common.
And having this variety of words for a problem statement will not impact much with respect to finding
the output like positive or negative review.
Try to understand what I'm actually trying to say over here.
I may have different kind of words like eating, eat, eaten, or there may be also other words like
I can also make a combination of like going, gone, right, gone goes right.
At the end of the day, it is basically talking about go right.
So go is a word stem of all these words that are present, right?
So it is not necessary that we need to have similar kind of words again and again, because this increases
the number of input features.
In short, because each and every word represents a vector, as we'll see as we'll go ahead, you know,
after text pre-processing will try to convert this text into vectors.
So having this similar kind of words instead of having this similar kind of words, I can just have
one word that is just like go right?
And it will try to and will try to solve the problem with respect to that.
So stemming is actually helping us to do this same thing.
So finding this word stem can be actually done with the help of stemming.
And there is also a concept which is called as Lemmatization will try to understand the difference as
we go ahead.
But first of all, let's go ahead and see that how with the help of NLTK, we can perform stemming.
Okay, so what I'm actually going to do is that I'm just going to take some examples.
Let's say that I have all these words.
Okay.
So I'm just going to remove this.
And let's say that I have all these words now.
Right now you have words like eating, eats, eaten, writing writes, programming programs, history
finally and finalized okay.
So I'm just going to execute it and let me make some more cells.
Okay.
So I'm just going to delete this cell because I don't require it.
Okay I'll just create a cell below.
Okay.
And let me express exp exp and let us go ahead.
Right.
So here you can see all these particular words are there.
Now let's see how we can find out the word stem of all this particular words with the help of stemming
the first stemming technique that we are probably going to use is something called as Porter stemmer.
Okay.
Porter stemmer.
And they are again different, different types of stemming techniques, which I will probably be showing
you.
And then we'll be able to understand that what all things it will be able to give us.
Okay.
Now in order to apply this Porter stemmer, it is very much simple in analytic.
Already those functionalities basically present.
So I will just write from NLTK dot stem.
Import porter stemmer okay, so once I initialize this here you can see this I'm going to use this Porter
stemmer which is just a kind of class.
And for this we have to initialize it.
We have to initialize an object for that.
So let me just create an object and this will basically be my stemming.
And once I do this now in the next step, what I'm actually going to do for each and every word, I'm
just going to apply the stemming process.
Okay.
So it's very simple.
How do I do it?
I will just iterate it.
So I'll say for word for word in words right.
And here I'm basically going to write print and let me write it down as word plus.
I'll just give some type of marking.
This is the word and the stemmed part will be nothing, but I'll be using the same object stemming dot
stem.
There is a functionality.
There is a function called as stem which will actually whenever we push any words inside this, it will
do the stemming thing that basically means for eating.
Probably it may give you eat for eats, it may give you eat.
Right.
So something like that.
So I'm just going to give my word over here.
So once I execute now here you can see that perfect eating is coming as eat eats comes as eat eaten
is coming as eaten.
Only writing is coming as right, which is good.
Writes comes as right.
Programming comes with program programs is nothing but program three.
Here you can see history is becoming history.
Uh, h I s t o r I.
So here it is a major issue.
Right.
And I'll also talk about the disadvantages.
Finally becomes final.
Finalized becomes final.
This is this is there right now let's say that, uh, over here it looks good, right?
Probably for Eton, you can see that nothing has happened.
It is given the same word.
But if you see some words like history now here you are actually getting history, right?
So the entire meaning of this particular word has actually changed.
And this is the major disadvantage of stemming when stemming is basically applied, you know, for some
of the words, you know, you may not get a correct exact meaning, the form of that specific word may
change.
So this is the major, major disadvantage with respect to stemming.
Let me show you some more examples.
Now suppose if I say stemming dot stem, I'm just going to apply this particular stem word on a word
called as congratulations.
Okay.
And if I execute it here you can see the word.
The meaning of the word is basically changing.
It should have told like something like congratulate right.
But here you can see congratulate.
It is being it is basically changing the form of the word.
Now the word does not have any kind of meaning.
Right.
So this is again one major, major disadvantage of stemming.
Okay.
Similarly, if I probably try to show you something like stemming not Stem, and if I probably write
something like sitting.
So here you will be able to see.
Let me just write it down.
Sit.
Now see first sitting it is giving a very good word.
So that basically means stemming works for uh very good number of words.
But for some words it does not give us a good answer.
Right.
So this is the major disadvantage of stemming.
And this all will get fixed with the help of lemmatization.
But whenever you have any kind of problem, statements like, uh, classification, problem review,
classification or we really want to see whether it is whether an email is a spam or a ham, right.
Whether it is spam or not, a spam, we should definitely go ahead with using stemming, you know,
and uh, And again, some of the words will not come in the right form.
But yes, instead of using Porter Stemmer, we have other different kind of stemming stemmer techniques
which we can definitely use to improve it.
Okay, now let me go.
With respect to the second one, now with respect to the second one, I'm just going to delete this
cell because we don't require it.
The second cell is that which is called as Régis Stemmer class.
Now this regular expression stemmer class it is nothing, but this is a class with the help of which
we can easily implement regular expression Stemmer algorithm.
So we can just provide a regular expression and it will be able to apply the stemming purpose in that.
Okay, so it basically takes a single regular expression and remove any prefix or suffix that matches
the expression.
Perfect.
Now what I'll do, I'll just create a cell below.
And again I will make some more cells and I'll try to show you some example.
Now first of all we need we need to initialize this.
So I will write from NLTK dot stem import.
What is the name regex.
Right.
So I'm just going to write reg stem perfect.
Then we basically need to initialize it.
So I'll say reg underscore stem is equal to stem right.
So I've initialize it.
This is perfect.
Uh, it is giving us an, uh, error saying that regular expression is required.
Now, this is super important.
Now let's press go ahead and press shift and Tab.
Now here you can see the first parameter that goes is something called as regular expression.
Okay.
And this is some minimum value.
We'll try to understand what exactly it is.
Now here you can see a stemmer that uses regular expression to identify morphology affixes.
Right.
Understand this.
morphological affixes.
I'll try to show you an example.
If you don't know the meaning of this, any substring that match the regular expression will match will
be removed.
So this morphological affixes is basically, in short, a regular expression which will match whatever
words we are basically giving, and if it matches, it will get removed.
Okay, now let's take some example.
Okay.
Over here, uh, and uh here you can see that this parliament, uh, is the minimum length of the string
to stem.
Okay.
So if the minimum length is somewhere around four, then only you will be able to apply this.
So I'm just going to apply this same thing and I'm going to paste it over here okay.
So here is my index.html.
Now what I'm actually going to do I'm just going to write reg dot stemmer.
Now you should understand over here what I have actually given.
I have given ING dollar.
Okay, here I am given s dollar, e dollar able dollar.
Okay, so I have given all this regular expression and this will make completely sense when I will be
implementing it.
Now you see, just pause the video and just let me know what is the output that you feel will be getting
for this.
Eating now.
Eating.
You should see over here ING.
Is there a regular expression?
Is there so.
But dollar is also there.
Okay, this is super important.
Dollar is also there.
Now if I try to execute this I am able to get something called as eat.
Okay.
Now in short what this is happening is that this is basically saying that wherever on the last word
it is ING, or it is s, or it is E, or it is able, Just try to remove that now.
The next example that I really want to give is that let's say I want to I have this particular word
I n g.
Okay.
So here you can see I n g eating right.
What do you think the output will be.
Will it.
Whether it will be eat or whether it will be something else.
Okay.
So here, if I try to execute it here you'll be seeing I n g eat.
Why.
Because the regular expression says that only in the last last dollar.
If I probably remove dollar.
If I just execute it now, see?
Everything will get removed.
Okay?
Or probably I just want in the starting.
So I'll just use this.
So here you can see I enjoy eating.
So this this will not work okay.
So we can basically have something like this uh, and we can basically check how we can actually remove
it.
So this is perfect.
So here you can see wherever I ng is there it is getting removed.
Perfect.
So this is good.
We have done this okay I ng it.
Now let's say that uh I also want to try something else.
Uh, you can definitely try with different different word.
You can use disable e s whatever regular expression you can basically.
Right.
You can go ahead and write it and you can check it okay.
So this is with respect to regular expression Stemmer class.
Now uh, the next one that we are basically going to discuss, uh, and uh, let's see where this is.
Porter stemmer this, this this this is uh, again it has written the same thing.
I don't want to write the same thing.
I'll go ahead with something called as Snowball Stemmer.
And this is also an amazing technique.
And this is a better technique when compared to okay, so snowball stemmer, I think I've given the
definition somewhere here.
No no no no okay.
Snowball stemmer is again a stemming technique.
But in this snowball stemmer it is.
It performs better than this Porter stemmer.
Okay, that is the reason why Snowball stemmer had actually come.
Initially we came up with Porter Stemmer.
We saw that a lot of things.
A lot of words were getting messed up, you know.
So that is the reason why we use Snowball Stemmer, because it gives a better accuracy when compared
to the, uh, porter stemmer.
When I say accuracy, better form of a word.
Okay, now for using snowball stemmer.
What I'll do, I'll write from sklearn dot stem import snowball stemmer.
So here you can see I'll just write snowball.
For my sklearn.
Oh, not for my sklearn.
Sorry for my NLTK because it is basically present in the NLTK, right?
So from NLTK dot stem import snowball stemmer and I'm just going to import it.
Then we are going to initialize with respect to snowball stemmer.
And let's see what are the parameters.
So here first of all snowball stemmer is also provided with different different languages like uh Arabic
okay Danish, uh, English, French, Finnish, French, German, Hungarian, Italian.
So you can use basically all these words.
Okay.
So for right now I'm just going to use English.
So what I'm going to do is that I'm in quotes I will just say English okay.
And uh, I'll just use this.
And finally I will basically use a snowball stemmer over here.
I will just create a variable okay.
And just execute it.
Perfect.
Now the next thing is that I will just use another condition saying that for word in words okay.
And I'll just write for words in words.
So I'll write snowball stemmer dot stem on word okay.
But I'll print this in a better way so that you'll be able to understand it.
So print word plus.
And this will be like this.
Something like this.
Just for formatting purpose.
I'm writing this so that it'll look good, better for you all.
And here I'm just going to write the word right.
So same thing what I did in the things.
So here you can see that I'm getting eating as eat eats as eat eat and as eat.
Right right.
This is all fine right?
For history.
Also, it is not being able to give the correct form.
Right?
Then you may be thinking, Krish.
Then what is the difference?
Right?
But let's see with respect to some porter stemmer where this porter stemmer will give some bad results
or bad form of the word, and where the snowball will give us a better form of a word.
So for this, I'm just going to, uh, execute this line of code.
So here you can see when I'm applying this stemming.
This stemming was for the Porter stemmer right.
When I apply stemming dot stem on fairly and sportingly.
So what is the output that I'm getting?
I'm getting fairly and sportingly right.
L I l I.
But if I try to use the same word with respect to snowball right here, you will be able to see.
Just let me write the snowball stemmer here.
We'll be able to see.
I am getting some amazing answers.
Snowballs stammer okay.
Okay, let me just remove this.
I think there is a problem with respect to this.
Now it will work.
Okay.
Snowball stemmer comma I will just copy this and I'll apply to the same word something called as sportingly.
Okay.
So now if you probably go and see I'm getting a good output which is like fair and sport.
So all together you will be seeing that snowball stemmer when applied to various other words performs
better than stemming.
Right.
So here see Porter Stemmer is obviously a technique where you'll be able to find out the word stem.
But for many of the words it is not being able to give a good answer.
Some of the examples are like fairly, sportingly and all.
Now, in order to overcome this disadvantage, Snowball stemmer is basically used.
And again guys understand these are some techniques which will actually help you to find out the word
Stem and where it is specifically getting used in text preprocessing.
You really need to clean the data.
You need to make sure that the data is ready so that we will be able to convert it into vectors in an
efficient way.
Right.
So in this part we have seen about stemming.
And again one major disadvantage of stemming is that obviously see even though Snowball Stemmer is performing
exceptionally well, but some of the words like history, there will be also something like going right.
So suppose if I probably write this right snowball dot stem right.
And if I use going I think going whether it will give us good or not.
So going is performing well if I right goes.
So here you can see for goes.
Also it gives us a bad word right.
Even stemming dot stem if I probably try to see this right.
So in short how much we try.
Right.
For some of the words obviously the form of the word is changing.
So this is one of the disadvantage with respect to stemming.
And understand that for use cases like chat bots and all these techniques cannot be used.
So for that we have to go ahead with something called as lemmatization, because Lemmatization solves
all this particular problem because it has the dictionary of all the words, all the root words that
is basically there.
So whatever word you basically give, it will be giving you a good grammatical form of the word.
Like If I'm giving goes, it will be go.
If I'm giving fairly, it will be fair.
You know, if I'm giving eating, it will be eat.
So that kind of disadvantage is getting removed completely with the help of Lemmatization.
And that particular part we'll be seeing in the next video.
So I hope you have understood till here.
Please make sure that you try practice with different different examples.
And yes, this was it from my side.
I will see you all in the next video.
Thank you.