Hello guys.
So we are going to continue a discussion with respect to natural language processing.
In this video we are going to discuss about Lemmatization.
In our previous video we have already seen something called as stemming.
We understood that stemming is the process of reducing a word to its word stem.
Right.
And we understood what is the disadvantage of stemming because for some of the words right, when we
perform stemming, we do not get the correct form of the word then and the entire meaning of the word
actually gets changed.
And please remember this word which is called as word stem, right?
So in short, this is just a kind of algorithm which tries to find out the word stem.
But for some of the words it is working absolutely fine.
We have seen different types of stemming techniques like porter, stemmer, regev stemmer, snowball,
stemmer.
And we also found out that snowball stemmer was better than Porter stemmer.
So all these things were covered today in this video.
Right now in this video, we are going to discuss about Lemmatization and the Lemmatization technique
that we are going to use is something called as word net Lemmatizer.
I've already told you what is the disadvantage of stemming, because the words that we are getting is
not in the correct form.
The meaning of the word is basically changing.
So in order to prevent that, we use something called as Lemmatizer.
And in this we have something called as word net Lemmatizer.
Now what is Lemmatizer okay.
Lemmatization technique is like stemming the output will we will get after.
Lemmatization is called lemma, which is a root word.
Understand this, which is a root word like.
Suppose if I have eating, then it will become.
Eat is the root word right?
If I probably talk about history, history is the root word, right?
If I talk about go goes, go is the root word.
Right.
Over here we are getting the stem word right in stemming right.
It will probably apply an algorithm.
And with respect to that it will try to find out the stem word right word stem.
We basically say it this as.
But here we get the root exact word.
So again the main aim of Lemmatizer is to give you the exact form of the word which is meaningful,
and it does not change the meaning as it was happening in stemming itself.
So here you have rather than the root stem, the output of the stem.
Now again I'll repeat the definition.
Lemmatization technique is like a stemming.
The output we will get after lemmatization is called lemma, which is a root word rather than the root
stem.
The output of stemming after lemmatization we will get.
We will be getting a valid word that means the same thing.
So suppose if there are words like eating, eats, eaten, it will become eat only right?
So it will be giving you a meaningful word which will represent many words over there.
Okay, so NLTK provides word net Lemmatizer class and I'll try to show you with respect to this and
understand guys, uh, you know, this lemmatization occurs or they it, it gets performed with the
help of this word net corpus reader.
So there will be a dictionary kind of words which will be comparing from there.
And it will be able to do the word net very much properly.
So first of all, let's go ahead and see that how we can basically implement it.
So first of all I will try to import from an LCC stem uh import word.
Net Lemmatizer.
Okay.
So I'm just going to import this and this word.
Net Lemmatizer.
Uh, again helps us to perform Lemmatization.
Uh, so first of all, I'll create an object.
So I will write Lemmatizer.
And here I'm just going to write WordNet Lemmatizer.
So here if I see this, it is nothing but WordNet.
So let me just go ahead and execute it.
So this has got executed perfectly.
Now one thing that you really need to see that okay.
So I will just be writing something called as let's say let's try some easy word okay.
So Lemmatizer is equal to oh I'm just going to say dot lemmatize.
So there is a function which is called as lemmatize.
And here I have to give my words.
So let's say if I'm giving going and if I try to see the answer I'm actually getting going.
So that basically means that it is a uh, it is trying to find out the root word with respect to this,
but let's go and see the functionality of Lemmatize.
In Lemmatize we give two important parameters.
One is word and this is something called as post tag.
Okay, I will talk about this post tag.
Right now it is written as n.
N basically means this word that I'm actually passing.
It is being treated as a noun.
So you may be thinking, Krish, how many uh post tax will be there?
So let me just write down a comment over here.
Okay.
So here you have different different post tag for noun.
We give it as small n for verb.
We give it as v for adjective, we give it as a for adverb, we give it as r.
Now what I'm actually going to do by default right now the post tag is n, right?
So for with respect to n I'm getting this output.
Okay.
Let's say I change it to V because I just want to show you whether this will be changed or not.
Now when I give post tag as n we I'm just saying that consider going as a noun keyword, but going is
not a noun keyword, right?
But if I consider with respect to verb, now you see with respect to verb, I am able to get a good
lemmatization that is like go, okay?
Similarly, if I probably try to see with respect to a adjective.
So here you can see I'm getting going, right?
And obviously this going is a kind of verb, right?
It is not an adjective.
And let's say if I try to go with respect to adverb here also I'm getting going right now with respect
to this going word.
Right?
I feel verb is the correct one, which we really need to selected.
Now let me do one thing though.
All the words that I had copied over here.
Right.
Let's apply this entire lemmatization on this entire words.
Okay.
So I'm just going to execute this and execute it.
And again I'm just going to write this for loop okay.
And I'm just going to copy this instead of writing stemming dot stem I'm just going to write Lemmatizer
dot Lemmatize.
Okay.
And I'm just going to apply for this words.
Now see this eating is becoming eating eats becomes eat eaten eaten writing writing writes write programming
programs programs has become program because again by default in this lemmatize remember that I have
given my post tag as n write.
We are considering that all these words, all these words are basically noun.
Okay, so programming programs history finally.
So here you'll be seeing that Lemmatization has not occurred that much.
We have not done any kind of we have not able to find out because it is being considered as a noun and
in noun.
Whenever we give noun.
Suppose let's say that all my words has names and in names.
Obviously those lemmatization does not occur, right?
So if I have like Krishna, if I have Sudhanshu Kumar, if I have this kind of words, which is name,
right.
In short, this is nouns, right?
We have some famous place name like Taj Mahal, India, uh, places name and.
All right.
So this all will be considered as nouns, right?
So uh, that basic difference you really need to know in order to understand this.
Let's say that I want to again convert this into adjective.
So here you can see eating has become it's it's writing.
Everything is same.
Right.
But now let me try with respect to verb because most of these words are basically verbs itself.
Right.
So here you can see eating has become eat eats has become eat writing write write programming program
this history.
Now you can see this, right?
Uh, history is becoming as history only in stemming.
We used to get something like history.
Right.
So this is quite amazing, right?
Finally, finally and this all.
And we could also see that in stemming.
Right.
We also have some of the words which are not actually being able to perform well.
Right.
Like fairly and sportingly and all right.
And even goes right.
So suppose if I probably go ahead and write something like Lemmatizer dot Lemmatize and let's say I'm
going to write something like goes, right.
So I'm going to get go which is good.
Right.
And here I can also play with post tag.
Right.
So let's say if I'm using a V post tag I'm going to get go.
Let me go and try with this.
So here I'm just going to write Lemmatizer okay.
And I'm just going to say try to do with this fairly.
Let's see what kind of output we are getting.
Now.
This Lemmatizer is super amazing, right?
Because it is giving you the good word form.
And the meaning of the word is also maintained.
Right?
So here you can see fairly sportingly.
Now let me write a post tag obviously because it was noun.
So it may not give you something.
So here also with respect to verb.
Also I'm getting the same answer which is good right?
So this is just to tell you that how good lemmatization is.
But one question that I really wanted to ask you, which will take more time word learning advertiser
or stemming?
The answer is simple word net Lemmatizer.
Why?
Because you can see NLTK provides word net lemmatizer class within a thin wrapper around the word net
corpus.
So it is going to compare from there, right?
It uses Morphe function to the WordNet corpus reader class to find a lemma.
So this will basically take time.
Right now I just have some number of words.
So it is happening very fast.
But if you have a paragraph, if you have a bigger sentence, Lemmatization is going to take time,
right.
So this is the basic difference between stemming and repetition for which use cases you can use this.
Uh, one simple use case is, uh, if I really want to write, uh, Q and a chat bots.
Write chat bots.
All this is an amazing examples for all this, right?
You can basically use this, right.
So let me write it down Q and A chat bots.
Right.
Uh, text summarization.
Right.
Uh, Q&A these are some of the examples which you can basically consider.
Uh, text summarization is also one example.
And in many, many companies it is being used.
So right.
So you can use basically all these things and you can basically uh implement the word advertiser over
there because you get the exact good form of the word that is the root form of the word, which is meaningful.
Right.
So I hope, uh, you were able to understand about Lemmatization, you are able to find out the differences
between stemming and lemmatization.
Uh, this was it.
Uh, and I will see you all in the next video.
Thank you.