Hello guys.
So we'll be continuing the discussion.
With respect to natural language processing.
We are still in text pre-processing techniques.
We have seen tokenization.
We have seen stemming.
We have also seen its different types.
Along with that we have seen Lemmatization.
Now we are going to consider a topic which is called as Stopwords.
So in this video I'm going to discuss about Stopwords the importance of Stopwords.
And again I'll show you with the help of NLTK.
Now, text processing is a very important step in natural language processing because you really need
to clean the data.
You need to make the data in the right format.
Later on, we'll try to convert all this text data into vectors, and then only we'll be able to train
the model.
Because model you know, whenever we say any machine learning model internally we really need to train
with some mathematical equations.
So whenever we train something with mathematical equations there, we really need to give the input
data in the form of numerical or floating values.
So let us go ahead and let's us understand what exactly Stopwords is.
So I have opened a new notebook file over here.
So here I have one amazing speech from doctor APJ Abdul Kalam.
He was the former President of India and it was an amazing speech.
You can probably read out completely over here and obviously it is given in the materials.
Now, what I'm actually going to do is that I'm going to probably talk about Stopwords and why it is
important that we should try to remove the stop words.
Okay.
And, uh, just for a definition, what exactly stop words.
Now, here in this particular speech, you can see that, uh, there are a lot of sentences like, I
have three visions for India in 3000 years of our history, people from all over the world have come
and invaded us.
So this is the entire speech.
It is an amazing speech.
If you are probably learning it, I would like to just, uh, tell you that.
Please read this.
You'll be getting a lot of information out of it.
Very motivational speech altogether.
Now, from this particular speech, you can see that.
And I can definitely say this as paragraph or corpus right now here there are some words like I the
have and, you know, uh, let's say off uh, the you know, two there y right.
All this kind of words, right?
It will not play a very big role when we are doing a task like spam classification.
Or let's say if you are trying to do some kind of task with respect to, uh, you know, uh, like spam
or ham classification, I have already told about that.
And uh, along with that, to just see that whether this is a positive review or negative review, but
some of the words like not can actually play a very important role.
Not and all.
So what we do is that with the help of stop words, you know, we try to remove this particular words
because, uh, with this kind of use cases where you are specifically focusing on some of the important
words to determine the output, this is all words like I the he she off there is not at all required.
So what we can do is that we can basically pass this entire paragraph to that particular stop words
and see that what all words can be basically removed, okay.
And that is the importance of stopwords in short.
So let us go ahead right now.
I'll just go ahead and execute it.
Let me make some cells so that it will be very much easy for you all to understand and how we can apply
Stopwords.
Along with Stopwords, you can also apply stemming.
So I'll show you both the combination, uh, which will be super important for everyone.
Okay.
So let's go ahead and let's uh, try to do that okay.
Now first of all, I really need to import, uh, for stemming.
You obviously know what we need to import.
So from for stemming I'll write for NLTK dot stem import porter stemmer.
So I can basically write Porter stemmer over here and I'll execute it along with this.
Uh, I obviously need to also import Stopwords because stopwords for English, it will be different
because you'll be having that entire list of words like the he, she and.
All right.
so what I'm going to do is that I'm going to say that from an NLP k dot corpus import stopwords.
So from this I will be able to use the stopwords itself.
So now I have imported uh from an antique corpus import Stopwords.
Now this Stopwords, you know, I have to also download it.
So for that, uh, let me do one thing.
I'll also import a NLTK and I will execute it.
And along with that, I'll write an LCC dot download.
And here in the parameter I am going to give about the stopwords.
And there are different different language stopwords also.
And we'll also try to see that.
So if I probably write this here, you'll be able to see that downloading package Stopwords to this
particular location.
So the package Stopwords is already up to date and I'm getting true.
So in short I've downloaded all the different different languages Stopwords which is already present
in the NLTK library.
Perfect.
Now we have done that.
Now let's see that what are the Stopwords that are available in English?
So in order to do that I am.
I have already imported from NLTK corpus import stopwords.
I'll take the Stopwords.
I'll copy and paste it over here and I will say dot download okay or instead of download I will just
write dot words.
And here I just need to give my language.
Like what language?
I really want to, uh, give it for like English or something else, like German and all.
So I'm just going to write this.
Let me just write English.
And if I execute this here, now you can see the list of all the stop words that you obviously have,
and all the stop words can actually be removed.
Right now.
You may be thinking, Krish, this may depend on data to data.
Right now here you can see that guys, this is a list.
You can also create your own stopwords in English like let's say over here.
Some of the important words are like are and couldn't.
Right.
These are words can actually play a very important role.
Uh, to find out whether a statement is positive or negative, like not is also there.
If you probably search for it not, you will also be able to find it not.
Okay.
So it is always a good way that you create your own stopwords and try to remove all those kind of words
from the paragraph.
So I hope everybody is able to understand now with respect to English, you have this.
Now let's see whether we have with respect to different different language.
And obviously you can go ahead and check the documentation, but I will just try to show you with respect
to German.
So in German also you have this specific stop words.
Along with this you can also use French.
You have this particular stop words.
So with respect to different different texts or different different language of text, you can definitely
apply different different stopwords with respect to that.
Now you may be thinking, is there Hindi or Arabic or some other?
I think for Arabic also, I think it is there.
Let's see whether it is there or not.
Yes, for Arabic also it is there, but I do not find it for Hindi I guess.
So again from the documentation you can check it out.
But till Arabic I was able to see it again.
All the information will be given in the documentation.
Now what I'm actually going to do is that my sentence is already English.
Right now I'm going to perform two important tasks.
One is I will apply stemming.
And before applying stemming, you know what I'm actually going to do wherever I find out the stop words,
I'm just going to remove the stop words from this particular paragraph so that this entire paragraph
will be shortened up.
Right.
So for that, what I'm actually going to do.
Now, see, whatever things you have learned from starting everything, I'm actually going to cover
up.
Okay.
So first thing first I'm just going to say from NLTK dot stem I'm going to import porter stemmer.
Porter stemmer.
And I'll go to just execute I'm just going to execute this okay.
And then what I'm actually going to do is that I'm just going to write Stemmer is equal to Porter stemmer
this.
We really need to initialize it.
Now when we do this task right now, the next step, what I'm actually going to do is that I'm going
to perform the tokenization on the entire paragraph.
So for that I can use NLTK, dot, send, tokenize.
And here I'm just going to give my paragraph.
Now see this guys here I'm going to get my entire paragraph, entire sentences like see I have three
visions for India.
Then in 3000 years this all things I am able to get.
And this is my second sentence.
Third sentence.
Fourth sentence.
Like this.
All the sentences in the form of list I'm able to get just by using cent underscore tokenize.
Right.
This is a tokenization process wherein we take a paragraph, divide that into sentences.
Okay, now let me do one thing, is that I'm just going to save this in a variable which is called as
sentences, which will let later become a list.
Right.
So this is my sentences.
And if you probably see the type of sentences I'm just going to basically see this.
It is a list now.
Perfect till here.
We have done it amazingly well right.
We have done Porter Stemmer on that.
Sorry.
We have initialized stemmer over here and we have tokenized it.
Now understand, what we are going to do is that I'm going to traverse through all the sentences.
First of all, apply a stopwords which all words are not present in the stopwords we are only going
to take that and apply stemming.
This is what we really want to do.
So here I'm saying that first of all apply stopwords and filter and then apply tokenization.
Right?
Sorry.
Then apply stemming.
So this is the step that I'm actually going to do.
Now see this very simple very important.
So I'll write a for loop saying that for I in range for I in range.
And here I'm going to basically give the length of the sentences.
I can also go with respect to sentences.
But there I'll not be getting the indexes here.
I'll be getting the indices.
Okay, so range basically says that whatever length I'm actually giving that becomes an index, right?
Zero to that specific length.
Now what I'm actually going to take, I'm going to take this specific n um, I and I'm going to write
n nltk dot word tokenize because I'm getting in the form of sentences, I need to get each and every
word right.
So I'm getting the word over here and inside word underscore tokenize I'll give I sorry sentence of
I because this will be an index sentences of I.
So this will be an index.
And here I'll be able to get the word.
So here what I'll do I'll make a list of words.
So in short I'll be getting the list of words inside the sentences.
Perfect.
Now till here we have done it.
Now after this we are going to apply one very important thing that is.
First of all, I need to apply stop words for each and every word and see whether it falls in the stop
word or not.
If it does not fall in the stop word, then only I have to do the stemming.
So understand the task step by step.
This is super important with respect to all the steps that I'm actually taking up.
Okay, so here I will write a list comprehension.
I will say stemmer dot stem okay.
And here I'm going to write dot word okay of the word because uh from this particular words, this word
is a list of words.
And I have to take each and every word.
So here I will write a for loop okay.
And this is called as list comprehension.
So I'll write for word in words if word not in see if the word is not present in the stop words then
only you apply stemming.
That is what I'm actually trying to do.
Okay, so here you can basically see if the word is not in.
I'll use a set.
Along with that I will download all the all the stop words with respect to English.
Right.
So why I'm using step set because some of the words may get repeated.
So I don't want that.
So I'm going to basically write it over here right now through this.
What I'm actually going to do I'll get that specific word that is not present in the stop words.
And only stemming will be getting applied to that specific word.
Perfect.
Now here, what I'm actually going to do is that I'm going to save it in a variable called as words
perfect.
I hope it is very much clear.
I'm getting back everything after doing the stemming back in the words itself.
And then finally, what I'm actually going to do is that I'm going to take this sentences, and I'm
going to replace it on the same index with respect to this words.
But once we get this words right, I need to join all these words together.
So how do I join?
There will obviously be a space dot.
I'll just use dot join so that it will join together and it will convert that into a sentences.
So this exactly is converting all the words into sentences, right.
This is very much simple.
And we have actually done this.
Perfect.
So here what all things we have done again let me repeat I'm I'm iterating through each and every sentence.
And then I'm doing a word tokenize that basically means for every sentences I'm getting the list of
words.
And from that list of words I'm iterating, I'm seeing that whether it is present in the stop words,
if it is not present, I'm doing the stemming.
After stemming, I'm again storing back in that same list, and then I'm converting all these words
into sentences I can say, converting all the list of words into sentences.
Perfect.
Now, once I execute it.
And now if I go and see my sentences here, you'll be able to see now I three vision India right in
3000 year history.
Right.
So here I h I s t o r y became RI okay people, people it became people.
World came, invaded us, invade it, became captured.
Land conquer mined from.
You can see all the special words like like I, I have.
Everything is gone.
See, see this I is there have is gone.
Right.
Though you will not be able to find out though anyway, right?
So whatever.
Stop.
Words were present over here that all got removed.
And then only we performed the stemming.
Right now you may be saying crush.
Uh, the stemming does not look very good, right?
So for that, what you need to do, I've already taught you with respect to snowball stemmer, so I
will just import this.
Okay.
And it is very simple.
Simple.
I think you can do the same task again.
So snowball stemmer and then I will try to import this with respect to English.
And obviously after this you'll be able to get good sentence.
Right.
So let me just remove this one.
So snowball stemmer I've already done it and I'm going to copy the same thing.
Okay.
And here I'm just going to say apply snowball stemmer stemming.
Right.
And then instead of stemmer I will just write this word that is snowball stemmer.
That's it.
Yeah.
And now once I execute it, uh, let me go back again, back to the sentences, because that sentences
have got changed now.
So where is the sentences?
Let's see.
Okay.
Now this is the sentences I've got executed now.
Now I'm just going to execute this.
Now if I probably go and see my sentences here, you can see that now it is good right now.
So uh, one more important thing that snowball has done.
See over here, I still have capital letters, right?
Like there may be some of the sentences which may be in small letter it also.
So that becomes a repeated word.
But since this is a capital letter, it will be considered as a separate word right for the model to
understand.
Right?
So what it does is that snowball.
One more advantage is that it is making sure that all the letter is becoming small.
Right.
So all the letter is becoming small.
And for some of the words it is not even giving a good result like poverty has become poverty.
But if you try to do this with the help of Lemmatization, you can also get a good word.
Now let's try it with the help of Lemmatization.
So I'm just going to, uh, do the same thing.
Okay?
I hope everybody is understood with respect to snowball stemmer.
Now, what I'm going to do is that again, going to go back to my Lemmatization code.
So I'm just going to import this NLTK dot stem.
And it is very simple guys.
I think we are just repeating things so that you also practice in a better way okay.
so here I've got the word net lemmatizer I'm going to initialize it.
Perfect.
This is done.
Now I'm going to go to go ahead and copy the same code right.
And I will just write it over here.
And instead of writing snowball I'm just going to copy this I'm going to paste it over here.
Perfect.
So I've done it.
But let me just go ahead and execute the sentence part again because I need to get the updated sentence.
So I think it is somewhere here.
Paragraph.
Okay.
And uh, here is the sentence.
Okay.
Perfect.
Now let me just go ahead and execute the same thing.
And now if you okay I'm getting an A has no Stem okay.
Sorry.
It should be lemmatize.
Lemmatize.
Okay.
So Lemmatize or Lemmatize.
Now, as you see, some time it took.
Right?
Because it is basically checking from the entire corpus.
Now if I probably go and see the sentences now, you have all amazing things with respect to words that
are coming up correctly and all the things are here, right.
So with respect to this, you are able to get some good thing.
Now one more thing you can do is that after this word you can also put the post tag.
Now if you put the post tag to V right now, you see what will be the output.
You'll get a better output I guess because most of the words it will be basically considering as, um,
you know, as an adverb or sorry as a verb.
So but anyhow, we will try to understand about post tag again.
More more things.
Right.
So I three visions India in 3000 years history people come would world come invade us capture land.
So all the stop words has got deleted.
Now we are getting a very good one.
You know, uh, at least better than stemming.
Okay.
And the other one that is snowball stemming.
Right.
So this was the entire process with respect to text pre-processing.
And here we have discussed with stop words and how you should also go ahead and do the text pre-processing.
I hope everybody got that idea.
Now in Lemmatization also you can see that it is not lowering it.
So what you can actually do is that you can basically lower all the sentences.
right.
So let's say if you write sentences of I is equal to sentences dot of I dot two lower right.
So you can basically give two lower.
Let's see whether it will work or not.
Uh, and uh let's see whether it will completely work or not.
I'm not sure whether dot two lower will work with respect to list, but I think it should work.
I'm just going to execute this again.
Go down okay and apply this Lemmatizer.
Okay.
Uh str object has no attribute to lower okay.
So it's okay.
Not a problem.
What I'm actually going to do I'm just going to comment this.
And before writing here I'll just say word to lower okay.
You have to definitely try different different things okay.
And it's all about Google.
You know, just Google it.
You'll be able to get it.
So again I'm going to execute this.
And now I think it will execute I know it is.
I'm doing a lot of up and downs.
But just try to follow the lecture.
Uh string object has no attribute to lower y to underscore lower is there.
To lower.
Um, okay.
Let me just see.
Uh, str two lower case.
Right.
Python.
Let me just see this.
It's okay if I don't have any other way to see it, but I think dot lower will definitely work.
Let's see.
Dot lower.
Okay.
Perfect.
It has worked.
Now, now if I go and see the sentences, it has done okay.
So I don't have any any regrets to search in the Google.
You should also do the search in the Google.
So now here you can see all the small letters are there along with the lemmatization.
So this is the entire text pre-processing.
We can also apply some regular expression before this so that we can do the cleaning of the sentence.
So I'll remove this word.
And I think uh, this will also work if I probably just comment this out with to lower or just lower.
Okay.
Just check it out.
It is up to you.
So I'll just comment this out so that you can try it out.
Okay.
But in short, uh, we have understood what Stopwords can actually do and how we can basically apply
things.
Okay, so yes, uh, this was it from my side.
I think you are liking this entire series.
Uh, we will be learning more things as we go ahead.
Um, uh, we have post tags and all.
We have entity name recognitions.
Also, many things are there which is going to come.
So yes, uh, keep on learning, keep on practicing.
Uh, I will see you all in the next video.
Thank you.