Hello guys.
So we are going to continue the discussion with respect to natural language processing.
We have already seen in our previous video about Bag of Words.
We have understood the intuition.
We have also made sure that we understood the advantages and disadvantages.
Now let's go ahead towards the practical part and see that how we can implement bag of words.
So over here I have an amazing data set which is called as spam classification.
So if I probably open this right.
So this is a data set which is having a label like ham.
And then I have a message right.
Then again I have ham, then again I have a message.
So here you can see spam.
And then again I have a message.
So like this I have this entire data set.
What I'm actually going to do is that I'm going to basically apply on this particular data set, uh,
bag of words model.
And later on as we go ahead, all the different techniques also will try to see it.
So over here I'm just going to create a new Python file okay.
And uh yeah.
Let me just go ahead and write, uh, bag of words.
Practicals.
Right.
Perfect.
So we will go line by line and we'll try to understand everything.
And, uh, first of all, we'll read the data set itself.
Okay.
Now, before I go ahead, uh, I really need some of the libraries.
So I'm going to import pandas as PD.
Okay.
And then I will just try to read this messages.
So I'll write PD dot read underscore CSV.
And here you can see that uh, obviously it is present inside a folder which is called a spam classification
master.
Or and then it is having spam.
So let's see I'm just going to write like this spam classification master.
And then spam classifier dot p y.
So I have something like spam SMS uh, folder.
And then I can basically go ahead and write my data set name.
So inside this you'll be able to see that I have this SMS spam collection.
Right.
So I'm going to write SMS spam a SMS spam collection.
So this is my file.
You'll be able to see it okay SMS spam
collection.
Okay, so this is my data set over here.
But remember, as I opened this particular data set, there are two things that you really need to focus
over here.
Uh, the space between this in, uh, dependent and independent feature.
That is my text.
It is basically a tab okay.
so I will basically be using a separator which will be a tab.
So in the next step I will write separator.
And this will basically be a slash slash t okay.
And then uh along with this, what I'm actually going to do is that I'm going to also assign uh a names
as my like what all features it should basically get assigned to one is label and the one is message,
right?
So I'm basically providing my own custom, uh, column name in short.
Right.
So once I do this, let's go ahead and execute it.
So you will be able to understand I'm just trying to read the data set over here.
So if I probably go and see the messages.
So here you'll be able to see label and message.
Right.
So I've just read this entire data set using a slash T that is a tab separator.
And then I've also provided two names Label and message, which is my column name.
Perfect.
Till here it is quite good.
We are able to see the data set and clearly we are able to see the data set itself right.
Now the first step obviously before applying bag of words we really need to clean the data.
So I'll write data cleaning and preprocessing.
Okay.
Let me make my mic a little bit straight.
Okay.
Data cleaning and pre-processing.
Now what?
All things we can actually clean the data about.
Right.
What what all processing we can basically do obviously lowering the sentences, applying stopwords if
I want applying um, uh lemmatization if I want.
So there are various ways what all things we can basically do.
Right.
So, uh, what I'm actually going to do is that I'm going to import some of the libraries for this.
And again, I also may need to make sure that any special character that are present in this particular
sentences, I should remove it, you know, because see this kind of words.
Uh, it has two dots.
You know, these are like special words, uh, hardly being used for this kind of use cases, which
we can directly do it with the help of Stopwords itself.
So I'm going to import regular expression, I'm going to import NLTK.
And along with that I will just write nltk dot download stop words.
Okay.
And already we have done the downloading of this particular stop words if you are following my previous
videos, but let's say that you have you have directly jumped over here.
I'm just going to download it once again for you.
Okay, so here you will be able to see I've written s stop words so it is already up to date.
Uh, all these things are there.
Now, uh, the next thing what I'm actually going to do is that I'm going to write from NLTK dot corpus
import Stopwords.
Import Stopwords.
And um, I'm also going to make sure that I download from NLTK dot stem I import the.
NLTK dot stem dot porter.
Import.
Porter.
Stemmer.
Right.
And I hope everybody knows why Porter stemmer is basically required.
So I'm just going to write it.
This one.
Porter stemmer this is specifically for the stemming purpose.
And along with that I will also make sure that I'll initialize this.
Okay.
So I'll write Porter Stemmer and I will just initialize it so that we will be able to do this.
Um, okay.
I really need to write from.
Okay.
So I'm not going to edit any errors as such.
So everything whatever I am facing the error I think you should also be facing.
Or you may be also writing with respect to some mistakes, right?
Whenever we are trying to solve this perfect.
Uh, we have imported Porter Stemmer because we really want to do the stemming.
Now, what I'm actually going to do is that I'm just going to create a list which is like corpus.
Okay, since I need to clean all the data right at the end of the day, after I clean the data, it
will be stored in this variable and let me iterate it for I in range of zero comma length of messages.
Now what I'm actually doing is that I will make sure that I go through each and every sentence.
I remove all the special character, I lower down all the sentences, I apply the stopwords and then
I probably apply stemming.
So there are many things that we are going to do, and this task is common and generic with respect
to all the use cases.
Let's say that later on you want to solve any other use cases, right?
You can follow the same approach and that will actually help you to solve this problem statement.
Okay.
So what I'm actually going to do over here is that I'm just going to write colon okay.
And then first of all let me go ahead and take the review.
Right.
So here I will say, uh, the first thing that I'm going to do is that after I go with respect to each
and every review, I first have to remove the special character.
That basically means I don't want anything else other than the, uh, words or letters over there.
Okay, so over here I'll be using re and there is a function which is called as dot sub.
Okay.
Now in this particular function I will just apply my regular expression.
Now how do I apply my regular expression?
You can definitely try out different different things, but I'll give a simple regular expression.
And there is also a website where you can easily, uh, see that what all regular expression you need
to apply to select those similar kind of words.
Okay.
So I'll say, uh, I'll just give one simple, uh, thing with respect to this.
You know, I will say that, okay.
With respect to only if you if you find A to Z, okay.
Or I can also say that apart from this word like a to Z, that small A to Z or capital A to Z.
Okay, apart from this word, any words, any special character that you find, just replace it with
blank.
Okay, I'm trying to use this because there are a lot of special characters in the text which I've basically
analyzed.
And here what I am saying is that apart from the text that is small A to Z and capital E to Z, replace
everything by blank.
So here I will be having message and this same thing.
I will try to apply it.
Now messages obviously has all the messages over here right.
But inside this I have a label called as message.
So I'm just going to apply that message.
And with respect to any index that is I, I'm just going to apply this.
Okay.
So review what I'm doing is that I'm just substituting right.
This sub is basically nothing but substituting A to z capital A to Z along like apart from this A to
Z, any special character that you get, just replace it with blank and try to replace it in our message
and store it in a review variable.
Okay, this is perfect.
Now the next step what I'm actually going to do I'm also going to make sure that I lower all these words
because I really need to lower this okay.
If I don't lower this then again it will be a problem after lowering the word what I will do since I
will also be having a space things are.
I will convert this entire review into a list of reviews.
So I will write review dot split.
Okay, just to create a list of reviews okay.
And then finally for each and every for each and every words, like, let's say if I probably just try
to, uh, execute this review okay.
How will I get.
So print review okay.
Okay.
It's okay.
We'll be able to see it.
Okay.
So what I have done is that I have substituted it, I have made it lower.
I have done a split with respect to each and every reviews.
Inside the sentence.
I'll get a list of words over here.
Now, after I get a list of words, what I'll do, I will apply the same concept that we did before,
right?
Uh, in, uh, lemmatization and all same process.
So over here, you'll be able to see that, uh, what kind of process?
Uh, with respect to this things here.
Same process.
I will try to apply the same combination.
Whatever I did.
If it is not present in Stopwords, then only you try to stem it, right?
Like how I did it in text pre-processing.
Same thing.
I will be doing it over here with respect to bag of words.
So I will say for word in review that basically means review has a list of words with respect to a specific
sentence.
I'm iterating through every word, if not word in.
If that word is not present in stop words dot words of English if it is not present in English.
Because this is what I showed you with respect to theoretical intuition.
Right?
I'm going to take that specific word and I'm going to basically apply PS dot stem.
Very simple okay.
Nothing so complicated.
Whatever.
I have done the same thing.
I've explained you with respect to this okay.
And this I will be taking it entirely into my review variable again because I here I'm basically applying
stopwords and stemming.
I will get all my review variable over here.
And finally you'll be able to see that I'll be writing review.
And then I'll be joining that all these words into a sentence again.
Okay.
And later on what I will do instead of printing it, know the list that I have actually made.
I will just add it over there.
So I will say corpus dot append and I will append all this review.
Okay, so once I probably execute it you'll be able to see it will take some time.
But at the end of the day we are going to clean it.
We are going to make sure that all the lowering of the words will happen.
And finally, we'll be able to get the entire corpus, because after this only we will be able to apply
the bag of words.
Now this is perfect.
It is completely done.
Now if I probably go and see the corpus now here you can see everything is basically done right.
And obviously you know the disadvantages of stemming.
Some of the words will not come right.
But uh, maximum number of words will come right.
If you want to make it more accurate, you can definitely apply Lemmatization.
And if you also want to make sure that you apply snowball stemmer, you can also do that.
That will be a task that I really want to give it to you as an assignment.
Okay, you can try both of them.
But again, understand with Lemmatization it will take more time.
Okay, so this is my entire text, uh, my input data.
Okay.
Input data with respect to all the things.
And uh, let's see what will happen in the next step that we are going to basically apply bag of words.
Now let's go ahead and create the bag of words.
As you know I already know what is my output variable?
Okay, my output variable is nothing but the spam and ham the label column, right?
And this is specifically my input features.
I need to convert this into vectors.
So now we are going to create the bag of words model.
Now this is super important guys how the bag of words model is created.
So just go and search for sklearn bag of words okay.
And uh here uh with respect to sklearn, you have a library which is called as countvectorizer.
Now, this countvectorizer I'm going to specifically use for the bag of words.
And here we are going to discuss about some of the important keywords.
Okay.
Then only we'll be able to understand okay.
There is, there are so many keywords that we will try to solve it.
Okay.
And uh, with respect to that only we'll try to, uh, solve more of the problem statement as we go
ahead.
Okay.
So, uh, let's go ahead and create it.
So first of all, I will just import countvectorizer.
So I'll say from sklearn dot feature extraction dot text we are going to import Countvectorizer.
Okay.
So here what I'm actually going to do is that I'm just going to write something like this from sklearn
dot feature extraction.
Dot text import count vectorizer.
Okay.
So and then I will initialize to count Vectorizer.
Now you need to understand some of the things with respect to the parameters okay.
Now here you can see that if I probably zoom in there is also something called as lowercase is equal
to true.
See we did not have to do the lowercase manually, right?
You can also use this particular library to and make sure that you just put lowercase is equal to true.
It will also do for you okay.
And they are also stopwords.
You can also apply stopwords.
But we have already done that, so I'm not going to focus on that.
This n gram range I will talk about it uh, in the upcoming sessions.
Right now.
Let's go ahead and focus more on this particular feature, which is called as Max underscore feature.
Now in this I have already told you see based on frequency, right.
We can pick up the top ten features, top 20 features, top 30 features.
Right.
And you know that whenever we try to solve any problem with respect to sentiment analysis and all,
we obviously get some kind of vocabulary size.
But when I see right how frequent those vocabulary is basically present.
Right.
And it is not necessary that every, every, every one will be available for 100 times 2 or 200 times,
there will be some of the words that will be just present for once and twice.
And sometimes those words will not play a very important role.
Okay.
So what we basically do is that we can set some max features.
Okay.
I'll just give you one idea that how we can set a max features, uh, with respect to this.
But other than that ng gram n gram underscore range will discuss it later on.
Not right now, but I really want to show you an example with respect to max underscore features by
default.
Other things we have already done it so we don't have to really worry about it okay.
And one by one will try to complete it.
And if you are not able to understand it, uh, don't worry, everything will be covered as we go step
by step.
So here you'll be seeing that Max features.
Let's say I'm saying that out of all these words, out of all these words, right from the vocabulary,
take the top 2500 words, which has having the maximum frequency.
Okay, so this is what I'm actually creating.
I'm creating a bag of words which is saying that take the top 500, top 2500 words from this particular
data set, which is having the maximum number of frequency.
So once I execute it.
So this is my CV okay.
Now after this all I will do is that I will just say CV dot fit underscore transform and I will apply
it into my entire data set like corpus.
Right.
The corpus is basically having my entire data set.
And I will try to convert this into an array and let me save this in my x variable.
Now if I probably go and see my x values.
So here you'll be able to see that I'm getting so many different different x values, and some or the
other way you'll be able to find ones and zeros.
Now see, I've taken the Max 2500 words.
Right.
And because of that you will be able to see this.
So in short, if I probably see x dot shape I've taken that okay.
Take 2500 max number of features.
So obviously the number of vocabulary size, the number of columns that I'm going to have is 2500.
Let's say if I say take the max 100 features, and I just try to do the transformation with respect
to the on the X itself.
So if you remember this, I'm just going to execute this on the corpus.
And now if I go and see this will be my shape with the next 100 vocabulary size.
Right.
And if I again go and see my x.
So here also you'll be able to see some where ones, ones will come and remaining all will be zero,
ones will come and remaining all will zero.
Once will come remaining all will be zero.
There will be multiple ones wherever the specific word is present.
Now you need to understand one more thing.
Over here.
I'm getting two.
So this basically becomes that this is not binary bag of words.
This is normal bag of words.
Now how do I make sure that I also enable this binary bag of words.
So for that you just go back to sklearn and there will be a feature which is called as binary.
Let's see where it is.
Uh, over here we are not able to find out okay.
Here it is binary.
Okay.
So what I'm actually going to do, I'm going to just go over here along with this Max feature.
I'm just going to write a comma and I'm going to write binary is equal to true right now.
Once I do this right now, you'll be able to see that you're having twos right after I apply.
This binary is equal to true.
No, you will not be able to find out any values that will be greater than one because even though it
is greater than one, it is always going to set it as one.
Okay?
And once I execute it now here you'll be able to see either I'll be getting ones and zeros.
Okay, so all this specific thing has got updated.
You'll have only ones and zeros.
Now it is up to you which you want to basically use.
And again it depends to problem to problem statement whether you want to go with binary bag of words
or not okay.
So I'm just going to write it over here.
For binary bag of words enable binary is equal to true.
Okay.
So this is done.
Very good.
Uh, we are in the right state and we are able to convert this into this.
And here I'm just going to execute it.
Let me keep binary is equal to true.
Otherwise I don't want you all to miss this.
So done.
And this is what we are able to get.
Now understand the array shape.
Since I've taken the top max 100 features which are having the maximum frequency, the column size will
be 100, right?
And this 5572 is basically the length of the entire list.
Okay, so here we have got a complete idea about how to basically convert this into bag of words.
You can play with max features, you can play with binary, you can play with other things.
But the most important thing is that you really need to understand what is n g underscore gram range,
which I will show you in the next video.
First, we'll understand the theoretical intuition behind it and then we will try to also implement
it.
But right now I feel that yes we have done a bag of words easily.
We have converted this word into entire vectors, which is a very, very good thing.
Very amazing thing because now we know at least how to convert a text into the vectors.
Now what I can take, I can take all these arrays as my independent features and all this particular
output like label ham and spam and will try to solve the problems.
Okay, so yes, uh, this was it from my side.
I hope you like this particular video.
Please make sure that you keep on practicing with different, different data sets.
But one assignment that I really want to give you over here is that please make sure that you try to
do it with the help of, uh, lemmatization.
Also, uh, instead of doing stemming, just try to do lemmatization because you may get a better accuracy
with respect to that also.
Okay.
So that will be an assignment.
So yes, uh, this was it from my side.
And yes, I will see you all in the next video.
Thank.