All right, everyone, so we velho successfully imported this later.
Now time to clean this year later.
So what would the review columns are available.
We are going to apply all those tax pre processing step.
Something like the removal of your stopwatches are making all the words, you know, lowercase letter.
Because even if the words is capital letter all, it's in a small letter.
It just doesn't make sense as far as interpretation of those particular words are concerned.
Order.
We are going to just remove all those penetration, Mark, because that is anyhow not going to add any
value to the positive reviews or the negative reviews.
So let's start with the very first one before that.
Quickly, a few shall in this leaning tax data section.
So far that we are going to use this mainly and a library.
So I'm going to input LDK first.
And I that a library we are going to use for the regular expression.
That will be a hardy.
So let me just as any.
Now, the first pre processing step we are going to do is a.
Stopwork Limor now is an illegal I.B. or soberness with a lot of stoplights.
So what we're going to do first.
Let's download this stopwork from this Analytica library.
And indicate not now.
And we can hear supply stop, pause.
All right.
So download it and unzip into proper location and indicate and escort data.
Let's observe whether.
It is available here or somewhere as let me express it.
I think some other places, the machine, they'll download it.
All right, so let's just leave it for time.
Next, let's import this stopwork module so from an etiquette.
Not quite a push.
We are going to put stoppers.
Let's have a look at the very first one.
Wow.
Load this place Lachman's here.
Looks like it.
This is a stopwatch.
So it needs to be removed.
Let me display the very first one.
Let's say just the previous column will be focusing on.
And out of that, let's concentrate on a very first one.
And this is our first review.
So we are going to apply this stopwork removal mechanism point of forced review.
So before that, let's just remove all those digits, punctuation mark.
And let's just concentrate on keeping just the alphabetic character on and to keep all alphabetical
order.
We are just going to use this, Heidi, a regular expression library.
So for that, we are going to use this party not.
So.
So that we search for some particular pattern and replace with some other string.
So first, which back then we are going to search for.
So the pattern will be something like all the capital later it does it.
And a small later, it was a.
And that you can describe inside this bigger brocades.
Caddick.
A to Z.
And capital, A to Z.
So only those characters which are alphabetical.
That will be included.
Apart from that, all those digits, all those punctuation mark will be removed.
And let me just replace with a blank string.
And we are going to apply this honor review.
So first, we will let us a play on our first review at this time.
And after that, in the next video will be just applying all those rules on every single records.
So let me assign it to somebody.
And we'll be keep applying all those.
One by one.
So review.
Let me display it, if you know.
All right.
So there is one issue here that it is just cap only alphabetic character.
But it all joint just because there were replacement.
I did with a blank string.
He's dead.
Let's just keep it one space.
And let's play review.
All right.
So you now you can see how everything got replaced with a one blank string.
All right.
So next step is we need to convert all those words into lowercase letter.
That will be very simple.
You can just simply use Debu is equal to review, not lower matola you can apply.
It's a very simple string macaroni and ledges of the review.
So you can see from people who are earlier this hell and this DeBlois mode become a lowercase.
All right, so everything becomes lowercase.
Now we are in a position to play this stopwork removal.
So for that, we have to us tokenized.
So for tokenization, we are going to use this review.
Not a simple split can let me assign it to you.
Let me display what is there inside the review now.
So now it has been tokenized and put it in two different different.
Tolkan has a list now from this particular token.
We are going to search for whether this particular were existing or stopwork database or not.
If we had this, we had to remove it.
If it doesn't exist, we we'll keep it.
So first of all, let's just play stop words.
Not worse.
And that will be English, so we can light a very simple looping mechanism and search for all those
words, which is a part of this dataset or not.
So let me keep it like for every word in review.
I hope you search for.
If you are not in this particular Stoppard's dot words in English.
I have to keep.
Because that is not a bottle stopper database.
So what we will do, we are going to create another list.
Let's say New Order, BWI.
We'll see say that will be a process to review.
And I'm just going to.
I'd like to leave you.
All right.
Something goes wrong.
We are at war.
Not a preview.
All right, so what it says that this object has no attribute and you see the list as, I think, appalling.
All right.
And let's just run it.
All right.
Let's just let's play a preview.
And you can see this got remote because it is a stopper.
Now this whole for a look, we can use it like a list comprehension also.
So what we can do everything.
We can put it as a comprehension.
So I'm just going to copy it and put it inside the list.
So word for word in a live you.
If.
This whole condition I can, captain.
And let me sign it to you.
It sent.
All right.
Let's just do the review now, review doesn't contain this word.
That means no stopwork got three more.
So there is one more step required for the preprocessing.
But that will be stemming.
And we'll see those Demming in the next video.