All right, everyone.
So the very first topic in text analytics.
I would say text processing or text preprocessing, which we are going to learn in this video, is a
tokenization.
So what is tokenization in a very lamington?
It's something like breaking your backs into some kind of PCs.
So let's say I have a sentence like I like happen.
So it's a very simple sentence and simple Englis.
So if I want to tokenized this particular sentence, there will be a plea.
Tolkan, in which it will be tokenized is so first token will be like an eye, second token will be
a light and a token will be happen.
But in English there are always exceptions and things are not that much easy.
Life may not be so simple.
So that is where a different organization algorithm and everything got implemented inside this specially
library.
So what will do?
I hope you didn't hear the full sentence.
And for all four, we are going to play this tokenization.
So let me assign it to let's say Esslin of Unmowed has asked to redeem it.
Yes.
Let me Miki S4.
All right.
So first, let's import that spacey library.
And then we will apply tokenization individually on on four different English sentence.
Let me write.
Import spacy.
And let just executed.
Now, before diving into tokenization, I would like to discuss something like all three.
Martin so Spacey already wonders with this number of leading Martin let me create new code.
And as a part of Spacey installation, that is a one kind of small size model already implemented.
So what did preprint Martin is?
Let me define it here so we can load this briefly model like Speci Dark Lord.
And here you can pass those preprint modelling.
Now, what this Braitling model will do will see.
Let's say the name, I will give it like.
In this court court.
This VEP underscored saying.
And let me assign it to some Martelli and l.P.
All right.
A model got implemented.
Now let's go to the new tab and search for spacy.
Go to the very close link.
And you'll be able to see on the top, maybe, yes.
But there is a link called modern.
All right.
So they have given a very good quick start with the model now specially already implemented.
There are two kinds of model.
So one is a core model and no one is stuck on model.
So mainly we will be dealing with this core, Martin.
And now this model is also available in a number of different languages, naggingly, Liz, France,
Spanish.
So English will be our core language.
If you want to dig down a little bit more, you'll be able to read about this.
What is the difference between this core model?
And there will be a starter, Martin.
How you can use it.
So if you want to load some particular model of some particular language, let's say we want to lorik
for English language.
And there are two ways we can Lohri like imported as a model or you can load it like speci law has.
We have Danny here and there, like us saw uses example.
Also, if you don't want to see that is a some simple code maybe created for you.
All right.
So that is what about the model?
If you want to go to detail about what that model will do?
You can read about it now.
If you click on this English, that is not the only model available like Ian or Vavasor in which we
help you load it.
In this case.
So Hesam here indicates that there is a small size model.
You can see the size of this particular model.
It's just eleven.
And this model Del created.
Based on the blocks, news and comments they acquired from some resources available on Internet that
is English and out of this model, you will be able to gain this post tax, talking to actors, dependency
pass and anemic.
So that is a small model.
They'll train on a small subset of net.
But for our purposes, we are going to work with this small model.
Now, suppose if you want to load a bigger model, you can just simply download that bigger Martin with
this Python Hyppönen Spacy download even called that underscore m m limits.
It's a medium sized model, so they have their own pipeline like manual.
You supply text.
It will do tagging, crossing and name.
And.
And you can see this model is a little higher in terms of size also.
So I look at it as just eleven m.b model.
Now it is a ninety one and be more than and at the same time we have nail gun like us index.
Accuracy also in any other so that we don't learn about that in yet.
That is a Neeme entity recognition and that is a large margin.
So large model little train on a very huge, huge amount of tax like blocks, news and comments on video
taken into consideration.
But that's a very big number.
And this model is literally very big in size.
That is a seven eighty nine megabytes.
And you may see any accuracy or might increase.
So that is about the English model.
And if you want to look to some specific model, you can do.
So let me just copy.
Now, here we are looking at this particular model.
Let us try to look at another model.
So if you want to add Ziggler any Biton says clip comite, I would say inside this clip, you can just
simply use explanation.
And all those subscript come on, you can type it here and it will download this particular.
Model in your local support.
If I tried to do the same thing in somewhat DSL with this Emily Martin and let me know, like an LP
and this one.
So if I tried to execute it.
You'll be able to see we got that who is head of this court web and has got Emily.
It doesn't seem to be psychic link, so that doesn't have one.
So what we can do is we can just simply fire this.
Come on.
And it will download this particular market.
You know, it looked fine.
So it will take little time.
Maybe.
So I'm just fast forwarding my video till this completes highlights almost downloaded.
Requirement audit is that despite.
And you be able to see you can now load the model via Stacy load.
They give given indication so.
So now what we can do, we can take a look at.
We know that a matter which we learn, so that will be of import.
And you can type here.
The model need imported.
So successfully guarding.
So next days, we just need to use this again on this court order and this court, ravenous Cordingly,
dark Lord can.
Now let me assign it to an MP on this one.
All right.
So you can see and in this cold one, the highways are not our medium size model also got loaded.
All right.
So that is all about.
In this video, we'll just see how good look the model, how to install the different models for a pre
train.
Models are available in this PESI library.
Now, the real game starts once we help model how to use this model for a different NLB cost like a
tokenized is in force.
So we'll see that tokenization in the next video.