All right, everyone.
So let's continue, it is the sun on tokenization, and for that, we have already created DSL in its
quest to an escort.
We are loading our model also.
So now we are gonna work with this NLB, Martin.
So once the model got created, we are going to supply this stream as soon as as an escort.
One by one to our model.
And we will tell you tokenized this particular sentence into different options.
So photic.
We are going to apply this and it'll be Pam Hess one.
So eventually we going to return as.
OK.
Now, what just happened at this particular stage?
You can see how apply this Esslin string to penalty Morton.
So all those intelligence has been site inside this NLB model.
Now, here we are dealing with a very small size model only in core Merab underscored assembly.
And it will return a better representation of it all.
Input text like after tokenization, bagging Horsh, tagging named entity recognition, standing and
everything it will do.
And create one another like you object for us.
Highlight.
So let me add Ranney.
And now the let me create a few more so we have to be true to every single book on Inside This Dark
One.
So let me play you a token in, let's say knock one three token.
Now, before that, let me painting one whole region S1 puzzle.
All right, so now you can see it has created a tokenized was one of our alternate inputs statement.
So now it's okay, like Apple is looking at mine.
Now you can see even if after you there is a dot, but it doesn't zoom back.
This is the end of the token.
Instead of that, it has taken into consideration this United Kingdom view dot here like has one entity
and like on one token, startup for dollar is another entity, one is another entity billion.
And we have an explanation.
So that is the beauty behind this particular.
Organization implementation inside this space.
Now let's go for that next one.
So what I'm going to do, I am this going to bring everything you know of one cell for as to.
So let me just make it Haspel and here, let us make it got to to.
Now, what is this as to and what is the challenges out there?
Inside this has to so if you observe here.
This you might be this senior.
There is no issue that will be a very simple tokenization based on the whitespace character.
But then there is something special, like an e-mail, like on a website.
So how special will this particular as one entity?
Or it will be like a support supporters.
Different had is different.
Name is different.
And then Quong is different.
Let's see how it goes.
So that will be for hopes.
It has to be down to.
All right, so now you can see it has taken in to consider send a full e-mail as of one single token.
And that is the beauty of this spacy tokenization support that too many dot com completely as a one
single clock.
At the same time, the whole you are hearing me dot com.
Also on the token.
So whenever you are dealing with such a kind of e-mail address or you are dealing with a.
Website address spacy, consider it as one single token.
And that is what the intelligence decided, say spicy hockey.
Let's go for the third one and let's see what is the turban.
So in case of third one, there are some Suffolk's out there and there are some graphics out here.
So let's see whether it has been considered as a one single token because it is like a joint.
There is no space in between.
And here also.
It's like a joint.
There is no space in between.
So let's see how spacy will treat after tokenization.
So let me just make it Dockley and.
All right.
Let's run it.
And you'll be able to see Tanny's is different.
KM Kiam is different.
And here as our dollar is different, 20 is different.
And that is what the real beauty behind this spacy tokenization let's do for the last one.
Let's watch a movie together so that his father has four.
So let me make it OK for.
Yes.
All right.
And you can see.
Let's watch is not pretty of one single token instead of the light is different here.
This apostrophe S is a different hand.
After that, everything remains seem like a watch o movie together and not quite right.
So that is all about the different kinds of sentence.
We have taken into consideration for tokenization and this organizes and is kind of very much advanced
tokenization got implemented.
So if you go to the documentation of this bessey or you can just simply say is for.
Go.
Good night, Yishan.
And some Koreans out there, and that is a very good tutorials.
All right.
So that is a tokenization.
I want to show you.
So let's say in this particular statement, let's go to and why.
And God, why God?
Exclamation mark.
So that is not me.
Has a human already know that.
And, you know, so there is a do this stuff.
So it's not legal just based on a whitespace character.
It will take all those designs for the tokenization.
Now, here in this case, let's.
And then go so very far shot because then like this and a second shot, it has a.
Then this suffix.
Then again, some suffix got removed.
Then there is an exception.
So you can see at the end light is different, as is different for strophic is going to there is no
issue for understanding.
Now this and why it has not taken into consideration different one.
But based on those mortar and mortars, intelligence is not.
Why not consider as a one single token?
Because we already know as a human that is referring to New York.
But this explanation, Mark, obviously it's a different.
All right.
So that is the beauty of this spacy tokenization and very advanced organization already implemented
here.
All right.
See you in the next video.
We'll see some more stuff related political spacy tokenization in the next video.