Hello guys.
So we are going to continue the discussion with respect to NLP.
In our previous video, we have already seen about tokenization.
We have understood about basic terminologies like corpus paragraph.
We have understood vocabulary, we have understood about words.
Now let's go ahead and do some practical things.
You know how much we'll be learning with respect to theory.
So first of all, I'll just go ahead and open my Python three notebook file.
So here, uh, first of all, uh, and and for this right we are going to use libraries like NLTK.
Now NLTK is one amazing library.
Let me just go through this and let me just show you.
NLTK is a leading platform for building Python programs to work with human language data.
So if you really want to work with NLP, things like tokenization, creating converting a sentence into
vectors can be easily done.
With the help of this NLTK libraries.
There is also one more library which we basically say it as spacy.
So if I probably search for spacy spacey NLP.
So here you'll be able to see that again.
These all are completely open source libraries.
You can also use Spacey.
And I will also be showing you with the help of Spacey how you can perform different, different things.
Right.
So Spacey is also there.
NLTK is also there.
One important assignment that I really want to give it to you is that try to find out the differences
between NLTK and Spacey and uh, just let me know probably in the comment section of this particular
video or in the upcoming videos.
Okay, so this one task, I want to really give it to you so that you try and try to understand what
is the difference between these two open source libraries.
Now, to begin with, uh, since uh, we are going to initially start with NLTK.
Now what I'm actually going to do, I'm first of all going to install NLTK.
Now here you can directly install it from here.
Or you can also open the command prompt and directly install NLTK.
Right.
So what you have to do is that just right pip install nltk.
So once you do this automatically the installation will be done.
So what I'm actually going to do I'm going to basically install from here.
And I'm going to show you a tokenization example.
So let me write it down.
Tokenization uh example okay.
Perfect.
Uh so this is the tokenization example.
So here you can see that uh collecting NLTK requirement already.
This is done.
The installation has been done.
A new release or release of Pip is available.
I don't want to update it right now, because this Pip will almost cover each and everything right now.
Right now.
Uh, this is perfect.
Now, what I'm actually going to do, I'm going to show you how we can perform tokenization.
Like, let's say if I have a paragraph, how I can convert it into sentences, and then how I can convert
into words all those things I'll be discussing and multiple ways of tokenization.
Also, I'll be showing you Okay, so let me go ahead and let me make few cells so that I can directly
go ahead and execute it.
Now one thing over here.
Yeah.
Let's go ahead and let's start now over here.
First of all I will try to define my own corpus okay.
Corpus basically means paragraph.
So let's say uh if I really want to create a multi line comments I have to basically use this triple
quotes I will write.
Hello.
Welcome comma.
Okay, I'm just giving some sentences over here so that you'll be able to understand okay.
So hello welcome to fresh nginx.
Tutorials.
Or let's say I'll just write I neurons right or I will write something like this.
Krish Nayak NLP Krishna x NLP tutorials okay and this is my sentence okay and I can continue with my
second line.
So I'll write.
Please do watch the entire course to I'm just writing exclamation.
I'm using different different characters over here to become expert in NLP.
Okay, so this is what I have actually done.
I've I've basically defined a simple corpus, uh, which is just like a paragraph which has around two
sentences.
And I'm just going to use this particular corpus.
So let's go ahead and see this corpus.
Now once I see this corpus here, you can see that even though I just I'm just printing this right.
I can also print this.
Since then, the slash n will go.
Slash n indicates the new line.
Okay, so if I'm probably printing corpus here, you'll be able to see the text how it is basically
visible.
Now with respect to tokenization, the first step that I'm actually going to do is that I'm going to
convert a sentence into paragraphs.
Okay.
So I'm going to basically convert a sentence into paragraphs.
So for this how do I convert it.
And with the help of NLTK it is definitely possible.
So I will write from NLTK dot tokenize okay.
And I'm going to just import.
Okay.
So in NLTK there is uh there is a library which is called as tokenize.
And if I import this cent underscore tokenize.
So what cent underscore tokenize does is that it tries to convert a paragraph into sentences.
So this is the function that we are basically going to use or functionality that is going, we are going
to use which is present inside NLTK dot tokenize.
Okay.
So you can just say that this is a kind of package inside this.
And I will initialize this package in order to convert our sentence into a, uh, sorry, a paragraph
into a sentences.
So here I'm just going to write, send, tokenize and let me just go ahead and give my corpus.
So once I give my corpus here, you can see that it is giving us a list a list of sentence.
So here you can see over here this is my sentence over here.
Hello welcome to Krishna.
NLP tutorial.
So here you can see this is my first sentence.
And this is my second sentence.
And from exclamation also it has divided it because I'm getting three right.
So here you can see.
Hello, welcome to Krishna NLP tutorials.
Full stop.
As soon as it finds full stop it is just going to make a next sentence.
So here, wherever slash n was present, it is using that and wherever exclamation is also present.
It is basically making sure that a new sentence is getting created with respect to that.
And that is what Sent tokenize actually does.
If you really want to find out the definition of this here, you can see that, uh, you can also provide
different different languages, what other languages it supports.
You can just go ahead and have a look onto that.
And uh, if you don't find much documentation out of it.
So what you can do over here, you can just go and search for NLTK, send, tokenize right, send,
tokenize.
So you will be able to find out the documentation page directly.
And you can refer it from here.
Right.
There are different different tokenization.
See send tokenize.
Is there word tokenize is there which we are basically going to discuss right now.
We are going to focus on this okay.
So in short it returns a tokenized copy of text using NLTK.
Recommended sentence tokenizer and current class.
It uses this called as punk send tokenizer.
So along with the full stop, it is making sure that wherever exclamation is basically coming, it is
going to make us a, uh, as a another sentence.
So this is perfect.
Uh, we are able to get this right now.
I will go with the next tokenization.
And before that, what I also want is that if I probably save this in a list of sentences.
So let's say this sentence is also called as documents, which I have already discussed in my previous
class, uh, previous session.
Right.
So if I probably go and see this documents, this is basically my list.
You can also check out with the help of type.
So type documents right.
And if I probably want to iterate through this let's say from send in documents from from for sentences
for sentence and documents, I can also print this sentence Parallely.
Right.
So here I can just define it as sentence.
So this is my first sentence, second sentence and third sentence.
This is perfect.
We were able to do with respect to the sentences.
Now let's go ahead and do it with respect to the word tokenize.
So next tokenization technique.
What I am actually going to do is that my next tokenization is that I can convert a paragraph.
I can convert a paragraph into words.
I can also convert a sentence into words.
Sentence into words.
Okay, perfect.
Now for this what I'm actually going to do.
First of all, let's go ahead and see with respect to paragraph, uh, so already, uh, you know that,
uh, uh, with respect to converting a paragraph into words, I'll be using again another library so
I can write from NLTK dot okay.
Tokenize from NLTK dot tokenize.
Okay.
spelling mistake is there.
So tokenize I'm going to import word.
Let me just write it down.
Word tokenize.
Okay.
So here we can basically use this word tokenize over here.
And let me just execute it over here.
And let me go ahead and write word tokenize.
And here if I directly give us my corpus here, you can see that each and every word has been divided.
Hello world and hello, welcome.
Where you can see all the characters like comma full stop has been treated as a separate character altogether
or separate words altogether.
Right?
So here you can definitely clearly see that each and every word has been splitted with respect to this
only one word that is not having this right.
So if I probably go and see over here.
Hello, welcome to Krishna.
So this kind of word has not been splitted.
This all is being considered as a single word.
But if I consider with respect to full stop, with respect to exclamation, with respect to comma,
it has been considered as a separate word.
So here also you can find it out with respect to this.
Right.
So this is very, very simple with respect to uh converting a paragraph into words.
And why do we do this?
Because each and every word will have a different importance.
And we really need to perform some pre-processing on top of it.
So right now initially when you get it, you know which are the important words, you have to take it,
you have to pre-process, you have to clean it, you know, and uh, that is the reason why we specifically
focus on each and every word.
Okay.
So this was about the word tokenize.
And uh, now with respect to sentences, also, you know that how you can basically do just go over
here and just write paste it over here for sentences in this.
Now here you are printing the sentences right now.
What you can basically do in that is that after you probably get this sentence, after you get the sentence
all together, you can just directly apply word tokenize, right.
So here you can basically write word tokenize on sentences.
So you will be able to print everything over here.
Right.
So hello welcome to this this this this uh please do watch the entire course.
This this is there.
Right.
Perfect.
So here we have seen that how we can basically convert a sentence into words right now, you can also
do one thing, uh, over here is that you can use another one more library.
Let me just talk about one more library.
Probably you have seen that.
So I'm going to just write word punked tokenize.
And in this word punked tokenize.
What we are basically going to do is that if I try to apply this, if I try to initialize this word
punk tokenize, and if I try to provide my corpus, let's say so here you just try to find out the difference.
And one difference that is clearly seen is that this apostrophe s has also got splitted before it was
not getting splitted, right?
C apostrophe s was a single word, but now you can see that it has been splitted.
So that is the reason why we are using this punctuations.
Um, this punctuation is it is basically, uh, making sure that the punctuation will also be treated
as a separate word.
Perfect.
So this is good.
There is also one more technique which is basically called as uh treebank word tokenizer.
And also again, I'll try to tell you the difference what exactly it is with respect to Treebank word
tokenizer.
So, uh, I'll just try to execute it and you try to find out the differences with respect to that.
Let's see how much you'll be able to do.
So I will write from nltk dot tokenize import Treebank tokenizer.
So I will initialize this Treebank tokenizer.
Okay, let's say that I'm initializing this into tokenizer something like this something to one variable.
And then I can basically use a function which is called as tokenizer dot tokenize.
And once I give my corpus here, you'll be able to see that I'm actually able to get it.
Okay, now just see over this.
What is the difference with respect to this?
You will definitely be able to find out some difference when comparing with this specific thing.
Okay, so just let me know what is the difference that you are able to see.
I know there is a very minute difference.
So with the a let me tell you the answer, you can just pause for some time and but you can check it
out.
But let me tell you an answer with the help of tree bank word tokenizer.
Here you can see that full stop will not be treated as a separate word.
It will be included in the previous word itself.
Now here you can see that full stop was a separate word.
Here.
Also, you can see that full stop is a separate word, right?
But with respect to the last word right, full stop will be separate, you know because here you can
see that, right?
If I probably see the sentence right after this, we have a new line.
Right?
And after this our sentence is getting closed for the last full stop.
Only it will be considering as a separate word.
But with respect to this particular full stop, it will be considering as a part of this it will not
be considered as a separate word only.
This is the difference with respect to this.
And again, it can be handy in some of the use cases, but not in all, but at in a generic way, we
basically most of the time use word tokenize or send tokenize.
Right.
So yes, uh, this was it.
With respect to the tokenization example, I hope you liked this particular video.
Please make sure that you practice.
You have to really practice a lot with respect to this.
You can take your own example, try out with different different sentences.
With that you will be becoming better with this specific algorithm.
So yes, I will see you all in the next video.
Thank you.