Hello guys.
So we are going to continue the discussion with respect to natural language processing.
In this video, we are going to cover some of the basic terminologies that is required in NLP.
You really need to understand these terminologies because I am going to repeat this terminologies again
and again when we are discussing the other topics.
So the topics that is going to get covered in this video is about corpus documents, vocabulary words.
You really need to know all these topics, what exactly it is with some basic examples.
Now usually whenever we get a paragraph, a paragraph is usually called as a corpus.
Okay.
With respect to documents, whenever you have any kind of sentences, you really need to understand
that this sentences are also usually called as documents.
What about vocabulary?
Vocabulary is nothing but all the unique words that are present in this paragraph that is basically
called as vocabulary.
Usually we have a dictionary, right?
We usually say that.
What is the vocabulary in that particular dictionary?
All the unique words are the count of all the unique words, or all the unique words that is present
in the dictionary that is called as vocabulary.
And with respect to the words, all the words that are present in a corpus that we will basically define
all those separately as a specific words itself.
So these are the basic terminologies that you really need to understand.
As said in this video we are going to discuss about something called as tokenization.
And tokenization is a very important step whenever we try to solve any kind of use cases with respect
to NLP.
Now what exactly is tokenization?
Right.
So let's say that I have a paragraph I write over here that my name is Krish, my name is Krish.
Okay.
And I have a.
I have a an interest in teaching.
I have an interest in teaching.
Machine learning and NLP and deep learning and DL.
Now, let's say that if I have this specific text, this text I can consider basically as paragraphs.
So this will be entirely corpus.
Okay.
So this is my entire corpus that is available, which is nothing but a paragraph of, uh, words.
Right.
So if I probably combine all these words, it becomes a paragraph.
Now tokenization is a process wherein we take either a paragraph or a sentences, and we convert this
into tokens.
Right.
Now suppose let's say I want to perform a tokenization on this particular paragraph.
And over here from this paragraph the tokens that are usually generated, it will basically be called
as sentences or documents.
So let's say that I will be applying a tokenization on this.
And with respect to this, let's say that they I will try to convert this entire paragraph into a sentence.
So I may also add one more line over here.
Let's say full stop.
I'm just writing one more full stop over here okay.
And I will also write that I am also a YouTuber.
Okay.
So these are the two sentences that is present in this paragraph.
So with respect to this particular tokenization, if I perform a tokenization on this paragraph, it
will basically create sentences.
My first sentence in this particular case will be my name is Krish okay.
And I have interest in.
Interest in teaching.
ML, NLP and NLP and DL.
Okay, so this what this is basically my document one or sentence one.
My next sentence that I'm going to probably write over here because the full stop is over here.
Right.
So when we convert from a paragraph, when we do talk tokenization from a paragraph into sentence,
it will be looking for this kind of characters like full stop or exclamation.
I'll show you practically how this can be actually done with the help of Python programming language.
So the second sentence that I will probably be had having is like, I am also a YouTuber, right?
I am also a YouTuber.
So again, if you really want to understand what exactly is tokenization?
Tokenization is a simple process wherein we are converting a sentences into sorry, where we are converting
a paragraph into sentences.
Now there may also be a scenario that let's say that I have some sentences.
Okay.
And on top of this I can also perform tokenization again.
So let's say on top of this I'm performing a tokenization.
Now this tokenization technique that I am probably applying will convert this sentences into words,
right?
So let's say I say over here it is basically getting converted into words.
So each and every word will be a separate word.
So my will be a separate word.
Name will be a separate word is will be a separate word.
Crush will be a separate word and will be a separate word I will be separate word.
Have interest in teaching.
All this will be a separate words itself.
Right.
So this process is also called as tokenization.
So in short words can also be a token.
Sentences can also be a token.
Right.
This is very important to understand.
And why it is required.
Because this is a part of text preprocessing.
Because each and every word in NLP needs to be converted into a vector.
So we really need to take up each word and try to do this kind of pre-processing.
And there are a lot of steps like cleaning and all, which I will also be showing you.
But in this video we are going trying to understand about tokenization.
So I hope you have got an idea about corpus.
You have got an idea about sentences.
Now let's go ahead and understand about vocabulary, which is also called as unique words.
Okay.
Now let's say I have two sentences I like to eat apple juice.
Sorry.
How can we eat apple juice?
I like to drink apple juice.
Okay.
here I will again continue and I'll write.
My friend likes.
Mango juice.
Okay.
Now let's say that this is my entire paragraph.
Okay.
Now, in this paragraph, you know how many sentences are there?
There are two sentences because there is a full stop over here.
Right.
So I will just divide this into tokens.
So let's say I'm going to perform something called as tokenization over here okay.
And this will get converted into tokens.
And right now the tokens that is present over here will be sentences.
Right.
So my first sentence will be I like to drink apple juice.
So this is my first sentence.
And second sentence is nothing.
But because there is a full stop.
My friend likes mango juice.
Mango juice.
Now see, when we have the sentences, obviously you can you can go and count each and every words,
right?
Let's say how many total number of words are over here.
So if I probably go and count 1234, five, five, six, seven, eight 910.
Right.
So if I again count it 123456789 1011.
So total I have 11 words.
But if I try to count the unique words, how many unique words are there?
If I make the count again so I will be one unique word, like another unique word
123456789.
See, like and likes are two different word.
So I'll say 910.
But this juice is getting repeated, so the total number of unique words will basically be ten words.
Right.
Let's say instead of this likes there was something called as like at that point of time, the number
of unique word, the number of unique word will be 123456789.
Right.
I will not count like and juice right.
But already likes is there.
So it will be counted as a separate word.
So whenever I get this unique word as ten words, that basically means in my dictionary, in my this
complete paragraph, this is my vocabulary.
So this is all the possible words that I have, right?
That is the ten words right now since I have converted this into like so I'm just going to make this
as nine words.
I hope you are able to understand the basic differences between corpus documents, vocabulary, and
words.
Right?
So this entire thing is super important when we are learning about tokenization.
Again, if somebody asks you what is the definition of tokenization, you can just say that tokenization
is a process to convert either a paragraph or a sentences into tokens.
If I convert a paragraph into tokens, that basically means I'm converting a paragraph into sentences.
I can also convert a paragraph into words.
Right.
So if I'm converting into a words, single single words becomes a token.
And if I'm trying to convert a paragraph into a sentence, every sentence will be a token, right?
So I hope you are able to understand this.
Now, in my next video, I'll try to show you, with the help of NLTK library, how you can perform
tokenization with the help of Python.
So yes, this was it.
I will see you all in the next video.
Thank you.