Hello guys.
So we're going to continue our discussion with respect to NLP.
In our previous videos we have discussed many topics till now.
You know like stemming Lemmatization we have seen Stopwords.
We have also done with the help of NLTK and Python.
Now let us just go ahead and revise once more.
Okay, I'm not revising each and every topic, but if you really want to solve a specific problem statement.
So let's consider that over here we have a problem statement called as sentiment analysis.
Now in order to solve the sentiment analysis you know what all things basically we do and how this topics
we have learned till now, where does it fit in the life cycle of an NLP project?
Now let's say that this is my text and this is my output okay.
And when I write d1, d2, d3, d4 these are basically documents one, document two.
Or it can also be called a sentence one sentence to sentence three.
Sentence four.
If I combine all these particular sentences, it becomes something called as corpus.
Right?
And uh, if I probably say corpus, another meaning can be paragraph also.
Right.
And over here we will be also able to see different different words.
We'll also be able to see unique number of words which is called as vocabulary perfect.
Now initially whenever you are given a problem statement that is with respect to NLP, you definitely
will be having a data set.
So let me just create a small, uh, block diagram over here.
So let's say this is the problem statement I really want to solve that is called as sentiment analysis
just for an example.
So initially what I will be having I'll be having a specific data set.
Now with respect to this particular data set.
The first step that I will definitely do is something called as text pre-processing.
Right.
So over here you will be able to see something called as text pre-processing.
So let's say this is my text pre-processing okay.
And I would like to give this as part one because here specifically if I talk with respect to this particular
text, pre-processing techniques usually write what all things we have learnt till now.
So first topic uh is nothing but something called as tokenization.
Right.
And tokenization is a process where we convert a paragraph into sentences or a sentences into words,
and many more things.
The second simple thing that we basically do is something called as lowering the use case, right?
Lowering the lowering the case of the words right now when I say case of the words, that basically
means or I can just say that, right?
I'm just lowering all the words itself.
Right.
So here let me just write this one.
I can just write something called as lowercase of the words.
Okay.
Why we require this?
Because understand lowercase the words.
Let's say I have a capital though, and small though, right?
Both the words are actually same, right?
Though can be present in any other sentences.
And we really need we really need to treat this as a single word itself.
Right.
So it is important after we perform tokenization, we really need to lower all the specific words.
Okay.
So that is the reason what we do now with respect to text pre-processing part one, uh, we usually
do this two simple step.
And along with that we can also apply our third thing.
It is with respect to regular expression.
And definitely I will try to show you, uh, why regular expression can be super important in this particular
case.
Now over here you'll be able to see regular expression can be, you know, removing the special characters.
It can be removing, uh, you know, any, any characters from that particular word or the entire sentence
based on some regular expression.
Right.
So that is what we basically do.
And again, this is a cleaning process.
When we say text pre-processing we basically say cleaning process.
The next step again, uh, when we further go and probably we have discussed about all these things,
only regular expression is left, which I will probably discuss, uh, as we go ahead.
Okay, now in the second step, what I'm actually going to do, I'm basically going to write something
called as Text pre-processing part two.
Right.
Text pre-processing.
And this is part two.
And what all things we have learnt here till now.
So some of the topics like stemming right.
Lemmatization Lemmatization and I hope.
What is the advantage of stemming Lemmatization we have already discussed.
Right.
And third one, uh, we basically have something called as Stopwords, which is perfectly going on right
now.
All these things are absolutely fine, right?
We have we are doing all this things stemming, lemmatization and all here.
Also, we are focusing on cleaning the text and our raw text once it becomes very much clean, we are
definitely, uh, the next step will basically be to take this particular test text and try to convert
this into a vectors.
You know, vectors is a numerical representation of a specific text.
It can be a sentence, it can be words.
Right?
We try to represent each and every word with some kind of vectors, you know, which will give some
meaningful representation of that specific word, so that we will be able to apply any kind of machine
learning algorithm like classification, you know, to solve any kind of use cases like sentiment analysis.
So we have actually covered till here the next step what will happen is that.
So here I'm just going to write out all the steps.
So this is my step one.
Step two right.
Step three.
Now when I talk about step four now this is super important right.
And uh I'll just give it as another color.
So this will basically be my step four.
After completing this step you know I will take this entire text, you know, and probably we will try
to convert this text into vectors.
So what we are going to do, we're going to convert this text into something called as vectors.
Vectors is nothing but a numerical representation of the specific text.
What are the techniques that is used that is super important.
And you really need to understand this techniques because as we go ahead right later, later on when
you start learning deep learning, also the concepts like word embedding, word two vec and all right,
and all these techniques that is basically used.
And even the advanced techniques like Transformer and Bert, they also use this technique of converting
the text into vectors.
And they have some amazing way.
One way is word embeddings, right?
That is, they have an amazing way to convert this text into vectors which will provide meaningful semantic
information to the text, right?
So just understand for right now, if you're not understanding what is text to vectors, it is just
like I have some kind of text.
Let's say I have this food is good.
This entire text will be represented by a simple numerical format okay.
Numerical data in short.
Now over here we are going to learn various different techniques.
Now first thing is that we are going to see what is called as one hot encoded okay one hot encode.
And then probably if you know machine learning you will definitely be able to know it.
And again one hot encoded is not a very efficient technique with respect to this kind of data.
And uh, right now nobody is using one hot encoding techniques specifically for text data.
Yes.
For machine learning problem statements there to convert a categorical features uh into numerical format.
We basically use one hot encoded but not in text.
But we'll try to understand the theoretical part of this.
And after this we will be going with something called as bag of words.
Okay.
Super important technique which we also say it as B or W right.
Very very super important technique with respect to this.
Uh, third technique uh, that we are specifically going to see, uh, over here is something called
as tf IDF.
TF IDF okay, so tf IDF is also a very good technique, uh, to convert the text into vectors.
Uh, and as we go ahead and learn different, different things, right, there will be some disadvantage
in that specific technique.
So we are using the next technique.
The fourth one that we are probably going to see is something called as word two vec.
Now this word two vec is also an amazing technique.
Uh, there's a concept in deep learning which is called as word embedding.
Word two vec is basically used.
You can also train your own word two vec.
You can also train a custom word two vec, or you can also use the same weights to train the newer data.
I know if it is not making sense, don't worry, I will explain each and everything as we go ahead.
Now the fifth part, uh, that we are going to basically see something called as average word two vec.
Again, a super important topic altogether.
And I will also try to see all these things.
Now why I'm writing all these things, because these are all the techniques that we are going to see,
uh, from the next video itself.
First we'll go with one hot encoded will understand the advantage and disadvantage.
Then we'll go and see bag of words.
We'll see that how bag of words work.
And then we'll see the practical implementation like how we can do it with the help of NLTK.
Right now after you do this, after we get the vector representation of a text and numerical information
about that specific text or numerical representation of the text, we take that numerical representation.
Let's say that this this entire sentence.
Right.
I'll give you an example how it will get converted.
Let's say, um, I'm just writing it as 1011.
Right.
Something.
Okay.
Some some numerical representation okay.
So this will be my numerical representation for this particular document.
That is D one.
D1.
Uh, how I'm getting this.
Don't worry, I've just put some as an example.
I will explain you as we go ahead with the video with respect to this numerical representation.
This will be my output.
So what we can do is that we can take this vectors and we can send it to the next stage wherein I will
just write it out as fifth.
And this will basically be my model trained a model getting trained.
Okay.
And over here you'll be able to see that once I give convert my text into vectors.
Here we will train with machine learning or DL algorithms.
Okay.
But right now since we are discussing NLP with machine learning I'm just going to write machine learning
algorithms.
Okay.
We are going to train this uh with ML algorithms okay.
And finally we will be able to do the prediction and find out the accuracy.
So this is what step by step we are basically going to do it.
But you really need to understand all these techniques uh all these techniques right.
In machine learning at least one hot encoded and all.
And uh, later on we are also going to see a library which is called as Gensim library, which will
actually help you to do, uh, word two vec because it, it is a huge model.
Okay.
Both to vec.
Uh, it already has many representation of the words.
Like it can provide a numerical representation of various words.
Uh, and again internally, word two vec and average word two vec also uses deep learning techniques.
Uh, again we will discuss about it.
The overall brief idea.
I'll try to give it to you.
And then what we'll do is that we'll solve some, uh, use cases wherein we'll take this particular
data and again, we'll follow this entire step.
And finally we'll train our machine learning algorithm.
Right.
So this is what we are going to do as we go ahead.
Uh, so I hope, uh, you're able to understand what is the flow and how we are going to solve this
particular problem.
The example of the data set is over here.
We'll try to convert this into a format and then train with the machine learning algorithms.
So yes.
Uh, I will see you all in the next video.
I hope you are able to understand things.
I hope you are able to understand each and every thing.
What I'm actually writing.
Uh, yes.
This was it.
Uh, I'll see you all in the next video.
Thank you.