Hello guys.
So we are going to continue the discussion with respect to natural language processing.
In this video we are going to discuss about the second architecture that is Skipgram.
Now already I have actually shown you how does Cbow actually works continuous bag of words.
And I also showed you that how the neural network gets trained right now.
What is the difference between Cbow and skip gram?
What is the difference between the architecture?
It is very simple guys.
Right now just focus.
I'm going to take the same data set over here.
Let's say that Enron company is related to data science.
I've written over here.
Right.
And with respect to this, you can see that I've created my input and output.
Now if I am using skip gram, then the thing that is going to change is that everything will be same.
Let's say that I have taken the window size as this.
So if I probably go ahead and show you now, what will happen is that before the input was this specific
test and the output was this specific test.
Now with the help of skip gram over here, the the initially let's say that if this was input now this
is going to become the input.
And this will basically be the output.
Right.
And this is with respect to window size is equal to five right.
Window size is equal to five.
All the steps will be same.
Only what we are doing is that we are changing the input and we are changing the output.
Before the input was this all text and the output was this.
Now what will happen is that entirely when we are creating this neural network now in the input let's
say that I have this is word related word two word right.
So in the input I'm going to basically have a input layer with seven vectors that will be going.
Why seven?
Because if we probably see this our vectors, how many number of, uh vocabulary?
What is the vocabulary size?
1234567.
Right.
So initially in the input layer I will be giving a input which will be basically having uh seven vectors
seven dimension vectors.
And then in the middle I will basically be having my window size vectors.
So window size is nothing but 512345.
Just understand this.
These are just nodes okay.
And in the output you can see that I'm having four words right.
So everything that is present with respect to this right.
Similarly this will get constructed over here at this point of time.
So over here in the output layer you will be able to see that I will be having one word, two word,
three word and the fourth word.
So this is what is the kind of output we will be getting.
And again here it is very much simple because every one will be like seven dimension over here again.
Right?
So I hope you are able to understand we have just changed the direction with respect to this.
Right.
So this will be my input layer.
So here you can see that we will basically be having a seven cross five matrix with respect to the weights.
Because weights initially be randomly and initialized.
Right.
And then we need to train this right.
And then we will be having with respect to this all, I'll be connecting here to here here to here.
And this will basically be a five cross seven uh matrix with respect to weights.
And then this will be five cross seven.
Similarly over here you will be seeing that this will be five cross seven.
And below one will also be a five cross seven.
But in with respect to that, you can see that initially when I give the input is, is will be uh,
the Is vectors will be going over here.
And as you know, that is is the third, uh, third word.
So this will be going like this.
Right.
So 0010000.
So this is the vectors that will be going over here.
And again since my window size is five over here you'll be able to see other vectors that will get initialized
uh randomly.
This all will be connected with weights.
So seven cross five uh weight matrix will be created.
And with respect to this our forward propagation will happen.
And obviously you know that if you probably know and write what all things happens in between the hidden
layer, the input weights is getting multiplied by the weights itself, and then a bias is added and
the activation function is applied on top of it.
In the output layer, we basically apply a softmax function softmax function so that we compute it with
y and y hat y hat is the predicted one, Y is the real data.
In this particular case, this is my Y.
Sorry, this is my Y right?
In the first case y is I neuron.
So I neuron whatever.
Um whatever things will be there, whatever vectors will be there here you will be initializing it.
Right.
So here I basically apply a softmax function let's say with respect to I neuron my y is nothing but
100000007 zeros.
And then y hat will be computed since we apply a softmax over here.
Right.
So y hat will be something.
Right.
Some values over here.
And then what we do, we calculate our loss function.
And we make sure that we keep on doing the forward and the backward propagation unless and until all
the loss function decreases right.
The loss value decreases.
And finally you will be seeing that whatever is connected right.
This particular word will be shown in the form of five vectors.
This word will be shown in the form of five vectors once the loss is completely minimized.
So same process.
You should definitely know how the Ann actually works, how the optimizer actually work.
And this was just a brief idea about script gram right now.
How can we improve this?
Or the the basic question is that when should we apply c by C bo.
So the question is when should we apply cbow or skip gram?
Right.
The simple thing is that according to the research, right.
Whenever you have a small data set, small corpus, we can definitely go with something like Cbow.
That is continuous bag of word.
If you have a huge data set, you should definitely go with Skip gram.
And that is proven, uh, in many research paper.
So I'm just giving you the direct, um, observation out of it so that you will be able to do this.
Now, let's say if you want to increase Cbow or skip gram, how can you basically do it?
One thing is that you should increase your training data set.
Increase the training data.
That basically means the more the training data, the better the accuracy, right?
Increase the training data.
The second thing is that you can also increase increase the window size.
Window size, which in turn which in turn leads to leads to increase of dimensions.
Increase of vector dimension.
This is super important.
Okay so here I'm saying how to improve Cbow or skip gram.
So this is what you can basically do.
Increase the window size okay.
This is super important.
Increase the window size I'm again going to take this separately or forget it out.
So over here again uh, let me write this.
We have to increase.
We can also increase the window size instead of having five.
I can make this as, uh, 100, you know, so obviously with the increase in the window size, increase
the window size.
If we are increasing the window size, that basically means the vector dimension is also increasing,
right?
The vector dimension is also increasing.
So when we keep on increasing and trying uh, try it, you'll be able to see that we'll be getting better
performance.
So this is also you can basically use you can you can increase it.
Now when we see in the next example right we'll be we'll be using a pre-trained model with respect to
Google.
So Google Word two vec right now this is basically trained in 3 billion words I guess it is 3 billion
words 3 billion words.
And it is going to give me a feature representation, feature representation of 300 vectors, sorry,
300 dimensions.
That basically means suppose if I have a word cricket.
Okay, since cricket is always there in the news, this 3 billion word is from the Google News, right?
And Google is a very big company guys with them.
This amount of data it is very much easy.
So what they are going to do is that if we give a word called as cricket, then it is going to basically
convert this into a 300 dimension.
300 dimension vectors.
Okay.
Vectors, this is super important.
300 dimension vectors okay.
I'm just going to write it as 300 dimension vectors.
And this all example I will try to show you, uh, when we use this, what we'll do is that in the upcoming
session we'll try to we'll try to use a pre-trained model also, and we'll try to also make sure that
we train a new data set from scratch with the help of word two vec.
And that we are going to basically do with the Gensim library.
Okay.
So yes, uh, this we will be doing in the next video.
So I hope you have understood both the architecture.
One is Cbow and one is the skip gram.
Only.
The thing that you really need to know is that you need to have the knowledge of an N or a fully connected
layer neural network.
If you know that, it will be very easy to understand.
So yes, this was it from my side.
I will see you all in the next video where we discuss about how to implement word two vec.
Okay, so yes, I will see you all in the next video.