Hello guys.
So we are going to continue our discussion with respect to word two VEC.
Uh, already we know what two vec are basically of two types.
One is cbow which we have already seen previously continuous bag of words and skip gram in this video.
What we are going to do is that we are going to understand that how word two vec model is basically
created.
What is the deep learning model that we are specifically saying?
You know how the inputs and outputs are there and how the model is basically trained.
One important thing is that you really need to have a prerequisite knowledge about an loss function
and optimizers.
So if you do not have this, I would suggest.
First of all, please make sure that you have some knowledge about this right before you understand
this.
Now with respect to word two vec, one more important thing is that in word two vec we also have a pre-trained
models.
Right now if I talk about a pre-trained model like Google, right?
Google has a pre-trained model with respect to word two vec, which is trained on three 3 billion words.
And we can also train a model from scratch.
Train a model from scratch okay.
And again, uh, the reason why I'm taking this, because you really need to understand that how that
feature representation is basically getting created.
Okay.
Now let me go ahead and let me say is that, uh, uh, let me take a simple corpus and let's say that
first of all, we'll start with cbow.
Okay.
So we are going to discuss about cbow, which is nothing but continuous bag of words and how this model
is basically created.
It is a type of word two vec right.
Continuous bag of words.
Now to solve any problem I will definitely have a data set.
So let's say that this is my corpus.
And remember corpus because all these models, you know like word two vec is trained on a huge data
set, huge data set like this particular pre-trained model from Google is basically trained on 3 billion
words.
So let's say I have a corpus or a statement or a data set or or a paragraph.
It can be anything.
And for just making you understand, I'm just going to take a simple paragraph.
I'm going to say that okay I neuron.
I neuron is is I neuron company or I neuron company.
Is related to.
Data science.
Let's say I have this particular corpus.
Now remember like this.
In a use case you will be having a bigger corpus.
It can have millions of words, right?
But let's say that I'm digging just a simple corpus over here.
Just a single liner.
Now you're going to understand that how a cbor word two vec is basically created, and how a model is
basically trained with the help of deep learning.
Now, first thing is that whenever we have a corpus, we really need to know what is our input data
and what is our output data.
Because word two vec altogether is a supervised machine learning, right?
So first of all, what we do is that we select a window size.
And I'll talk about this window size and why it is super important.
Let's say that I'm going to select a window size of five.
Now this window size is super important to basically create your input data and output data okay.
Super super important.
Like what should be your input data and what should be your output data so that you can train your model.
Right now this window size five indicates that how many words I need to select initially.
So let's say I am selecting five words.
So here is my five words.
Now from this particular five words I will take the center word.
Center word.
Now understand how I will take up this window size five words and convert into input and output data.
So let's say this is my input data and this is my output data okay.
Now the central element that I've actually taken over here is is right now in the input I will be having
I neuron company.
And then on the right hand side I have related and I have two okay.
Now why why you may be thinking that I'm taking the forward and the backward, because understand,
if I'm taking this as a central word and this will basically be my output word, okay is okay.
This is is basically my output word.
Now it should be knowing that what all words are in the forward context and what all words are in the
backward context.
So that is the reason we are creating in this particular way.
So that this is output should be knowing about its forward word and this backward word.
Just to get some idea about the context of that specific sentence.
Now this is the first step.
Now when I took the window size is five, the initial five words, I divided my data set into input
and output.
Perfect.
Now the next step is that I will go ahead and I'll move this window by one step and take the next five
words.
So the next five words over here is nothing but.
So here this is my sentence one.
So sorry.
Input one.
And now this will become my input two.
And here I will be having company.
Okay.
Uh and again from all these five words which is my center word.
So this is basically my center word.
Related.
Right.
So related.
So here I'll write company is uh two and data.
So this is my second input.
And over here the output will basically be related.
Right.
Then.
Similarly I will go to the next step and I will push this windows to one step more.
Okay.
So here now I'll be having my third sentence with respect to my input and output.
And the third sentence again which will be central word.
So this will basically be my central word.
So here you have to.
Okay.
And here I'm going to have is related.
Okay.
And then the right hand side.
Data and science.
Now you may be thinking, Krish, should we only take the window size as five?
No.
You can take any value.
And why?
This window size is playing an important role.
I'll just say in some time you can take up any value.
But don't take an even number.
Take an odd number so that I will be getting the central element, which I'm taking as an output will
have the correct number of words in the forward context and in the backward context.
Okay, so is related data science is there right now which is a central word over here.
Two.
Right.
So I'm just going to write it as two.
Now this became my input and output.
Now what I'm actually going to do is that I'm going to train my model with this.
Very simple right.
I'm going to train my model with this.
Now how the training will basically happen.
Now, one very important thing that you need to understand over here, you'll be seeing in your own
company related to all these inputs and outputs.
So I cannot probably send this text directly.
I need to convert this into some vectors initially to send it in as an input to the neural network also.
So for this, what I'm actually going to do, first of all, you know that how many number of words
I have, how many number of words I have in the vocabulary I have I neuron I have company I have is
related to data science.
Right.
So they are around 1234567.
Right.
Seven words are there.
Now if I probably use one hot encoding technique.
Now see this okay.
This is super important in one hot encoding technique.
If I probably consider a neuron let's consider the first sentence over here I have a neuron and then
I have company.
Then I have related.
And then I have two.
Right?
So for all these words how I'll be giving the one hot code.
A one hot encoding representation wherever there will be a neuron.
I'm just going to make it as one remaining.
All will be zeros.
So there will be around 1234, five six zeros.
Similarly when when companies there I'll make this as one and remaining all will be zeros right.
Similarly related related is in the fourth word.
So I'm going to make it as 001000.
And then two is present after this.
So I'm basically going to write it as 0100.
Right.
So this is this is pretty much clear till here.
Right.
So here you can see that I have, I have basically done this a simple one hot encoded format.
That basically means if I really want to pass I neuron I need to give this as my vector.
This is what is the understanding.
And this vector is basically given by seven dimensions.
So seven vectors I am giving it over here right 10000.
If I'm sending company as my next word then this should be the vector.
It should go right then.
If I'm sending related this should be the vector I should go.
So similarly all these particular words will be converted into this particular vector using one hot
encoding.
Now let's go to the next step which is super super important okay.
Super super important.
What does Cbow basically mean?
Continuous bag of words okay, this is nothing, but this is a fully connected neural network.
Now you will be able to understand how these models are created.
Fully connected neural network okay.
Now in this fully connected neural network you will be able to see one very, very important thing.
One is first of all just understand how many number of inputs should I be giving, right?
Like how many words I will be giving as my input, right?
Since my window size is same, window size is five.
All my inputs are fixed.
I hope that is very much clear right now.
In this particular problem statement you'll be seeing, I'm giving four words in every sentence, so
my input is basically fixed.
Now I neuron when I give my word, I neuron let's say in the first case I give my sentence one and this
is my sentence one.
When I give I neuron I neuron is represented by seven vectors.
Over here it is represented by this vector.
Then company is basically represented by this vector.
So if I probably see in fully connected layer my first input layer will basically be nothing.
But.
So here you will be able to see this will be my input okay.
And this is super important guys.
See this.
So this will be my input.
My first input word okay.
And understand I'm giving I'm creating this circle as my inputs okay.
I'm just creating this circle as my input.
So if you see 12345677 inputs I'm giving it right now.
When I give this seven inputs then similarly how many words will be going?
Four words will be going.
Right.
So one word two word.
So this is basically my input layer.
So this layer is nothing but my input layer in a fully connected layer a simple an if I probably consider
an example of an.
So here also how many uh inputs I will be having.
1234567.
Right.
And this is my first first word, second word, third word, fourth word.
So I will be having four different words over here that will be going.
And each word will be having a dimension of seven vectors 123, four, five, six, seven.
Right.
I'm not giving this value.
Don't consider that these all are zeros okay?
I'm just saying that these are my input layer.
Input layer input circles okay.
That is how we create it in an right.
And then probably I have my last one.
So this is my input, right?
I'm basically designing the neural network, how it will look like when we are training a word two vec.
So these are my four four words.
Understand.
In the first case I'm going to pass a neuron over here.
So let's say I'm going to pass over here a neuron.
This will be my input over here this is represented a neuron will be represented by 100000.
Right.
And similarly if I go to the second word that is like company then it will be represented by another
word like this.
Like it will be represented by a different vector like 0100000.
Right.
Seven it will be 1234567 right now.
Similarly other words will be represented like this okay.
So this becomes my input layer okay.
And every input is basically given by a vector of seven seven dimensions.
So because I am representing every word based on the vocabulary size using one hot encoding.
Now this becomes my input layer.
Now let's go to the middle layer that is called as the hidden layer.
Now in this hidden layer this is super important.
Just pause the video and guess what will be the size?
You know the our window size is how much our window size is basically five, right?
So I'm just going to make this as my window size.
Okay.
So this is my window size.
Now in my window size if you remember how many, how many we are having in our window size.
Our window size is nothing but five.
So I will be having 12345.
Right.
Window size is basically five.
So in my hidden layer I'll be having this five vectors okay.
So just understand that with respect to this five our window size will be set over here okay.
Now with respect to the output in output how many values I have, I just have one value and each word
I just have one word in the output.
Right.
And each word is represented by a vector of seven.
Because if I'm also considering this is using one hot encoding, I'm going to get this vectors of this
dimension that is seven.
Right.
So what I will be doing in output, I will basically be having another output layer like this which
will again be having seven different outputs 1234567.
Okay.
Now this is how my fully connected neural net will look like.
Neural network will look like.
Now you need to understand one thing over here.
Each and every node, each and every node will be connected to the other node.
Right.
Like this.
It will be connected.
Like.
Like how an an will work.
It will be connected like this only, right?
Similarly right now similarly this all will be connected to this also.
So in short, I can basically make a very simple connection like this which will look like this itself.
And this will be entirely connected to this.
Right.
Understand all these lines will have some initialized weight.
initialized weights.
And we need to train these weights.
And this is what it happens in an write.
Similarly this will be connected to this.
This will be connected to this.
This will be connected to this right.
And finally this will also be connected to this.
Right.
So everything is basically getting connected.
And from the hidden layer this is my hidden layer one HL one and this is my output layer right now from
this hidden layer it will basically get connected over here.
And this will get connected over here.
Okay.
Now understand one very very important thing okay.
This is super, super important.
Fine.
We are connecting it with the help of loss function.
Will will also do forward and backward propagation.
Now let's say we will consider let's let's pass this particular word AI neuron company related to.
So I have passed AI neuron company.
And here also you'll be able to see I'm passing related to.
Very simple okay.
Let me just zoom out a bit okay.
Now once I pass all these things what happens over here with respect to this seven output, I already
know what is my real output is right.
I'll be getting some values over here.
Okay.
I'll be getting some values.
Okay.
But the real output is what if I consider is is is my third word.
So is is my real output.
So this will basically be represented in this vector format that is 0010000.
Right.
But after training while we are training the model with different different weights, this is my true
output.
This is my Y.
I may also get different y hat right?
I may get some values like 0.25.
I may get some values like 0.33.
Then like this 01000 something like this.
Then what we do, we basically calculate the loss function.
And based on this loss we need to reduce this.
We do the backward propagation right backward propagation.
And we do it unless and until the difference between y and and y hat are minimal.
Okay.
And this process is continuous.
Very simple.
But now you really need to understand one very important thing.
Very very important thing.
Now since this is giving me a specific output okay.
This is basically giving me a specific output.
When I say my my middle layer is basically window size of five Window size of five.
That basically means over here in the word two vec.
When I said I will be getting a 300 dimensions over here when I'm using Google Word two vec.
This is all because of this window size okay.
That basically means if my window size is five, I'm going to get the output as five for every word.
That basically means when a word is getting converted into a vector.
I am going to get a size of five vectors and this will basically be my final output.
Now I hope you are able to understand again.
Let me repeat it.
The reason I have actually selected window size is equal to five, because I want to probably provide
a feature representation with a vector size of five.
Okay.
That basically means every word will be converted into a five vector.
Now, when I took an example of Google, which was getting converted into 300 dimension, that basically
means my window size is 300 and more.
The bigger window size, the better the model can basically perform.
Okay, so in this case you will you'll be able to see that over here.
My window size is five.
That basically means if I see from starting right, my metrics for every word will be seven cross five.
That many number of weights will be there.
Then here also I'll be having seven cross five weights here.
Also, I'll be having seven cross five weights because I'm giving seven different vectors here.
Also I'm having seven cross five.
But in this case I will basically be having five cross seven.
Now what does five cross seven basically mean?
When this loss gets reduced?
Then my final vector will look something like this.
This all will get connected to this.
Let's say it is getting connected to this one.
It is getting connected to this one.
So once we have this particular connection, let's say our first word over here is what is our first
word.
If you probably see what is our first word with respect to this particular vectors with vocabulary you'll
be seeing I neuron right.
This is my vocabulary.
The first word that is basically getting represented over here is I neuron right.
So I neuron will have a output dimension of five vectors because I'm getting this five vectors over
here joined to this.
So this five vectors will be like 0.92.
It can be 0.94.
Based on the training it can be 0.25, it can be 0.36 and it can be 0.45.
And this is based on some feature representation.
So I hope you are able to understand.
And this is how a neuron will be represented.
The second word that we have in the vocabulary that is company, right.
This will again get connected to this company.
This will also get connected.
This will also get and this entire word will be the vector for the company itself and for the company.
We may have a different vector, but again the size will be five dimension because our window size is
five.
And this training of forward and the backward propagation, when the loss is minimal, then only we
will be able to get the vectors and that vector is basically taken, and it will be represented in the
uh format of the feature representation for each and every word.
So I hope you are able to understand like how word two vec actually works with respect to CV or Cbow.
Right?
That is continuous bag of words.
If you don't know about and then it will be quite difficult.
But again understand the loss function, it will keep on reducing.
And finally, whatever vectors you are getting at the last that will be combined and that will be taken
as the first word, then second word and third word like that.
Okay, so yes, this was it about Cbow.
Uh, in the next video, I'll be talking about Skip gram.
Right.
Uh, and then we'll talk about the advantages and disadvantages of this particular word two vec.
So yes, uh, I will see you all in the next video.
Uh, this was it.
Thank you.
Bye bye.