Hello guys.
So let us go ahead and continue the discussion with respect to word embedding already, I gave you an
idea right where word embedding or embedding layer is specifically used.
See word embedding is a technique which is just converting a word into vectors.
And specifically in a neural network we have to use this as a layer, another layer like how we have
a dense layer, right.
Similarly, embedding layer will also be there.
And the main work of embedding layer is that it will be using some kind of word embedding technique,
where it will take an input and it will convert that into vectors.
Okay.
Now when I'm discussing about this word embedding and uh it is also called as feature representation.
Okay.
What exactly it is.
Okay.
We will discuss about it.
Right.
Let's say I have a data set okay.
And in this data set uh, let's say I have text and this is my output.
Now, with respect to this particular text, uh, let's consider that I have some sentence X11X12X13X14.
This output is zero.
Let's say again you have X21X22X23X24.
Uh then you have this output one.
Similarly you have lot of data like this okay.
Available okay.
Now as I told you, whenever I need to give any input for my neural network, the list specifically
simple RNN.
This is how the representation is right with respect to the simple RNN right from here also.
So let's consider that this is my simple RNN.
If I want to give any inputs over here, one thing that you really need to understand is that whatever,
whatever width I of t whatever I give words with respect to timestamp, I need to convert this into
vectors by using some embedding layer.
Okay.
So I'll, I'll try to convert this into some vectors.
Now let me just go ahead and talk about one of the process which is most initially it was most commonly
used, which is called as one hot encoding.
Right.
Now, let's say first of all, if I really want to use some word embedding okay.
It is also called as feature representation.
Understand this word okay.
This I will discuss about what exactly is feature representation.
Very important thing okay.
So initially let's say I will not use word embedding I'll use some other techniques like uh one hot
encoding.
Let's say, uh, we will use some other technique and we will discuss what is the disadvantage of this
particular technique.
And then we will go to word embedding.
So let's say with the help of first technique that we are going to use is one hot encoding.
Now in one hot encoding what happens is that we define a vocabulary size.
Vocabulary size is basically the the total number of words that you have in this vocabulary.
So let's say if I go ahead and say it is 10,000.
Okay, so 10,000 words I have.
Okay.
Now if I consider one hot representation one hot representation, what does this basically mean is that
if I have a word something like this, let's say man, in this sentence I have a word which is called
as man or over here I'll just define man.
So every word will be converted based on this one.
Hot representation.
What does this one hot representation basically say?
This man will basically get converted to a vector of 10,000 dimensions.
So over here you'll be able to see that this this vector that you will be seeing will be having 10,000
words.
And let's say these are like represented somewhere like this and somewhere like this.
I have Dot and here I will be having one.
Then I will be having zero and continuously all zero.
Okay.
So one hot representation basically says that in this vocabulary in whichever index man word is present,
that will be one and remaining all will be zero.
So how we are representing man by one hot representation, we are basically using this vector, a vocabulary
size that many number of vectors will be there.
Every value will be zero only on the index where man is present.
Let's say if this is 5000 index, right this way we will specifically make this value as one.
So man will be shown in.
This vector will be represented using this vector according to one hot representation Okay.
Similarly, let's say there is also another word which is called as boy.
And let's say for boy in this index.
Boy index in 2000 index.
Let's say I will go and write 000 like this up to one, then 000, then like this up to zero.
Right.
So this size will be 10,000.
In the 2000 index I'm just imagining in the 2000 index boy is present.
So that is the.
This word boy is present in this vocabulary.
So this value is one.
Okay.
So this way the entire one hot representation is basically done for a specific word.
And it is converted into a vector.
Right.
Now just imagine you are having a word.
You're converting this every word into 10,000 dimensions.
And you are sending it in the form of vectors, which is having zeros and ones and all, and only at
one place there is one and remaining all are zero.
So this is really a sparse matrix.
This is basically called as a sparse matrix.
And what is the problem with this sparse matrix.
The problem with this sparse matrix is that whenever we try to use this, it leads to something called
as overfitting, because you have just zeros and ones, there is much, no much calculation as such,
right?
So this is really important for you all to understand how these things are basically taking place,
right.
So obviously we cannot use one hot representation.
It is not an efficient technique.
Right.
Because any number of zeros and ones we have based on the vocabulary size, we are making it so much
big, right.
Similarly you can see by so many number of words.
Are there like so many uh.
Uh, this one hot representation, you just have zero, zero and one place.
You just have one.
So not an efficient way to use this.
Right.
And this is what we in order to overcome this disadvantage, we specifically use something called as
word embedding.
Now if you don't know about word two vec I again suggest please go ahead and revise about word two vec.
Word two vec is one type of word to word embedding technique, and this can also be used in the embedding
layer.
Now what does this basically say.
Word embedding.
What does it say.
Okay.
And please remember this word called as feature representation okay.
It creates a feature representation for every word that is available over here in this data set.
Okay.
What does that basically mean?
Let's say in my data set I have some words.
Okay.
Let's say I'll just go ahead and make this like boy.
I have something like girl I have something like King, I have something like Queen.
Then I have something like Apple.
Then I have something like mango.
Okay, let's consider this.
All words are there.
And again over here also my vocabulary sizes.
Let's consider 10,000.
Okay.
Let's say boy is present in 2000.
Index girl is present in 5000.
Index King is present in 6000 index.
The word in this vocabulary, it is present in that index 9000 Queen is present.
Apple you have something like 1000 and mango you may have in 7000.
Okay.
Now let's consider these words.
Now I need to convert these words right.
Let's say these are some of the words that I have picked from the first sentence.
I'm just considering it okay.
Now how every words is basically converted into vectors that we really need to understand with the help
of word embedding.
If you remember word two vec, I I've already shown you how word two vec is basically trained, right?
There we uh, there is two types.
Script gram is there, cbow is there.
We have already seen how forward propagation backward propagation will happen.
Now, in the case of word embedding, or in the case of embedding layer, which is using some kind of
word embedding for all these words, first of all we will go ahead and select some feature representation.
Let's say for this we will go ahead and select some feature representation.
And let's say I'll say hey, in this feature representation I will take each and every words and I will
change this into so many number, or I'll represent every word in some dimensions according to this
feature representation.
So here I will say let's go ahead and do one thing.
Let's take 300 dimension okay.
Let's take 300 dimensions or vector and represent each word in this 300 dimension.
Okay.
Let's represent each word as 300 dimension okay.
It is very much difficult to understand how the inner working of word two vec is, or this word embedding
is because we will not be able to see all the features.
But here, let's go ahead and consider that I am going to represent some features over here.
I'll probably use some feature representation and I'll write some words.
Okay.
That will actually help you to understand.
Let's say that I'm going to consider gender over here one of the feature.
Then I'm going to consider Royal.
Then I am going to consider age.
And then I'm going to consider food.
Okay.
These are some of the features that I've used.
And let's say like this.
They are 300 dimensions.
More, more.
So many words are there till here.
Okay.
Now what word embedding does is that based on this features that we have considered how these features
are basically coming up.
I have already shown you that in word two VEC training.
Please go ahead and see that.
Okay.
We are going to train this separately.
And this is what is used in the embedding layer.
Right now we are just considering this feature representation.
And each word will probably will try to find out the relation between this word and this word, this
word and this word, this word and this word, and will assign some vectors.
Okay.
So this boy word will be given by this 300 dimension features or three 300 dimension vectors.
By providing by seeing the relationship between this particular word.
Again, let me repeat it.
Okay.
So here what we are doing is that if we consider this word boy, this boy will be converted into vectors
in such a way that we'll try to find out the relationship between all this, all this, all this, all
this features.
Right.
And then we'll get one vector value.
Let's say one example.
I'll say boy to gender.
If I write minus one okay I I'll just go ahead and write minus one.
I'm just assigning some value.
Let's say minus one.
Then girl two gender will be plus one because these are opposite right opposite to boys girl.
So that is the reason I have written minus one and plus one.
Now if I see the relationship between Royal and Boy, we don't say hey, it is a royal boy or something,
so there will be hardly some relationship.
So for this, let's say I'll go ahead and write one value 0.01.
Similarly, royal girl girl can be said royal, so I'll just go ahead and give some value more than
boys okay.
Now similarly, if I take King to Royal they'll say hey royal King.
So obviously this value will be higher.
Let's say 0.95 okay.
If I say hey Queen to Royal, yes we obviously say Queen Royal.
Okay.
But if I say king to gender okay.
If I see some relation let's say it is 0.92 queen to gender.
Again, I'll go ahead and write 0.92 and again this will be opposite okay.
Apple to gender.
Is there any relationship with respect to gender.
So it will be 0.0 manga to gender okay I'll just keep one value 0.1 because there is hardly no relationship.
But do we use royal to apple.
Do we say royal apple.
No.
Okay.
So it will be 0.02 let's say.
And for this it will be 0.01.
I'm putting some values over here based on the relationship.
Right.
So like this.
All these values will with this representation with this relation it will be trained and it will be
provided a vector.
How do we train it.
Please go ahead and see my word two vec tutorial.
Um there you will be able to understand it okay.
Word two vec word two vec.
Okay.
It is necessary that you need to understand what to vec okay.
Now this boy is given by this vectors.
This is now important for you to understand right.
So whenever I pass by word over here first of all I will give this word to my embedding layer and my
embedding layer.
If it is using this 300 dimension it is just going to convert this into a vector with respect to 300
dimension with all this relationship vector values, okay, like minus one, 0.0, one and all.
And once we pass this, then only our simple RNN will start getting trained.
Okay.
Now, uh, this is what exactly is all about.
So here there are some important parameters you should remember.
One is obviously the vocabulary size.
So in this case I have taken 10,000.
The second is what is the features dimension that I really need to use.
So feature dimension let's say I'm using over here as 300.
So there is a word two vec technique uh from Google from uh there is also a library called as glove.
Right.
This has specifically 300 dimensions.
Okay.
Then uh this two are the important parameters.
And, uh, you have to remember this because in the next video, what I'm actually going to do is that
I'm going to show you a practical, practical implementation how word embedding will work.
Word embedding layer will work.
And this is what we are going to discuss in the next video.
And uh here I will just take some examples and then convert that into vectors before I, uh, convert
that into a vector with the help of word embedding layer.
Okay.
Then after that we will start our end to end project.
Okay.
So yes, uh, this was it from my side.
I hope you liked this particular video, but I hope you got an idea with respect to word embedding or
what exactly is embedding layer or feature representation?
I will see you all in the next video.
Thank you.