Hello guys.
So we are going to continue the discussion with respect to simple RNN architecture.
And in this video we are going to discuss about forward propagation in simple RNN.
Right.
Already in our previous video we just saw about the entire architecture.
How does a basic RNN looks like?
I hope you have understood this.
And what is this unfolding technique?
Okay, now let's take this example.
The food is good.
So we will try to solve this problem with the help of simple RNN.
And we'll understand how does the forward propagation actually happens, uh, with the help of RNN itself.
So let's consider that here I have all these words.
And uh, with respect to this particular words, first of all, I'm just going to go ahead and find
out how many unique words are there.
Okay.
So let's say the number of unique words.
Are there food okay.
Good.
Um and I'll say bad.
And I can also say not okay.
So total number of words are one, two three, four, five okay.
Now let's consider initially I need to convert this words into vectors okay.
Now though there are multiple ways to convert it.
We can also use word two vec.
We can use multiple techniques okay.
But I'll just use the most basic one which is called as one hot encoding okay.
So we will go ahead and use this one hot encoding.
Now one hot encoding.
What it does is that it converts each word into vectors okay.
So let's say if I have this the word okay since I have five words.
So wherever there is available that will become zero and remaining all will become zero.
Right.
So this basically becomes my first word okay.
Wherever that is available.
So that actually becomes one because the number of unique words are five one, two three, four, five.
Right.
So wherever there is present so that actually becomes one remaining all will become zero.
Okay.
Similarly, uh, if I go ahead and consider with respect to food, wherever food is present, that will
become one remaining or will become zero.
Okay, so this actually is my another vector for the food, uh, for the word food for my next word
that I have over here is, uh, let's say good, because I'm not going to consider is because it is
a stop word.
So wherever there is good in the word, This third one will become zero.
And then remaining all will be zeros itself, sorry, third one will become one as a vector and remaining
all will be zeros.
Right?
And finally you have this bad.
Bad basically means let's say this is my fourth word.
So this will become one and remaining all will become zero.
And finally when you have not, this will be one and this will become zero.
So this exactly is the entire vector for my sentence one.
Okay, so for my sentence one this is how my vector looks like.
Now I need to train this entire sequential data with my, uh, neural network that is RNN, and probably
make sure that how to do the prediction will probably see over there how to do this, how to train this
particular neural network for this particular classification problem.
That is nothing but text classification okay.
So now we will go ahead and, uh, consider our, uh, RNN, right.
Now, in this case, uh, as you have seen, the generic architecture of RNN is nothing, but it will
be something like this.
So I will be having my input.
This will be my hidden layer.
This will be my output.
Okay.
So input output.
Along with this I will be having a feedback loop okay.
And this feedback loop is nothing.
But it will be sent to each and every hidden neurons in that particular hidden layer.
All the information, all the output.
Okay, so here first of all you need to understand how many inputs I need to give, right?
Because at a time I will be giving one word right to my neural network for the training purpose.
And every word is represented by this five vectors.
So here I'm just going to consider this 12345.
So this will be my five inputs that I will be giving to my, uh, RNN neural network.
Okay.
Now in my hidden layer, let's consider that I'm just going to use three neurons.
Okay.
So here I will be considering one neuron two neuron and three neuron okay.
And timestamp is equal to one.
I will be passing this word which is nothing but X11X12XX13.
Okay, so I'll be passing this all words right 123.
Like that.
Right.
All the words will be passed.
So here you can see for the first word this will be passed over here as X11.
And I know what is the vector.
It is nothing but 10000.
All right.
This is the vectors that I'm going to pass through this inputs.
So this basically becomes my input layer okay.
Now this will basically get connected to all the hidden neurons right over here over here.
Then you have here then you have here.
So this will basically get connected to all the hidden neurons.
Okay perfect.
Right now this is my hidden layer.
As you all know, this is my hidden layer one.
But with respect to an RNN, we need to have a self loop, right?
What does this self loop basically mean?
That whatever is the output, I will be passing it to itself and all the other hidden neurons.
So let's consider that I will now go ahead and use time stamp T two.
And here I'm going to pass my another word that is nothing but X12.
Okay.
Now in this case when my X12 is passed 345 right x one to X12 is the word that is formed.
And here I have my another neuron that is with one, two and three.
Right.
Sorry.
One.
Two.
Three.
I will just go ahead and use the same hidden neuron over here.
One, two and three.
Right.
This needs to pass over here.
Right?
And when I'm passing this information at t is equal to two.
As you all know, this information that we have computed at t is equal to one.
So here we'll take this input.
Multiply by all the weights and whatever output I get it will be returned back with the help of self.
Uh, feedback loop itself.
Okay.
This, in short, is what is basically happening is that this input will probably go over here.
Sorry.
This output from this will go to the next timestamp node.
Then this will also go over here.
And this will also go over here.
Right.
So this is one right.
And this is what is basically happening right at t is equal to one.
Like when we use this self loop right or feedback loop, what we are basically doing, whatever output
is actually coming up, we are sending it to its all the hidden neurons.
And that is what is basically happening at time stamp is equal to two, right.
And similarly you will be able to see that again.
We will go ahead and pass this information to this.
Uh, we will go ahead and pass this information to this.
And we'll also go ahead and pass this information to this, whatever output I'm actually getting.
And finally, you'll also be seeing that, uh, this will also get passed to my third norm.
And this will also get passed.
The output will also get passed over here.
And here also you'll be able to see it will get passed over here.
So that basically means at timestamp t is equal to two.
This hidden neurons will also have the context of this word.
that is the over here.
This word is nothing but food, right?
So when this food is basically passed in, the neural network in the timestamp is equal to two.
This context, this neural network, this hidden layer will also have the context of the previous word.
And similarly at t is equal to three.
Same thing is going to happen, right?
This output will be passed to the next hidden layer.
Right?
And here I'm just going to give my inputs my next word.
And similarly when the sentence ends this neuron, this entire RNN is basically going to have the context
of all the previous words from timestamp is equal to t is equal to one.
Right now this is the basic basic funda.
I know the diagram looks quite messy over here, but you can just understand in this way that after
calculating whatever, uh, over here, whatever operation is specifically happening with respect to
the forward propagation, in forward propagation, what do we do?
We take this inputs, we multiply it by the weights, and then we add a bias to this hidden layer.
And whatever output is basically coming we are passing it to itself.
And the other hidden neurons at time stamp is equal to two so that they get the context information
right.
And that is how we go ahead and do our forward propagation.
Right.
Now this is how you can easily do each and every step.
So in a generic way, if you also want to probably go ahead and write it, it is very much simple.
So I can go ahead and say, hey, let's see.
This is my five inputs that I am passing.
This is my hidden layer okay.
This is my hidden layer with any number of neurons.
Okay, so right now in this case I've used how many?
Three neurons.
Right.
And finally this is my output.
So this will get passed.
This will get passed.
This will get passed to everyone right.
Similarly this will get passed to everyone.
This will get passed to everyone okay.
Okay.
All the hidden neurons.
And finally, we get the output over here.
It will get connected to this.
But there will be an additional feedback loop that we will be using.
And finally we get our output right.
So input one.
Input two.
Input three.
Input four.
Input five.
This is based on the one hot encoding.
Whatever I am actually getting with respect to the words that I have right over here, whatever vectors
I basically get that I will be passing it over here.
Right?
So if I want to probably see how many weight matrix will be there, how many weights, how many bias
will be there.
Here you can see that I have five inputs.
And here I have three hidden neurons in the hidden layer.
Right.
So this basically becomes five cross three.
So let's go ahead and calculate how many weights will be there.
So here it will become five cross three.
That is nothing but 15 weights right.
Then over here you will be able to see that since I'm passing one cross five and the weights will be
five, cross three right.
One cross five, one cross five and five cross three one cross five.
Why?
Because.
one input with five.
With one row with five inputs.
When we do the dot operation between them, right?
Weights multiplied by the bias.
Then in short, we are going to get 1.31 cross three.
So for every neuron over here I'll get one cross three here.
Also I'll get one cross three here.
Also I'll get one cross three.
So this in total for this entire hidden layer will be three cross three.
So three cross three is nothing but nine weights okay.
Okay.
And then finally you will be also able to see that this will be my three cross one, which will be nothing
but three weights.
Because here also we initialize weights right.
So total number of weights that you will be able to see 15 plus nine, 24, 24 plus three right.
Three weights 27.
But here we are also going to use bias.
So here I will be getting three bias.
And here I'll be getting one bias.
So 27 plus four will be 31.
So 31 number of total trainable parameters will be their.
Trainable parameters will be there okay.
So this is how it specifically happens with respect to the forward propagation okay.
But again we need to understand in forward propagation uh how what all operations specifically happens.
Okay.
So I'm going to take this entire example for the forward propagation.
I'll copy this I'll paste it over here.
Okay.
Now in order to understand forward propagation with time.
So here I'm going to go ahead and write forward propagation.
Forward propagation with time.
So now let's go ahead and discuss about the forward propagation okay.
Let's say that I have my input over here.
That is nothing.
But so let's say uh, my data set is present over here.
The data set is nothing.
But I will go ahead and write.
The food is good.
Okay.
So as I said, uh, and finally, my output that sees over here is one.
Let's consider it.
Okay.
So let's say this is my X11X12X13 and X14.
Okay.
At timestamp t is equal to one I will be giving X11.
This X11 will be in the form of vectors, right?
Let's say that I have converted by one hot encoding.
And I got five characters over here.
And I'm sending this to X11 okay.
Now this initially we need to convert this word into vectors first of all okay.
Now when this passes to the hidden layer.
So this is my hidden layer here some weights will be initialized okay.
So here my first weights will be initialized.
Then uh this input will be multiplied by the weights okay.
Here a bias will be added.
So at every node there will be bias here.
Also bias will be there bias and bias.
Okay.
Based on the number of hidden neurons.
So once we use this x one uh, along with this weights I will be specifically getting my output one
over here.
Oh one okay.
Now this oh one, when it is getting transferred back to the, uh, to its particular hidden nodes.
Right.
Uh, then here also this weights will be assigned in this.
So let's consider this weight as w dash okay.
Then again at timestamp is equal to two.
So we will first of all discuss about this particular operation.
What exactly happens.
So if I go ahead and see or calculate with respect to forward propagation with respect to forward propagation,
please focus on this okay.
Forward.
Propagation.
So with respect to forward propagation here, you'll be able to see my O one will be nothing, but it
will be a function of function.
Of function basically means an activation function, because in every hidden neuron we also apply an
activation function okay.
A function of here we'll go ahead and multiply X11 multiplied by w.
Okay this w.
And then we will go ahead and add a bias okay.
It can be be one.
It can be anything.
I'll just go ahead and write by be one okay.
This f is nothing but it is an activation function.
So f is nothing.
But over here as an activation function.
So this is with respect to time stamp T is equal to one okay.
Not t is equal to two.
What will happen at t is equal to two.
I'll be sending my next word that is X12.
And here I'll be having my words uh weights w dash.
Uh sorry w and here we will go ahead and compute our o two.
Now while we are computing O two, this o one will also get multiplied by the weights w dash because
here also we are having some different weights.
So next time if I really want to go ahead and calculate my O2.
So again here we'll be using a function f.
And here I'll go ahead and write x one to multiplied by w okay x one to multiplied by w.
Or let me write it in white color.
So here I'll be using a function.
Okay.
So first thing first over here you'll be able to see X12 which is my vector will be multiplied by W.
Plus I have to also used oh one multiplied by w dash right.
So oh one multiplied by w dash.
Now along with this I also need to add bias.
So here I will go ahead and add bias.
That is B1 right.
And then we go ahead and apply the activation function Right.
This is very much simple.
Then again if I want to go ahead and compute my O three now o three will be nothing but over here.
Right.
This will be my O three.
Right.
Now in order to calculate O three again I have w over here and this will be my w dash.
This will be also my w dash right.
This weights will not get updated during forward propagation.
All these weights will be getting updated with respect to time during backward propagation.
I'm just showing you in forward propagation what all operation is going to specifically happen.
Right.
So here you'll be able to see that if I go ahead and show you with respect to O three.
Now O three is nothing but my function off.
So O three when I'm sending O three is nothing.
But I want to calculate this.
So I have to use X13 multiplied by w.
Let me quickly go ahead and write X13 multiplied by w plus.
On the left hand side you have o two multiplied by W-O2 multiplied by w dash plus another bias.
Right.
And that is how you basically get o three.
Right?
And finally when you get o four again by doing the same operation o four.
Since this is a classification problem, right.
It is a binary classification problem.
Then here we are going to use.
So let's say that here I go ahead and probably calculate my O for this will forward.
We will use this output.
If it is a binary classification problem we will go ahead and apply something called as softmax.
Softmax is basically not softmax.
Sorry.
We will go ahead and apply sigmoid okay.
So sigmoid activation function for a binary classification over here.
So this will basically give my binary output okay zero and one.
If I have multi class then instead of using sigmoid I may go ahead and use softmax softmax activation
function.
And finally I get my y hat here.
I may get my y hat right.
Once I get my y hat, then what will happen?
Whatever will be my y y hat, we will go ahead and compute the loss, and we will try to reduce this
loss by doing back propagation.
So back propagation I'll be discussing how it happens in my next video.
But forward propagation you saw this.
This is how the entire operation will happen with respect to each and every word.
Write in a specific sentence.
And that is how the forward propagation with time actually happens.
And this function is a kind of activation function for every word we will go ahead and use.
Uh, we'll go ahead and multiply each and everything as we go ahead till the end of the sentence.
Okay.
So, uh, I hope, uh, you got an idea about forward propagation, and that is how it actually happens.
Uh, um, Um, but here also, you got an idea with respect to weights, how many weights, how many
bias will specifically be there?
Uh, initially my this inputs will be one cross five, then five cross three weights are basically getting
applied.
When we do dot operation I'm going to get one cross three.
So one cross three will be with respect to every node there are three nodes.
So it will be three cross three total nine weights.
Since it is getting passed to each and every, uh, each and every hidden node.
So that is the reason I'm basically using three weights.
Uh, and here you will be, uh, sorry, three cross three.
It will become.
Right.
And then finally here you can see again another three cross one is there.
And uh, when we try to multiply this again, three cross one, I will be getting three weights.
And here will be one bias.
And this is how the forward propagation basically happens.
So in short here it took so much time.
because if you understand this architecture, right, uh, trust me, it will be very much easy to understand
all the upcoming architectures like LSTM or GRU and.
All right.
But here, I hope you got an idea.
I broke down completely to make you understand about RNN, right?
So yes, uh, this was it, uh, from my side.
Uh, I hope you like this particular video.
I will see you all in the next video where we'll be discussing about backward propagation.
Thank you.