Hey, everyone.
So the next project, which we are going to work is a tax and reason, and for that we are going to
use this tense of law, get us and our recurrent neural network D.M. long short term memory.
So what exactly a tax condition project is?
Let me go to some random text on Wikipedia and what we are trying to achieve from this tax demolition
project and how this hellish Deum network or recurrent neural net could help us.
So let's say we go to English, Wikipedia.
Suppose if you give enough amount of tax to the machine, like thousands of thousands of this kind of
Wikipedia are articles directly to this neutral nickel.
And then what we want from the neural network is, if you suppose gives some tax of data like this one.
Then we're expecting that machine or this neural network or recurrent neural network will predict that
what will be possible values or what would be the possible next?
OK.
So in this particular case, it will be.
Suppose.
If I do some other text like this.
So the next token will be sexy.
So in this way, the text generation project work and we want Machine to predict what would be a next
tax holiday.
So let's start with the installation and a startup.
I'm not going to execute any court.
I just want to walk through that, how things got implemented and what other preprocessing steps are.
So that will be a tens of law and thing and request lively, important.
Next important thing is data pre processing.
And for pre processing we are going to use this sex period are defined as input data.
So if we just follow this particular link.
You can see that this is the sex period that the file be and has been taken from this M.I.T. Web site.
So it has a good amount of English grammar literature.
So good amount of combination is available.
And now we are going to generate the sequence out of this particular whole set of texts and those sequence
we will freely to the Alistaire, Nicole.
And those are less D.M. Network.
Eventually, at a prediction time will provide us the information that based on your seed sequence,
what would be the possible next token?
All right.
So in this particular cell, we're just getting those data.
And here we are just displaying a response, not text.
Let's try to split this whole tax rate slicing so line by line.
And this is the very first line we hope displayed.
If you'll go to the original tax.
This is the handshake, a text file represented by Project Gutenberg.
So it is the intricate text file represented right Gutenberg.
Now we'll take on this 253 numbers line.
And from those particular line onwards only, we are going to take all those data.
Now, why this 253?
Because before that, all those.
Publisher related information in everything is situated, the village tax off, or I will say this licensing
agreement they've written, the legal text will start after this.
Two hundred fifty three lines.
And after this, repeatedly lines all those data.
I'm just slicing down and put it into a data medium.
So after that, this data zero will be our first line.
Let me just copy and legislate here.
All right.
So actual text of this sex bill will start at this particular location.
Now, before that, there are already 253 sentence.
If you try to find what is the length of this particular data variable v. how one looks.
Twenty four thousand two hundred in four sentences.
Or I would say lines are.
Let's just join all of them so that I don't whatever data we will get, that will just the remote part
of this one.
So nothing else will.
Then they'll know only this much part.
We have removed those licensing part and some publisher related information.
All right, let's proceed.
Next, we are removing all those punctuation mark.
So just splitting those data, removing punctuation mark.
Trying to remove.
Is there any known alphabetic character study?
So we are just keeping only alphabetical character, Hanna Dean.
We are just lowering it.
So this particular function will take all those full text and clean it up.
So this three steps, it will perform like removing punctuation mark, removing non alphabetical characters
and make every single token has a lowercase.
All right.
So after those cleaning data here we are just trying to display first 50 tokens.
So token has we started from from Pharis created.
And you can see the very first sentence is also from Forrest.
How many tokens card generated out of it?
So that will be it lacks.
Ninety eight thousand one ninety nine.
And how many unique tokens are dead?
So that will be just twenty seven thousand nine.
Fifty six.
That's also I mean, too much, actually, because in daily routine life, whatever the English we use,
that is, we way less than this much amount of vocabularies.
All right.
So now we have our data available in terms of tokens.
So next thing is we need to generate the sequence.
So in this particular looping mechanism, what we are trying to do, we are just taking 50 land sequence.
And putting in two different different values in this line, really.
So headmen will get one like ninety nine thousand ninety one.
That is just because we are just taking into consideration first 200 key words only.
So if you see first 200, give us the last 50 words.
Do not have anything to predict next.
That's why just one ninety nine thousand nine fifty one token has been taken into consideration.
So this is our first line and you can say this is our first sequence.
So that will be a token zero and token.
So from a self.
All right.
These are alignments that will be of a next sequence.
Now, if you consider the very first sequence and a second sequence, the second sequence will be just
sifted by one token.
So here it was starting from I mean, from from tokin, whereas the second particular sequence is starting
from this farer sequence.
If you go at the very end of both of the sequence.
Here, the last one will be lie, tie and self.
So here will be like myself and there will be one more.
So this is what shifted to one place.
So he had also broken one end token talking.
Fifteen you can consider.
So this is a token one.
And this one will be token 51.
All right.
So we have created a sequence next is we need to organize it and we have to apply those sequence to
no mapping.
And for that same like earlier projects we have.
We are going to use this to organize it.
And on top of tokenized, we are going to play all these lines very well.
And then we will get one big objects like sequences, all those sequences out of something like a numbers.
So here every single sequence has been represented by some vector number.
So out of this particular sequence, we are just trying to grab X and Y.
So this particular part will try to find except the last one put it into X very well.
And this particular variable will try to find only the last character of each.
Eventually we are trying to bready.
So this will occupy 50 tokens and this will be just fifty one token.
So eventually from this particular 50 tokens we are trying to predict.
This 51 talking in the next time it will try to select the first token 250.
Those number, another 50 tokens.
And the next will be immediate.
Next token.
Same way like.
We got X zero.
So that will be a kind of numbers.
If you find the shape of this X zero.
So that will be a 50 numbers and Y zero.
So Y zero will be over three zero seven.
So that is what we are trying to predict when we help, given this much number.
Now, all those number has been represented.
Is presentation, not some token?
So that broken.
Name will be given by this one underscoring Sodo is one, and these two seem like if you try to find
three eight zero six from this next story, you will get some number.
Some token.
So there will be a total vocabulary size of thirteen thousand and eight.
But here, the next thing I starting from Gedo, so real vocabulary size, you will get like a thirteen
thousand eight plus one.
So it will be at thirteen thousand nine.
And the unique tokens, you will get like twenty seven thousand ninety six.
Now, whatever the output values are, the V how to convert it into some kind of categorical value.
So based on the vocabulary, size of a neuron should fire.
So let's say we are trying to predict some character.
Let's have from this first 50.
We got some idea.
So let's.
This is the first week.
I'm just assuming and from that I am just trying to predict a tie.
So PI has been represented by a vector of this walkup size.
And whatever the new neuron will fire, we will predict that this particular token we help predicting.
So in case of X, say we have of a P values and sequence lendings, also 50.
So that is all about our preprocessing data.
V how created our input dataset and output does it.
So next thing is we need to build the model.
So building model, we will see the next city.