So guys, let us go ahead and start the data collection mechanism over here.
Okay.
Now in this data collection mechanism, first of all, uh, we definitely require some kind of input
data.
Right.
And without input data, it will be very difficult to work.
Right.
So first of all, what I will do, I will just go ahead and write import NLTK and in NLTK, uh, I have
a very important, uh, package.
Right.
So I'll say NLTK dot download.
Okay.
We will be specifically using this called as Gutenberg okay.
And then this Gutenberg we will be seeing there will be one text file, uh, you know, and we'll try
to load that particular text file so quickly.
Uh, I will just go ahead and write from NLTK dot corpus import Gutenberg.
Okay.
So I'm just going to specifically use this Gutenberg library.
Along with this I will just go ahead and import pandas as PD.
So I'm also going to use pandas as PD along with this.
The next thing is that we will go ahead and load the data set.
Now in order to load the data set quickly, I will go ahead and write.
Data is equal to Gutenberg dot raw.
And here we inside this Gutenberg package there is a raw data set which is called as Shakespeare hamlet
dot txt.
Okay, so we will try to pick up this specific data set okay.
Now this particular data set is already present inside this package.
Now we will take this data set and we'll save to a file okay I will go ahead and write with open.
And let me quickly go ahead and write my hamlet dot txt.
I will create a new file.
I'll say hey, open this in my write mode and use a context called as S file.
And I will just go ahead and write file, dot write and whatever data I'm actually getting from that
particular txt file.
I'm just going to write in this okay.
So this is a very simple way of doing the data collection.
Uh, over here in the Gutenberg package there is a txt file which is called as six dash hamlet dot txt.
Okay, so once I execute this you will be able to see there will be one txt file that will get created.
Okay, I'm getting an error.
Let's see what is the error.
Partially initialized module has no attribute internals, mostly likely due to circular import.
Okay.
Uh, let me do one thing.
Let me restart this kernel.
Okay.
Sometime this kind of issues will definitely come.
So I will just go ahead and execute it.
So here you can see NLTK Gutenberg is already up to date okay.
And now here if you see hamlet dot txt okay.
So why I got that particular error.
Because I had before starting the recording of the video.
I had imported it once and I was just trying to do it.
So now, uh, I just restarted the kernel and now I can you can see over here I'm able to get this particular
hamlet dot txt.
Now this is what is the hamlet dot txt.
The Tragedy of Hamlet by William Shakespeare 1599.
And it's a huge data set.
And we'll take this entire data set and we'll train with LSTM RNN.
Okay, one more thing that I really want to go ahead and tell you.
If you do not have a powerful system, please try to execute.
And all this code in Google Colab.
In Google Colab you have free GPUs right now in my system, I have a powerful system workstation, so
I think I will be able to install all these things and work with this in my local machine.
Okay.
Now this is the first step.
Now we will go ahead with data preprocessing okay.
So quickly let's go ahead and do this.
So for data preprocessing I'm going to import numpy as NP okay one of the library.
Along with this what I will say I will go ahead and write from TensorFlow dot Keras dot preprocessing
okay dot text I'm going to import tokenizer.
Now with respect to this uh, I will also go ahead and import from TensorFlow dot keras dot preprocessing
uh dot sequence.
I'm going to go ahead and import pad underscore sequence.
Okay.
See uh, I will talk about why we are using this particular library.
Okay.
Uh, everything will make sense once I'm first of all, let us go ahead and import it.
Okay.
Then I will also go ahead and write from sklearn dot modelselection import traintestsplit.
Okay.
So this is specifically for traintestsplit okay.
Now let me talk about the first one.
Tokenizer is just uh, a library that is present inside Tensorflow.keras, uh, which will actually
help you to convert your text into vectors.
Um, adding sequence specifically makes sure that all the sentence length will be same.
Uh, in the in the while we are training the entire LSTM RNN, and this is specifically for doing your
train test split.
Now I have to apply.
I will go ahead and load my data set and I have to apply this entire tokenizer and padding sequences
specifically in our data set.
Right.
So I will go ahead and write with open.
And here let me just go ahead and write hamlet dot txt.
Here I'm going to basically use my read mode and I'll say hey as file.
And I will just go ahead and write.
Text is equal to file dot read.
And here we are specifically going to use this dot lower.
Okay.
So first of all I'm just lowering all the text uh and probably taking in the specific variable okay.
Uh after this I will go ahead and tokenize the text.
So in order to take a tokenize it I will say tokenizer.
Tokenizer will be my variable.
I will go ahead and initialize this particular to my tokenizer object or tokenizer class.
After this, you'll be able to see that I will quickly go ahead and write tokenizer dot fit underscore
on underscore text.
Okay, so this is an function which we are specifically going to use this.
And this text needs to be given in the form of list okay.
So this is a function that will be specifically used in um you know we just need to use this fit underscore
on tests so that it will apply this entire tokenization on the text that I'm actually giving.
Okay.
Now we will just go ahead and calculate our total words.
So let me just go ahead and write total words is equal to length of tokenizer dot word index.
We will get the word index okay.
So word underscore index.
And since we know that it is starting from zero index.
So I will just go ahead and write plus one okay.
Now if I go ahead and see my total words um, okay.
There is uh, it is saying that hey, uh, no module name TensorFlow, Keras, dot pre-processing.
So here guys, uh, I have just made a minor mistake where I have done the spelling change instead of
writing processing.
I have written this.
Okay, so double s, it should be like how we have written for the top one right now.
Let's go ahead and execute this.
Now, the total number of words that I'm actually getting is 4818.
Okay.
And, uh, you'll be able to see that if I just go ahead and write tokenizer dot word index.
Right.
You'll also be able to see all the word index like for the the number is one.
The index is one then and you have the index is two.
For two you have the index is three of four.
Like this.
For every word you will be able to see the indexes okay.
Now this is what my tokenizer work is specifically doing.
Now let me quickly go ahead and, uh, make sure that we have to try to create some input and output
sequence right now.
How do I actually go ahead and create input and output sequence for this particular data set.
But here you can see with respect to every word I am able to get an index by using this tokenizer.
Okay.
And that is why this tokenizer is specifically used.
Right.
So when we are fitting it uh, over here we are basically creating indexes, creating indexes for words.
Okay.
Perfect.
Till here.
Uh I think everybody is clear right now.
Let's go ahead and let's go ahead and create my input sequence.
So I will go ahead and create my input sequence sequences Okay.
So input sequence is basically means I will have a input text.
And then I should have for that input what will be the next word.
Right.
Something like that.
So I will go ahead and create a list which will say input sequences is equal to this list.
I'll say for line in text dot split, I will split this entire text with respect to a new line.
Okay, for every line.
Okay.
Then I will go ahead and write token.
Underscore list is equal to and I will use the same tokenizer.
And there will be something called a text to sequence.
There is a function which is called as text underscore two sequences okay.
And I will take every line for this okay.
Every line.
And from that particular line, let's take the zeroth index okay.
Inside this particular token list okay.
So let's see you'll be able to understand the output okay.
What will happen.
Just wait for some time.
So I will go ahead and write for I in range okay I will say one comma the length of token underscore
list okay.
Since I'm going to add every tokens inside this okay, I'm just going to write for every line with respect
to all the elements.
I will say n underscore gram underscore sequence is equal to token underscore list with respect to colon
I plus one okay.
You'll understand it.
What exactly we are doing okay.
So here input sequence dot append will be nothing but n gram sequence okay.
Now see what we are exactly doing over here.
We are splitting each and every line okay.
We take uh, that particular line and we apply to this particular tokenizer.
And if you don't know about text to sequence okay.
So this is what it is going to do is that it will just going to convert that text into some sequences,
uh, wherein we'll have some kind of vectors.
Also, I will show you.
Okay.
just just, uh, wait.
Like, finally you'll be able to see the what will be the output.
So let me execute this.
Now, you know, this will basically be my input sequence.
Now if you see the input sequence.
Right, right.
So for the first sentence right, let's say this is my first sentence.
Actus primus.
You know supreme right.
So first of all see I'm able to get this 1687.
Right.
So in short what this is basically doing is that it is converting every words into sentence, line by
line, right line by line.
And we are getting that particular value.
Now see first sentence is basically having two words.
Second sentence is having three words.
Well then you are having this.
Then you're having this.
And this is how it is basically getting divided.
Like all the sentence is basically converted into an input sequence.
Okay.
So let me do one thing.
Let me just create this variable like this input underscore sequence input underscore sequence.
So this is the kind of data pre-processing that we really need to do okay.
Perfect.
Till here.
Uh I hope everybody is fine.
We have just converted our every sentence into sequence of indexes.
Now the one thing that we really need to do is about our pad sequence.
We have to make sure that all the sentences are of equal length.
Right.
So for this I will just go ahead and apply my pad sequences Okay.
Here I'm going to basically use max underscore sequence.
Underscore length is equal to max of.
I will just go ahead and apply this max of max of.
I will use one list and I'll ask length of x and I'll add like for x in input underscore sequence.
Now what I'm actually going to do is that for every uh c when we are saying we are iterating through
each and every sequences for every sentence, and then we are calculating the length, right?
And whatever max length we specifically get right over here.
So if I just go ahead and execute and see this max underscore sequence underscore length, you'll be
able to see that 14.
Right?
14 is the maximum length with respect to all the sentences that we have over here right now.
What I'm actually going to do I'll take uh input sequence, something like this I will apply this input
sequences and I'll say hey go ahead and use this NP dot array.
Uh NP dot array okay.
And I will apply this pad sequences in all the input sequence okay.
all the input sequence with max length is equal to nothing but this max sequence length.
Okay.
The reason why max sequence length I'm basically saying I'm saying hey, make all the sentences to maximum
14 uh, indexes or 14 characters.
And here what kind of padding I will be using.
So padding will be nothing.
But it will be uh something like pre padding or post padding.
It is up to you.
Now if you go ahead and see the input sequence now, every sentences will be of the same length and
the maximum length will be of 14 sentences.
And we have added zero for in the front right.
So that is the reason you'll be able to find this kind of input sequence.
Right.
So here we have specifically done the prediction okay.
Now uh, it's time, uh, that we divide this entire data set into training and test data set.
Now see training and test data set.
How should we specifically divide in my x axis?
Uh, let's I consider for the first word right here, I will probably take just one example.
What I will do is that I will just take, uh, or I'll let me write it over here.
Okay.
So here I will say creation create prediction predictors predictors and label okay.
So here I'm going to write import TensorFlow as tf okay I will just go ahead and write my.
Just a second.
Import TensorFlow tf okay.
Now once I import TensorFlow tf I will go ahead and use my x and y coordinate.
So this x will be my independent features, y will be my dependent feature.
I will go ahead and write input sequences.
Input underscore sequences I will take all the words and just remove the final word okay.
And in the output sequence, which is my y, I will take this input sequences, I will take all the
words and I'll just take the last word okay.
So the last word will specifically go to y.
And the first word will specify first all the words till the last will go to my independent feature
okay.
Now similarly if I go ahead and write y Y, I'll just go ahead and say, hey, tf dot Keras.
Uh, so I will just go ahead and write KF dot keras, dot utils dot two underscore.
Or first of all, let me just go ahead and execute this okay.
So here you'll be able to see this is my x okay.
All the input features.
And when I go ahead and see my y.
So here you'll be able to see this indexes okay.
With respect to this.
So many number of indexes will be there okay.
Now for this why we cannot just keep this into indexes right.
Based on the number of classes some words may be repeated.
Some words may not be repeated.
We'll try to convert this into categories.
So in order to convert this into categories I will say tf dot keras dot utils dot two underscore categorical.
Here I'm going to basically take y comma number underscore Underscore classes is equal to total underscore
words okay.
Now if I go ahead and see my y it will be like completely wherever this particular index will be present
that will be one remaining.
All will be zero.
Wherever this index will be present, that will be one and remaining all will be zero.
Wherever this index is present, that will be one.
Otherwise remaining all will be zero.
So we have actually converted our output feature with respect to all the categories right Now it's time
that we go ahead and split our data set into train and test.
So here you can go and do this I have taken train test split x y test size is 0.2 okay.
So once I execute this now we are going to uh we are ready to probably go ahead and train our LSTM,
RNN right.
Train our LSTM RNN.
rnn.
So, um, uh, this process of LSTM RNN training is going to definitely take time.
Uh, so uh, till here I think we have done the data ingestion and data pre-processing.
We have our input features with respect to the index.
And we have our output features right now.
It's time that we go ahead and train our entire model.
And that will be shown in our next video.
So I hope you like this particular video.
This was it from my side.
I will see you in the next video.
Thank you.