Hello guys.
So we are going to continue our discussion with respect to Long Chain.
Already we have discussed about data ingestion techniques where we discussed about various document
loaders.
Now in this video we are going to go to the next step and discuss about data transformation.
Now in this data transformation, we will try to see how we will be able to convert this documents into
chunks of text again in long chain.
Uh, we have lot of different, different ways how we can probably go ahead and convert all these documents
into this text chunks.
And why do we do this?
So that, uh, every LM model has its own limitation of the context size?
In order to take care of that, we have to probably divide this particular documents into smaller text
chunks.
Now let's go ahead and let's let me just go ahead and show it to you how we can actually do this.
I will keep my face hidden so that your focus will be completely on the screen.
Okay.
if you remember, in our last class, I had shown you how to read a PDF file.
Text file?
You know, along with that, a web based loader, how to probably read it from a website or safe.
Then you also have Wikipedia.
So many different things we saw.
Right?
Let's take one of the example.
So let's read this particular PDF file in this particular file itself.
Right I will go ahead and select the kernel.
So let's go ahead and select it now once I execute it.
So this these are my docs that you will be able to see from this particular attention dot PDF.
Now what we are going to do is that we are going to take this entire PDF.
And then we are going to, uh, convert this into text.
Okay.
So here first method that we are going to do is that we'll see how to recursively okay.
Or let me just go ahead and make this as a markdown and write the title over here.
How to recursively split text by characters okay.
So we first method is this particular technique.
So here let me just go and do it.
So here uh what we are going to do is that we are going to recursively split, take up all these documents
and split this text by characters itself.
So first of all, before we go ahead, you know, we require a library.
Now what is a library that will be required over here?
I will just go ahead and write it down.
So this is nothing but lang chain dash text dash splitters okay.
So this is the library that we will be requiring in order to use all the different types of text splitting
techniques.
So I will go ahead and uh quickly do pip install.
So here you can see pip install minus our requirement dot txt.
Now this way you can actually open your terminal.
The other way is directly go to this particular option and click on terminal right.
So you will be able to open this right.
I'm using a shortcut which is nothing but this backspace along with control.
So control and back splash.
When you do that you will be able to open this particular terminal okay.
So already my installation is done with respect to the long chain text splitter.
Now I will go ahead and write my code.
Now first what we will do is that we will go ahead and import from long chain underscore text underscore
splitters.
Okay.
And we are going to first of all import which is called as recursive character text splitter.
Now this is responsible in recursively splitting the text by characters.
So here we have already loaded the document okay I have my entire document over here.
Right.
So this is my docs folder.
Okay.
Now what I'm actually going to do is that I'll just go ahead and write text underscore splitter okay.
And here will be my variable.
And we will go ahead and use this recursive character text splitter.
Inside this there are some parameters which you really need to know.
The first parameter is something called as chunk size.
So here you basically define the chunk.
Let's say I go ahead and define.
My chunk is somewhere around 100.
And uh, I will just based on this chunk size or let's make it to 500.
And here you'll be able to see that I will keep my second parameter which is called as chunk overlap.
Okay.
So this will basically be my chunk underscore overlap.
You'll understand each and every thing.
First of all, let me just write the code so that you get to know what I'm exactly I'm doing.
So I here I'm saying, hey, try to split all these things by considering the chunk size of 500.
That basically means when I'm when I'm dividing my documents into smaller chunks of text, each chunk
of text should have a maximum chunk size of 500 characters.
And there can be an overlap of 50 characters.
Okay.
And uh, here, uh, I'll use this two specific parameter for uh, okay.
After this, uh, we will use this text splitter.
And we have a lot of option over here.
So first option is nothing but create documents.
You also have option like uh let's see create documents is there.
Okay.
So let's let's go ahead and use this create documents okay.
Uh so here what I'm actually going to do I'll just go ahead and write create documents.
And along with this particular documents I'm just going to give my docs file.
Okay.
So once I do the split okay I'm actually going to get some values over here.
And this will basically be my final underscore documents okay.
Once I get this over here you will be able to see that I'll go ahead and execute it.
So here it says hey expecting string or byte like object.
Now see there is one problem that you will be seeing over here when we are dividing this entire text
into documents.
Okay.
So here I have used split underscore create underscore documents.
But you can actually see my return type of this particular docs right.
What is the return type of this particular docs.
It is nothing but document.
So if I go ahead and see the type of docs.
So here you'll be able to see that, hey, uh, you'll be able to see, hey, it is a list.
Okay.
If I go ahead and see Docs of Zero here, you'll be able to see it is a kind of document.
Okay.
But the function that we have used is saying create document.
Okay.
So, uh, create documents will be helpful when we use along with the text file, uh, or any other
file.
I'll also show you one of the example with respect to the text file.
Okay.
Instead of using create underscore document, we will just go ahead and directly use split underscore
document.
Because see already we know here I'm going to pass all my documents over here.
And based on this I'm going to split it.
So now if I go ahead and execute it now here is my entire final documents.
Right now this final document is again a list of documents.
So let me do one thing okay.
Let me just go ahead and print.
Okay.
Final document of zero.
And then I will go ahead and print final document of one okay.
So here you will be able to see this is my page content.
Now focus on these words University of Toronto with all this information okay.
And if I go last.
Okay.
So here also you can see University of Toronto right now why we are actually getting this.
Because there is a overlap right.
So this is my first first document.
The first document with 500 characters.
And as I said there will be an overlap of how many characters.
There will be an overlap of 50.
So that is the reason you will be able to see that some of the last, uh, characters in this will be
again getting repeated in the forward.
Right?
So University of Toronto is there then here you are actually getting this information.
You can compare it.
Okay.
So that entire information is basically getting displayed over here.
Right.
So this is one way how you can probably split the documents using uh, this recursive character text
splitter.
This is nothing.
But here you are trying to recursively split all the text by characters.
Okay.
Now this is one way.
Let me do one thing.
Let me, uh, if I if you remember, I had actually created this speech text.
Okay, so I'll say reveal in File Explorer.
I will copy this data, I'll put it in my next folder in Data Transformer.
And let's go ahead and read this also.
So in order to read it uh, you know, how do we read it right here.
Uh, you can probably convert this entire things into, uh, uh, you know, probably a documents or
how you specifically want.
So let's go ahead and read in this way.
So here I'm going to use this here.
Uh, let's go ahead and write loader dot load.
And finally this basically becomes my documents okay.
So this will be my documents.
And here we will go to write loader dot load okay.
Now I will go ahead and write print or let's go ahead and display this particular docs.
Now whenever I do this by default I'll be getting the return type as document okay.
Let's say uh I will try to read this document that is speech dot text in another way.
Okay.
So here I'm going to write with open write open function I hope everybody knows in Python.
And uh here what I'm going to do.
I'm going to basically write my speech dot text okay.
And then we are just going to create an file variable f with some context over here.
And let's say I will go ahead and create a variable which is called a speech.
Speech.
Okay.
And finally we will go ahead and read this entire speech.
Okay.
So once we read this let's uh, you can also basically, um, you know, see, like, what kind of document
this is.
Exactly.
Okay.
And, uh, let's say for here, I'll go ahead and initialize this particular speech variable to something
like blank okay.
Blank right now okay.
Now let's see this particular speech.
You'll be able to see that I'm able to read the document directly.
So this is not giving me a document type.
This is directly giving me the uh text that is present inside this txt file by just reading this.
Now I'm just going to use this text splitter with recursive character text splitter.
Okay.
Now here I'm going to give a smaller chunk size because I know that the elements are very less.
So let's go ahead and give the chunk size of uh 100.
And then I will give the chunk overlap of 20.
Okay.
So these are the two information that I will give.
Then I will take this text.
You know my return type is specifically text from this particular speech variable, right.
Let me remove this right now here what I am actually going to do, I'm going to basically go ahead and
write text splitter dot.
Now if I want to probably see after this particular step, you know that I have to convert all this
text into, uh, vectors and then afterwards store all these vectors into a vector database.
It is a good practice because over here, uh, whenever we work with Lang chain, it makes sure that
hey, it tries to convert everything into a document type.
Because if we try to convert a text into a document type, we will have more features which we can specifically
implement.
Now in order to convert this text splitter, whatever text split we are basically doing.
And if I use this particular function which is called as create documents.
Okay.
Now in this create documents, if I give my speech okay in the form of list okay.
This will basically give you your entire text.
So I will just go ahead and print my text.
Okay.
So here you can see it has got converted into this particular uh document.
Right.
And finally you are able to see this okay.
So text splitter dot uh create documents okay.
Right now the 100 chunk size was there.
I think it is less than 100.
So I'll just go ahead and make it 50.
Let's go ahead and make it ten.
Okay.
So now if I go ahead and execute it let's see.
Let's see this okay.
So I will just go ahead and write print text of zero okay.
So this is my first one that I'm actually getting.
The world must be made safe for democracy okay.
So let's increase this size.
So here you can see the world must be safe for democracy.
Its peace must be planted upon the tested foundation of.
Okay, now, if I go ahead and print my second document text of one, you'll be able to see.
See this.
The world must be safe for democracy.
So here, let me just increase the chunk size.
Then it will be much more better.
So here you can see world must be, um, safe for democracy.
It must be planted upon the tested foundation of a foundation of.
See, it is getting repeated.
This 20 chunk size is basically getting repeated.
And this is my second chunk that you will be able to see, right?
So for all these things, we actually use something called as uh, recursive character text splitter.
I've also shown you if you have a text, if you just have a text, how you can actually convert this
into a document type.
And if I just go ahead and see this type, it is nothing, but it is document type.
So if I go ahead and just go ahead and write type of this, it is nothing, but it is a document type.
Okay.
So uh, this was some of the basic examples of using recursive character text splitter.
Now in my next video, uh, I will be showing you more other splitting technique.
You can also do it by characters.
You can do it by HTML headers and all.
So that is what I'm actually going to discuss in the next video.
Thank you.