Hello guys.
So we are going to continue a discussion with respect to text splitting technique.
In our previous video we saw about recursive character text splitter.
We saw all the code over here.
I forgot to talk about this specific points, which I have again mentioned it over here.
The text.
This text splitter that is recursive character text splitter is the recommended one for the generic
text okay.
It is parameterized by a list of characters.
It tries to split on them in order until the chunks are small enough.
Okay, the default list that basically means they use this specific list like slash and slash n or a
new line blank space, right?
They use this characters to probably do the splitting.
Okay, this has the effect of trying to keep all the paragraphs and then sentences and then words together
as long as possible.
Okay, there are also two important things how the text is split by the list of characters, which is
used over here, and then how the chunk size is measured by number of characters.
Okay.
Similarly, in this video we are going to discuss about character text splitter.
Okay.
So this is also another way this this split based on a given character sequence.
Okay.
Right now the default character is slash and slashing over there in the list of character you add slash
and slash and slash in blank.
Okay, here you have slash and slashing.
Similarly here also you have two points.
How is the text is split by single character separator.
How the uh this the single character separator we will be providing.
Okay.
How the chunk size is measured by the number of characters.
So let me go ahead and show you this example also.
So here you can see that I have this particular text loader okay I'm reading this speech dot txt okay.
And again it is in the same folder location that I have actually created over here in the data transformer.
Okay.
So once I go ahead and execute this so here you'll be able to see this is my entire document page okay.
Just by using loader dot load, you'll be able to get the documents which we have seen in our previous
video.
Now in order to import it I will be using from long chain underscore text splitter.
Import character text splitter.
Before we used to import recursive character text splitter okay.
Then we'll go ahead and initialize this character text splitter.
The first parameter that we need to give is a separator.
So here I'm giving new line.
New line as the character as a separator.
You can also give blank if you want okay then you have chunk size I have given chunk size of 100 chunk
of 20.
Okay then we are just splitting this entire documents with the help of this document.
So here you'll be able to see, uh, these are my documents.
The world must be safe for this.
And all this particular documents will be over here.
Okay, now let's focus on this.
It says that he created a chunk of four 6470, which is longer than the specified 100.
Okay.
Why it may have done this.
The reason is very simple because it was not able to find out the separator.
Okay.
So that is the reason.
What it has done is that wherever it was able to find out the separator based on that it is basically
created the chunk size.
Okay.
Now this is very simple okay.
Now coming to the next one again we will try to open the speech dot txt.
We are again using this character text splitter.
By default if you are not giving separator then it will be slash and slash.
And here we are seeing chunk size is equal to 100.
Chunk overlap is equal to 20.
And then we'll use this creator underscore splitter dot create documents of speech.
Then we'll use the text of zero text of one.
So this code I have already written it over here.
That is the reason I'm just showing you so that you can actually go ahead and check it out.
So once I execute it this is my page content.
And here you will be able to see that all the information with respect to this page content is possible
Now again the question comes Krish, which one should we use?
Should we use recursive character text or character text editor because this is the most generic one
okay.
So whenever you have an opportunity, you have a choice from this particular tool you want to select.
Go ahead with character recursive character text editor.
So I hope you like this particular video.
I will see you all in the next video.
And this time we are going to talk about HTML text editor.
Okay, so yes, I will see you all in the next video.
Thank you.