Hello guys.
So we are going to continue a discussion with respect to Text Splitter.
And in this video we are going to discuss about how to split JSON data.
Let's say that if we are getting a chunk of JSON data from, uh, from an API right after that, if
we read that entire JSON data, how do we split that into chunks so that we'll be able to pass it to
our LLM model.
So here, uh, the JSON splitter, this JSON splitter that we are going to use, uh split JSON data
while allowing control over chunk sizes.
It traverses JSON data depth first and builds smaller JSON chunks.
It attempts to keep nested JSON objects whole, but will split them if needed to, uh, keep chunks
between a minimum chunk size and the maximum chunk size.
If the value is not a nested JSON, but rather a very large string, the string will not be split.
If you need a hard cap on the chunk size, consider composing this with a recursive character.
Text character split up.
So what you can basically do, you can also uh, use along with recursive text splitter.
There is an optional pre-processing step to uh split list by first converting them into JSON dictionary
and then splitting them first.
Okay, how the text is splitted based on JSON value, how the chunk size is measured by the number of
characters.
The same thing is basically happening over here.
Now let me just go ahead and show you how we can probably go ahead and implement this.
For this, I'm going to take this specific example as my API.
Okay.
So this is a very huge API right.
So big API with all the information that I'm actually getting I'm saying not huge API, but if I'm hitting
this API I'm able to get so much of information right over here.
Right.
So what we'll do is that here, obviously we cannot read this entire chunk and just use it right.
Instead what I'll do I'll try to, uh, convert this entire API response, um, in chunks, and then
we'll see that how things will work.
Okay.
So, uh, let's quickly go ahead and do this.
So first of all, what I will do I will be using import JSON along with this.
First of all let me go ahead and select the kernel.
And here I'm going to go ahead and import my requests library okay.
Um let's go ahead and call my JSON data.
Now inside the JSON data, we'll be using this request library and I'll be saying hey request dot get
okay.
Uh request dot get.
So once I write request dot get, all I have to do is that I have to give my URL over here.
Right.
So this is a Get request since I'm reading this entire API.
Right.
So I'll go ahead and hit this particular API over here and here.
What I'm actually going to do I'll also convert this into dot JSON.
So once I get this this will basically be my JSON data, right?
So if you want to probably go ahead and see your JSON data, it will be a huge file.
So this is the entire information with respect to the JSON over here.
Right.
And right now here, uh, most of the information is not visible because uh, if you click on scrollable
element then you'll be able to see it.
Now what we will do, we will try to from we will use this from Lang chain text splitter.
We will import something called as recursive character or recursive JSON splitter.
Right.
Because here we know that they are also nested uh elements inside this particular JSON.
Okay.
Like JSON has a value inside that value there is another JSON.
Okay.
So what we will do over here, we will quickly go ahead and write our JSON splitter.
So I will just go ahead and create my JSON underscore splitter.
And here we will be initializing this recursive JSON splitter.
Here we can give a maximum chunk size.
Let's say the maximum chunk size that I want to give is 300 okay.
And uh after this I will go ahead and write JSON.
Underscore chunks is equal to JSON splitter dot split underscore JSON okay.
And here we are going to give our entire JSON Jason underscore data.
Okay.
And this is what is my Jason underscore data, which I have actually got it over here.
Okay.
Once I give this, uh, then, uh, you will be able to see that I will be able to get this Jason chunks.
So now if I go ahead and execute this Jason chunks, you'll be able to see, hey, uh, we are getting
something, but, uh, right now again, you will not be able to see it.
So what we'll do, we'll just go ahead and print the top three chunks.
So I'll write from chunk for chunk in JSON underscore chunk.
Let's take the top three element okay.
Drop three document.
And then we'll go ahead and print the chunk.
So here I will just go ahead and print it.
So this is my first chunk open I in the key value pairs.
Then the second one with respect to paths.
Then third one with respect to path.
Right.
And this is how easily you are able to probably get this entire information.
You may be also having a scenario right now.
If you see this chunk, it is normally in the key value pair.
But let's say I want to get this into documents.
So the splitter can also can also output also output documents.
Okay.
So here I'm going to create my documents.
And let's go ahead and use the splitter again.
Sorry JSON splitter dot here.
Instead of writing this I will use create documents again.
Um in the create documents we will give a parameter which will be my text.
And here I'm just going to give my JSON underscore data okay.
Now once I get to my docs again I can go ahead and display for doc in docs off.
Let's see the top three docs.
I will just go ahead and print the doc.
So here you can see this is my documents, right.
The same information that I am able to get it okay.
Now let's say that I want to also directly get the string content okay.
I don't want in this particular format like page underscore content or something like that.
Right.
So for that also what you can do, you can directly get this particular text just by writing splitter
or JSON splitter dot split underscore text.
Right.
So there is also split underscore text the split underscore JSON.
And here I'm just going to get my JSON data.
Okay.
And if I go ahead and write print let's say text of zero text of zero.
And let's go ahead and print the text of one.
Text of one, you'll be able to display both of them.
Right.
So both this information is basically getting displayed.
So I hope, uh, you got this idea about how you can split with, uh, JSON, uh, with the help of
recursive JSON split data.
Uh, recursive.
Uh, let me just see the library again.
Recursive JSON splitter.
Okay.
And uh, again, these are just the splitting techniques.
Uh, now in the upcoming videos we are going to see about different different embedding techniques.