Hello guys.
So we are going to continue our discussion with respect to text summarization.
We have already seen about stuff documents uh stuff document chain summarization.
Then we saw map reduce chain summarization.
And finally we are going to go ahead and explore about refine chain summarization.
Now this is uh uh, you know, just a slight modification, uh, when compared to the map reduce.
Let's say you have this entire documents.
Okay.
Now when we have this entire documents, let me just go ahead and rub this again and write it in a better
way.
So let's say if I have a document over here.
So these are my documents.
Let's say I have divided this documents into chunks, right.
Smaller chunks.
So this is my chunk one.
Chunk two.
Like this I have lot of chunks.
Now what we will do whenever we use this refine chain summarization we will take this first chunk.
We will give it to a prompt template.
Along with this prompt, we will go ahead and pass to the LLM to get our summarization.
So let's say once I get this summarization.
Then this is one specific output that we are going to get.
Okay.
Now for the second chunk what it will happen before sending this to the prompt.
This to the prompt and the LM.
We are also going to take a reference of this summarization result along with this.
Right.
So if I just really want to explain this, uh, over here, it is basically given by refine basically
means updating a rolling summary by iterating over the document in sequence.
Okay.
If I really want to just talk about refine, this basically means for every chunk, whenever we are
passing it to the prompt, and once we get the summarization of that, it is going to combine with the
next chunk.
And then again it is going to send to the prompt and LLM.
We will get our next summarization over here.
Okay.
So let's say that I have got my next summarization here Right?
And then what will happen?
This summary will be later combined with this.
And here we will be able to get our another like we'll be able to pass to the another prompt and another
LLM.
So this is basically getting rolled up right.
So that is the reason when we saw in the documentation it is saying that hey updating a rolling summary,
we are taking the summary and we are rolling it over one by one as we go ahead with respect to the next
chunk.
Right?
So this is my chunk one, chunk two like that.
And finally after covering all the chunks we get the final summary right.
So this is one technique uh, with respect to refine.
And here you could see that how it is a little bit different from MapReduce and the stuff one okay.
So guys now let's go ahead and see the practical implementation of refine chain for summarization.
So for this we don't have to do much.
Just use the same load summarize chain.
And here we are going to use the first parameter as my lm model.
The second parameter will basically be my chain type.
Whenever we use a chain type over here by refine, then we will be able to use it.
The third thing that we are going to use is something called as verbose is equal to true to just see
the all the information.
And finally when I just go ahead and run chain dot run with respect to our final documents, I think
we should be able to get our output summary.
So this will basically be my output summary is equal to this.
And let's print this output summary over here.
So here you can see.
Now see over here also sending the food I could do for him along this this refining is basically happening
right.
See at the end of the day refine summarization basically means it's more about refining the summary
itself.
The first chunk of summary, when we get it, we are sending along with the second chunk and again passing
it to the prompt and LLM model.
And finally, at each and every time, we are specifically doing that particular process to get a much
better form of summary itself.
And that is what we can actually do with the help of load summarization.
Um, so in this all series of video we have seen about all the three types.
One is stuff document chain.
Then we have MapReduce.
Then finally we have about define chain.
We have also seen the theoretical intuition.
Yes this was it from my side.
Uh, in the next video we are going to develop an end to end project implementation for both structured
and unstructured type of content.
Okay.
And that is what we are going to do in the next video.
So yes, I will see you all in the next video.
Thank you.