Hello guys.
So we are going to continue the discussion with respect to Lang Chin.
Uh, in this video, we are going to discuss about some of the very important components of Lang Chin,
uh, that we use while generating a gen AI application.
Already in our previous video, we have seen the entire ecosystem of Lang Chin, and we had also done
the setup of Lang Smith.
Uh, and, uh, before I go ahead and do the coding, you know, and start using this library, uh,
we will just try to understand what are the most common components that we will be specifically using.
And in this video, I'll be taking an example of a Rag application that is nothing but retrieval augmented
generation.
Now what does this Rag actually means?
Is that and we are just going to create a lot of projects as we go ahead.
But let me just give you a brief idea about Rag.
What exactly is a Rag application.
Let's say uh, and by just seeing this particular diagram, you'll be able to understand okay.
Now drag augmented generation basically means let's say you have a specific data source okay.
Now this data source may be some kind of PDF file.
Right.
Let's say you have thousands and thousands of PDF files.
Now if you want to make this PDF file as a document Q&A application?
That basically means whenever I ask a question automatically, my gen AI application.
Okay, my let's say if I am probably creating a gen AI app over here.
Okay, so this is my gen app.
What I will do is that if a user asks any query, it should be able to probably send this query to this
entire data source.
And from this PDF we should be able to get the response.
Okay, so this in short is the entire process of Rag.
But again I've just given you a 10,000 height overview right in 10,000ft height overview.
So that is the reason I just told that, hey, we have some kind of dataset where we have thousands
and thousands of PDF.
And now what I'm doing is that I'm creating a generative AI application, uh, this generative AI application,
what task it performs is that whenever a input is given from a user or regarding any question from this
particular PDF, I should be able to get it by having a conversation with this entire data source,
right?
That is nothing but tons of PDF that you have.
Okay, this is a just a basic example, but we'll discuss more about it now if I just consider this
application okay.
And most of the common components that we use in engine to develop this kind of application and similar
kind of applications itself.
So we'll try to understand step by step how we will probably be developing.
And then along with that, what all components are used in each and every step.
We'll try to understand it okay.
So initially what we do is that whenever I have this kind of use case, first of all we need to load
the data set right now.
Loading the data set will be from various data sources.
Okay.
Now this data source can be anything.
So and that is the reason we have used this diagram.
It can be a PDF.
It can be a Excel file.
It can be a JSON file.
It can be an image.
It can be videos.
It can be URLs.
Right.
So initially whenever we develop this kind of application the first step that we specifically use is
data ingestion.
So this is the most important component in launching okay.
And in the upcoming video.
In the next video I will try to show you, with the help of Lang Chain, how you will be able to perform
data ingestion with respect to various data sources.
Okay, so that is the first component that we are going to discuss.
Then what we do is that after taking after after basically using this entire data, like let's say we
have ingested this data, we have loaded this particular data, then the next step will be that we will
divide this data will split this data into smaller chunks.
Okay.
So here what we are doing, we will take this entire data and we will split into text chunks.
Okay.
It can be text chunks.
It can be document chunks.
The reason is very much simple why we are basically doing this.
Because see, in the later stages we will be specifically using LM models.
Or it can be multi model.
Right.
And one of the very important uh property of this particular LM model is that it has some limitation
limitation with respect to context.
So for different different LM models that we will be seeing like OpenAI or Google Gemini Pro or any
different kind of models that are even available in hugging face open source models like llama three.
They have some limitation with respect to the context size okay.
So based on this particular context size, what we will be doing is that there is a restriction the
size like uh, if my LM model I can only give this much maximum context size of data so that it will
be able to understand from where it needs to pick the query or response, and it needs to give it as
an output.
Right.
So that is the reason what we do is that we divide this particular data.
Right.
And this data can be having 1000 PDFs.
If you read all the all the data from a thousand PDFs, it can become a huge data itself.
So we take that data and we'll split that into text chunks okay.
So that is the second step that you will be seeing.
And splitting this chunks is done by various other techniques.
So here I will just consider this as a data transformation technique.
So this is nothing, but this is my data transformation here.
My main aim is to transform the data or divide this data into chunks.
Okay.
So this is the second important, uh, component that we will be doing.
And again, based on our upcoming videos we will discuss each and every components.
Okay.
Now coming to the third and in the important step which is called as embed.
Okay.
Embed is nothing, but it is just embedding okay.
So here we are basically going to do text embedding.
Now what is text embedding basically mean.
We will take this entire text and we will convert into vectors okay.
Now why do we require vectors.
Because see whenever we query this kind of data source you know unless and until we don't convert this
text into vectors over there, algorithms such as similarity search because see, one of the most important,
uh, algorithm that gets applied when we are working with this kind of text data is nothing but similarity
search okay.
Similarity search.
And it will basically be using a cosine search over here.
It will try to understand based on the vectors which are the similar context, which are the similar
sentence that I'm actually searching for.
And based on that, it will retrieve those particular context.
Okay.
So that is the reason we basically use embedding techniques.
Now in this embedding techniques you have different different kinds of embedding.
OpenAI has OpenAI embeddings.
You also have open source embeddings for different different model.
Even in hugging face you'll be seeing a lot of open source model.
It.
All of them specifically has its own embedding techniques.
Okay, even Google Gemini Pro has its own embedding techniques.
So we will again be seeing that whenever we are developing that particular application.
So this was the the third step now coming to the fourth step.
So here you can see I will take this text I'll convert this into vectors.
The vectors will look something like this okay.
Now this vectors.
Once we convert it we really need to store it somewhere.
Right.
Because we can't just keep it in our local file.
Right.
So for this we will be using a store.
And this store is nothing.
But this is basically called as vector store database.
Okay.
Vector store DB we will be seeing different different vector store db.
First example is nothing but we will go ahead and see fires.
Then there is something called as chroma DB.
In our course we'll also be seeing Astra DB.
Okay so different different vector databases.
We'll be seeing how we can basically save this entire data into this kind of vector database that we
are going to see.
Okay.
And that is what this is exactly a vector database.
And what you can also do is that from this vector database you can also directly query anything that
you want.
But the result or the response that you are basically going to get will be the context information.
Okay.
It will be the context information, and it usually uses the similarity search.
Okay.
So this is my fourth step.
That usually happens now once I save this entirely into the vector db vector store db okay.
Now we go to the second most important step okay.
And this is my second um second you can just consider that this is my second module.
This is my first module.
And the first module you have this particular four steps okay.
And in the second module you have some more steps okay.
Now while we are working with vector store DB okay.
In the second module you will be seeing that a user will be able to ask any questions okay.
And what we do we also design our separate prompt.
So here is your user question.
Along with this we combine with this particular prompt.
This prompt usually says that hey you are an LM model.
You need to behave something like this.
I'll say, hey, uh, act as a AI, AI researcher and try to answer all the questions that is given
by the user.
Okay, so if I give this kind of prompt, any LM model that we are going to use, it will act like an
AI researcher.
Okay, so here prompt is just like the message that I am giving to my AI assistant.
Assistant okay.
AI assistant like what it needs to behave like.
Okay.
So once I give that particular prompt template uh prompt, then any question that is given over here,
we will try to combine with this particular prompt okay.
So as soon as we give this particular question, okay, this question initially will be searched from
this vector store.
Okay.
From this vector store.
Then from that particular vector store we will get some context information.
So for doing that purpose to basically query from the vector store we really need to understand two
important things.
One is there is something called as document store flow chain.
Okay.
This is one thing which we will be learning about okay.
This is also one very important component.
And the second thing is something called as the retrieval chain.
Or I'll not just say two, I can just consider it as one because document stuff load chain will be part
of that.
Okay.
So here we need to understand about something called as the retrieval chain.
So this retrieval chain how it is basically created.
And what is this retrieval chain.
This retrieval chain is nothing but it is an interface which is responsible in querying the vector store
DB vector store db.
Okay.
And based on this particular query, I will be able to get the response.
And once I get the response through this retrieval chain, I will basically be having my context info.
Context info basically means based on the query, where is the similar data in that particular PDF,
that entire context will be picked up, right?
And along with this context info, we will be combining this prompt template and finally giving to the
LM model to get my output response.
Output response.
So this in short is the entire cycle over here.
Okay.
Now what will happen in the next video?
First of all we'll see what are the different different data ingestion technique in action.
You'll be amazed to see like what all different different data ingestion techniques are there.
Right.
The second thing will that I'm actually going to show you if I take up any data, how will I be able
to divide that into chunks.
This is also amazing.
Like multiple properties are specifically there.
Then the third thing that I'm actually going to show you, which all embedding techniques we can specifically
use, right.
The embedding techniques like Firefox, Chrome, adb, Astra DB will be seeing multiple embedding techniques
okay.
Okay, sorry, not face chroma db.
This is vector store db.
But we will try to see.
With respect to OpenAI, I have a different embedding technique with respect to open source models.
I have a different embedding technique.
So we will try to see that embedding techniques which whose responsibility will be for converting your
text data into vectors.
And finally, you'll also be seeing that we'll also discuss about vector store DB.
Right.
And then we'll combine all these things and we will be creating some amazing projects.
We'll also be learning about retrieval chain.
So step by step we will break down each of these things.
Because these are most of the common components that are used in every projects of JNI.
Right.
So I hope you like this particular video.
These are some of the very important components of language in which you should never miss it out.
And definitely you should know about these things.
So yes, this was it for my side.
I will see you all in the next video.
Thank you.