Hello guys.
So in this video we are going to discuss about something called as hybrid search.
Okay.
And uh, this kind of search specifically uh, we use in our Rag applications also.
Right.
So till now if you are following all the modules, uh, the rag that we have specifically created,
you know, let's say that if I have, uh, documents, let's say if this is my documents, then what
we do is that we try to divide this documents into chunk of documents.
Let's say this is d one, this is d two.
This is d three.
Like this we usually divide into multiple chunk of documents.
Right.
And this is d n.
Now after dividing all these documents into chunk of documents then we specifically convert this into
vectors.
right?
So here we are basically converting into embedding vectors.
Then after we convert all this text to vectors, the next step is that we we store all these vectors
in some kind of vector database.
Right.
So let's say one of the vector databases that we have already seen, vector database.
We also say it as something called as vector store DB.
Right now once this entire information is basically stored over here right, it will be stored it in
the form of vectors.
Right.
We will be having some kind of numbers over here itself right now.
When a user queries, whenever a user queries right over here, then what happens?
Usually in this particular scenario this particular query is also converted into vectors.
And then a vector dense search or vector search is basically done.
Vector search is basically done in this particular vector database.
So once we have this particular vectors we do a vector search from this particular vector database.
And here are some of the algorithms like um uh over here one of the algorithm that may uh, you know,
take place over here for basically to match the exact vector search, we can actually use cosine similarity.
Right.
And based on this cosine similarity check, we will be able to get the response right.
And once we get this response we can combine this with our prompt template.
Plus LM model.
And finally we can summarize any information as my final output.
Right.
So in this scenario you will be able to see that the kind of search that we are doing right.
The vector dense search.
This search is basically called as semantic search.
Semantic search.
Semantic search basically means we are trying to find out the similar vectors from this particular vector
database, and based on that, when that similarity search is done, we are able to get the response.
And this kind of rag, most of the Rag application uses this kind of search.
And if I talk about vector databases like face chroma right.
They have this entire features like this.
Right.
But I really want to talk about one more amazing search mechanism, which is called as hybrid search.
Okay, so here I have this hybrid search.
Now inside this hybrid search here we are just not focusing on only vector dense search mechanism.
Right.
Here we specifically combine combine.
multiple search techniques.
Right.
And what kind of techniques we will specifically apply over here.
We will be combining multiple search techniques.
Now specifically if I really want to divide all this particular technique one is the semantic search.
The second one is something called as syntactic search.
Okay.
Now I will talk more about it.
What exactly are these kind of search basically.
So one is semantic search.
One is syntactic search.
Semantic search.
Specifically we say it as a dense vector search.
So here we are specifically trying to search elements that are similar right.
And obviously when we are using this dense vector search we basically are checking out the similar kind
of content.
Right?
If I probably say with respect to syntactic search.
Right.
Or I can also say this as exact search, I can also say this as keyword search.
Right.
Keyword search.
And if I also want to probably provide some more names like exact search and uh, keyword search.
Uh, and this usually happens with respect to a smart sparse.
Or I can also say this as sparse vector search.
So here you could see in semantic search I have a vector store right.
Which has all the vectors embedded over here.
Right.
So here all my text is basically converted into dense vectors.
And this dense vector is basically store it in my stored in my, um, uh, my uh, vector store DB.
Right.
In the case of synthetic search, you will be able to see that we will be having sparse matrix.
So all this text will basically be converted into sparse matrix.
Now if you remember or if you know right what is sparse matrix.
And how do we get a sparse matrix.
Uh, if you remember if you know about this techniques like one hot encoding bag of words or TF-IDF,
right?
Whenever we try to use this techniques, we will be able to convert our text into sparse matrix.
Now, when we say sparse matrix, that basically means there will be maximum numbers of zeros and ones,
right wherever a specific word will be available in a sentence that will be one and remaining all will
be zeros.
Right.
So this this is basically called as a sparse matrix.
And with the help of this sparse matrix here we are just focusing on a specific keyword like this keyword.
Right.
Whatever keyword we are actually trying to search.
And uh, this sparse matrix also has some very important properties because from this we will be able
to probably do the keyword search itself.
and this is nothing, but it can also be considered as an exact search.
So in the case of hybrid search, it is just going to combine both this particular techniques, that
is semantic search and syntactic search.
Or I can also say this as exact search or keyword search.
Right.
It is just not going to be dependent on one of the search mechanism.
And similarly there are different different kind of databases that are available Just to show you as
an example, uh, uh, like there's a database like Cassandra, right?
Um, we will be using a library called as cashier.
Right.
And with the help of this cashier.
You'll be able to see that we'll be able to access the Cassandra database.
I will show this as we go ahead.
Okay.
But right now, just try to understand over here what exactly Hybrid Search is doing.
Right?
It is providing it is combining multiple search methodologies.
Um, and uh, if I probably talk about in most of the e-commerce website and all hybrid search is definitely
implemented in the back end.
Okay.
Now let's go ahead and see that.
How does and hybrid search work okay.
So here.
We will go ahead and write how does hybrid search work?
So let's say, uh, in the case of hybrid search, what happens?
Let's say if I have a specific documents for all these documents when we are storing this, uh, when
we are storing this in our vector database, let's say this is my vector database, okay?
And this vector database supports both.
Right.
What does it basically supports.
It supports vector search.
Dense vector search.
Along with this it also supports keyword search.
It has both this particular property right now whenever I have this particular document inside this
inside this particular database or inside this particular vector store DB, this documents will also
be converted into a sparse matrix, right, by using techniques like TF-IDF or bag of words.
So this kind of embeddings, it will be converted and it will be stored inside this particular vector
database.
Along with that, for the same documents, we will also try to convert this into vectors, which will
be my dense vectors.
Dense vectors does not have maximum number of zeros or ones.
Okay, that is the difference.
And we will try to store this particular vectors inside this particular vector store db.
And again here you can use different different techniques.
Or the techniques can be anything as you want.
Uh like we have seen.
Right.
We have used different different embedding techniques.
We have used OpenAI embedding techniques, we have used llama embedding techniques.
Or if you want hugging face embedding techniques.
So by all these embedding techniques we can convert our text into dense vectors.
And we can store this in this particular vector database.
Now the idea is very simple since we are storing this in both this particular format sparse matrix and
dense vector matrix whenever a user puts up any kind of queries now with respect to this particular
query, I will be able to retrieve two different kind of result.
Okay.
One result, uh, will be something related to my exact match or exact search or keyword search.
And with respect to this keyword search, let's say whatever results I am getting, it will be my top
K results, right?
This will be based on this particular keyword search or exact search.
So here uh sorry keyword search.
I have written it downwards.
Right.
So here you'll be able to see that I'll be whenever a user probably queries any information over here
from this particular vector database.
First of all, for the user query again their query will also be divided into two types.
It will be converted into two types.
One is the sparse vectors and the other one is something like dense vectors.
Right.
So both these vectors it will be converted.
Then this part based on the sparse vectors.
Uh, here we are specifically going to get the keyword search.
And here we are going to get the results right.
Let's say the top k results I'll be getting over here based on this dense vectors, I am actually going
to do the vector search.
And again I'll be getting another another results over here.
Right.
Let's say this is my result one result one top k result one.
And this is my top k result two okay.
Now top k result one and top k result two are specifically there.
This is basically done on the vector search which is nothing, but it is a semantic search.
This is basically a semantic search, similar kind of elements.
And this is nothing but this is a keyword search, right?
Based on the sparse vectors that we have actually taken.
Now in hybrid search, what we do is that we combine both of them based on weightage okay.
And then we finally get the final response.
Now the question arises how we are going to probably combine this particular result one and result two
to weightage based on this information that we have got and for.
And finally, when we get the response, we can then further combine this with our LM model and summarize
this entire data.
Okay, that is something separate.
But still the question arises how do we combine result one and result two, and how based on weightage
we take our final result, right.
Final result.
And that is what we will be seeing in our next video.
Um, the technique is very simple and the technique name is something called as reciprocal reciprocal
rank fusion in hybrid search.
Okay.
In hybrid search.
So this technique we will be discussing in our next video which will be the core behind this particular
hybrid search.
But the idea is very simple.
In hybrid search we take a combination of two different things.
We try to do the search based on semantic synthetic, which will be the combination of dense vector
search and exact search or keyword search.
Okay.
So both this particular technique we basically use.
So yes, this was it.
In the next video we are going to discuss about the reciprocal rank fusion and hybrid search and how
we can basically combine both this particular results based on some kind of weightage.
Right.
When I say top k result one, top k result two, that I may get five results right?
Based on the keyword search, I may get five results based on the vector search.
Something like that.
Right.
So yes, this was it.
I will see you in the next video.
Thank you.