So guys, now let's go ahead and proceed with the discussion with respect to reciprocal rank fusion
and hybrid search.
As I said, whenever we try to query or whenever user queries from this particular vector database,
which supports both vector search and keyword search, which is a combination of semantic search and
exact search over here.
First of all, uh, to get the keyword search, we have to convert this into sparse vectors, and the
sparse vectors will be probably searching from this particular vector database.
And and we get top k results.
Similarly, with the help of dense vectors, we also get a vector search.
And here we are going to get a result that is top k results okay.
Now combining both of them we have to probably consider a response.
Right.
How to probably get the final response.
That is what I'm actually going to discuss now.
Okay.
So here uh, with the help of reciprocal rank fusion and hybrid search, we will be considering two
important things.
Okay, let's say this is my vector database.
Okay.
And from this vector database, you know that whenever user queries.
So based on this queries let's say if this is my dense vectors first we are going to do the vector search.
And we are going to get the top k results, right.
So when I say top k results, let's say that this is my first document.
Right.
And this can be my second document.
Right.
And this can be my third document.
Like this I will be getting a lot of documents similarly over here based on my sparse matrix search.
That is the keyword search that is nothing but keyword search.
I'm actually going to get my top K documents.
So here also I will go ahead and draw this.
So let's say.
Here I'm going to get some other sentences okay.
And the ordering will be different.
Right.
So here uh let's say this is the order that I've actually got.
But I got document one, two, three, four, five, let's say the five documents I've actually got.
And here also we have got some other documents, so let's say some other documents.
But here the numbering is different.
Let's say the numbering is like 43251 okay.
The numbering is different.
This is my result One top K documents and this is my result.
Two top K documents.
Okay, now, considering this, uh, see here I've got in this ordering, that is one, 2345 sentence.
That is that uh, I got in my top k like document one, document two, document three, document four,
document five.
Similarly over here document four, document three.
Document two.
Document five.
Document one.
This is the ranks that we have specifically got.
Now how to calculate the final score.
Because here this is based on my semantic search.
This is based on my keyword search.
When I say semantic search that is basically uh based on cosine similarity okay.
So we are trying to find out the similar vectors and then we are trying to find out our final score.
So in order to calculate the final score the formula is very simple.
Here we'll write summation of one divided by uh a constant plus plus the rank of documents okay.
Now this particular constant that is C it depends on various database.
It depends on various
databases.
Okay.
Now with respect to various databases, this may range between 1 to 60 okay.
And this is a constant case okay.
In various databases they'll set this value to 1020.
And some of the databases I've seen the value to 60 okay.
So since this is a constant we are just not going to focus much into this.
Instead, I will just take up this formula.
Now summation of I is equal to um one divided by rank of t.
Okay, since this is a constant, uh, for calculating in every, uh, the rank of every, uh, documents,
uh, will probably give you the same answer with respect to that particular constant, same impact it
may be creating.
Okay.
Now how do we compute the rank.
Right.
So for this particular sentence you'll be seeing one divided by rank of D.
So if I go ahead and compute this this is nothing but one by one okay.
Uh this is nothing but one by two because this is rank two, right?
This is rank three.
So I'll get one by three.
It is nothing but 0.33.
Then here I have 0.25.
And this is my fifth document.
Let's say the rank is five.
I'm actually going to get point two okay.
Similarly, in this particular case uh this is my document four.
Right.
And if I try to compute this particular rank okay it will be again one by one, one by two, uh, one
by three, one by four, one by five.
So that basically means the document of D4 is basically getting a rank of one in this keyword search
here.
D4 is basically getting 0.25.
Okay, now when we try to combine the final score okay, this is important.
See this.
So if I finally go ahead and take up document one.
So if this is my document one and if I try to calculate the overall rank then what we have to do we
have to do the summation.
So let's see document one is giving one over here and 1.51 by five.
So I will just go ahead and write one plus point two.
So this will be 1.2 right.
Similarly if I go ahead and calculate for document two right.
So document two over here gives 0.5 right.
So here I'm actually going to get 0.5 plus the document two over here is one by three.
That is nothing but 0.33 right?
So here you can see I'm actually going to get 0.83.
So this is my score that I'm actually going to get.
And obviously this score is better than this score.
So I will be getting the document one based on my 50% probability of 50%.
Uh, focus on keyword search, right.
And 50% on semantic search, right?
If you change the weightage, right.
If you change the weightage, then you'll be able to see that some values may also change.
Similarly you have this document three.
Now in the case of document three you'll be able to see that here I have .33.
So let's calculate this .33 plus document three is nothing but point five.
Again here I'm going to get the same value.
See .83.
Now both these ranks are same right.
So if I want to retrieve the top k result obviously this will be shown first.
And then based on this two right where the focus is more whether it is on keyword search or semantic
search.
Let's say if I say 70% focus is there on the keyword search, then over here you can see point five
value is higher than point three.
So it is just going to give you the display the document three in the second rank.
Right.
So this way the weightage can also be provided in keyword search and semantic search okay.
And similarly we go ahead and calculate each and every documents.
So finally you can see that here we can combine result two and result one top key elements.
And then we can use this reciprocal rank fusion in hybrid search by assigning ranks.
And we will be able to calculate a score which will be finally given as a response.
And here we can also assign weightage.
And this can be combined with an prompt template.
And we can summarize with our LM model.
Right.
So this gives an idea about the entire hybrid search right now uh in this section in this next section
what we are going to do is that we're going to also see a practical example of how an hybrid search
is basically done.
But before that, if you know about one more search, right?
That is based on graph knowledge search okay.
Graph knowledge search.
So in this graph knowledge search, uh, I hope you may have heard about graph DB.
One of the most popular open source graph DB is Neo4j's.
Okay.
Now if we are specifically using graph knowledge, uh, sorry, graph DB to store the vectors, let's
say if this is my, uh, database and this database is nothing, but it is my graph DB.
In the case of graph db, I will be having like this nodes which will be connected with relationships.
Right.
So this can be my node one, this can be my node two.
And it will be connected with relationships.
And we can whenever a user tries to query, user tries to query.
In this they are three types of queries that happens.
One is keyword search which you can also be called as semantic search.
Sorry, this can be also called as um uh synthetic search.
Or it can be also called as exact search.
Uh then along with this, it's also suppose semantic search.
Right.
And finally you will be also able to see that it supports, uh, keyword search.
Semantic search.
Along with this one more search is there that is called as graph knowledge search.
Graph knowledge search.
So it supports all these techniques which is quite amazing.
Just imagine hybrids are just supports this two type.
But now with this help of graph knowledge search, we will still be able to create a very good efficient
Rag application, right?
With the help of graph DB.
And this we will be seeing about graph DB in the upcoming videos, uh, in the upcoming sections.
But uh, let's go ahead and see some practical example for uh hybrid search.