1
00:00:00,000 --> 00:00:01,000
Hello guys.

2
00:00:01,000 --> 00:00:06,000
So in this video we are going to discuss about something called as hybrid search.

3
00:00:06,000 --> 00:00:07,000
Okay.

4
00:00:07,000 --> 00:00:14,000
And uh, this kind of search specifically uh, we use in our Rag applications also.

5
00:00:14,000 --> 00:00:14,000
Right.

6
00:00:14,000 --> 00:00:21,000
So till now if you are following all the modules, uh, the rag that we have specifically created,

7
00:00:21,000 --> 00:00:28,000
you know, let's say that if I have, uh, documents, let's say if this is my documents, then what

8
00:00:28,000 --> 00:00:32,000
we do is that we try to divide this documents into chunk of documents.

9
00:00:32,000 --> 00:00:35,000
Let's say this is d one, this is d two.

10
00:00:35,000 --> 00:00:37,000
This is d three.

11
00:00:37,000 --> 00:00:41,000
Like this we usually divide into multiple chunk of documents.

12
00:00:41,000 --> 00:00:42,000
Right.

13
00:00:42,000 --> 00:00:43,000
And this is d n.

14
00:00:44,000 --> 00:00:51,000
Now after dividing all these documents into chunk of documents then we specifically convert this into

15
00:00:51,000 --> 00:00:52,000
vectors.

16
00:00:52,000 --> 00:00:53,000
right?

17
00:00:54,000 --> 00:01:00,000
So here we are basically converting into embedding vectors.

18
00:01:01,000 --> 00:01:11,000
Then after we convert all this text to vectors, the next step is that we we store all these vectors

19
00:01:11,000 --> 00:01:14,000
in some kind of vector database.

20
00:01:15,000 --> 00:01:16,000
Right.

21
00:01:16,000 --> 00:01:23,000
So let's say one of the vector databases that we have already seen, vector database.

22
00:01:25,000 --> 00:01:28,000
We also say it as something called as vector store DB.

23
00:01:31,000 --> 00:01:38,000
Right now once this entire information is basically stored over here right, it will be stored it in

24
00:01:38,000 --> 00:01:39,000
the form of vectors.

25
00:01:39,000 --> 00:01:40,000
Right.

26
00:01:40,000 --> 00:01:44,000
We will be having some kind of numbers over here itself right now.

27
00:01:44,000 --> 00:01:54,000
When a user queries, whenever a user queries right over here, then what happens?

28
00:01:54,000 --> 00:01:59,000
Usually in this particular scenario this particular query is also converted into vectors.

29
00:02:00,000 --> 00:02:06,000
And then a vector dense search or vector search is basically done.

30
00:02:07,000 --> 00:02:13,000
Vector search is basically done in this particular vector database.

31
00:02:13,000 --> 00:02:19,000
So once we have this particular vectors we do a vector search from this particular vector database.

32
00:02:19,000 --> 00:02:27,000
And here are some of the algorithms like um uh over here one of the algorithm that may uh, you know,

33
00:02:27,000 --> 00:02:33,000
take place over here for basically to match the exact vector search, we can actually use cosine similarity.

34
00:02:34,000 --> 00:02:34,000
Right.

35
00:02:34,000 --> 00:02:40,000
And based on this cosine similarity check, we will be able to get the response right.

36
00:02:41,000 --> 00:02:47,000
And once we get this response we can combine this with our prompt template.

37
00:02:50,000 --> 00:02:51,000
Plus LM model.

38
00:02:52,000 --> 00:02:57,000
And finally we can summarize any information as my final output.

39
00:02:57,000 --> 00:02:57,000
Right.

40
00:02:57,000 --> 00:03:03,000
So in this scenario you will be able to see that the kind of search that we are doing right.

41
00:03:03,000 --> 00:03:04,000
The vector dense search.

42
00:03:04,000 --> 00:03:08,000
This search is basically called as semantic search.

43
00:03:10,000 --> 00:03:11,000
Semantic search.

44
00:03:11,000 --> 00:03:16,000
Semantic search basically means we are trying to find out the similar vectors from this particular vector

45
00:03:16,000 --> 00:03:22,000
database, and based on that, when that similarity search is done, we are able to get the response.

46
00:03:23,000 --> 00:03:27,000
And this kind of rag, most of the Rag application uses this kind of search.

47
00:03:27,000 --> 00:03:33,000
And if I talk about vector databases like face chroma right.

48
00:03:34,000 --> 00:03:36,000
They have this entire features like this.

49
00:03:36,000 --> 00:03:37,000
Right.

50
00:03:37,000 --> 00:03:44,000
But I really want to talk about one more amazing search mechanism, which is called as hybrid search.

51
00:03:44,000 --> 00:03:50,000
Okay, so here I have this hybrid search.

52
00:03:51,000 --> 00:03:59,000
Now inside this hybrid search here we are just not focusing on only vector dense search mechanism.

53
00:03:59,000 --> 00:03:59,000
Right.

54
00:03:59,000 --> 00:04:05,000
Here we specifically combine combine.

55
00:04:06,000 --> 00:04:10,000
multiple search techniques.

56
00:04:14,000 --> 00:04:15,000
Right.

57
00:04:15,000 --> 00:04:22,000
And what kind of techniques we will specifically apply over here.

58
00:04:22,000 --> 00:04:24,000
We will be combining multiple search techniques.

59
00:04:24,000 --> 00:04:31,000
Now specifically if I really want to divide all this particular technique one is the semantic search.

60
00:04:35,000 --> 00:04:39,000
The second one is something called as syntactic search.

61
00:04:40,000 --> 00:04:40,000
Okay.

62
00:04:41,000 --> 00:04:43,000
Now I will talk more about it.

63
00:04:43,000 --> 00:04:45,000
What exactly are these kind of search basically.

64
00:04:46,000 --> 00:04:47,000
So one is semantic search.

65
00:04:47,000 --> 00:04:49,000
One is syntactic search.

66
00:04:49,000 --> 00:04:49,000
Semantic search.

67
00:04:49,000 --> 00:04:54,000
Specifically we say it as a dense vector search.

68
00:04:56,000 --> 00:05:02,000
So here we are specifically trying to search elements that are similar right.

69
00:05:03,000 --> 00:05:08,000
And obviously when we are using this dense vector search we basically are checking out the similar kind

70
00:05:08,000 --> 00:05:09,000
of content.

71
00:05:09,000 --> 00:05:10,000
Right?

72
00:05:10,000 --> 00:05:12,000
If I probably say with respect to syntactic search.

73
00:05:12,000 --> 00:05:13,000
Right.

74
00:05:13,000 --> 00:05:21,000
Or I can also say this as exact search, I can also say this as keyword search.

75
00:05:22,000 --> 00:05:22,000
Right.

76
00:05:22,000 --> 00:05:24,000
Keyword search.

77
00:05:24,000 --> 00:05:31,000
And if I also want to probably provide some more names like exact search and uh, keyword search.

78
00:05:31,000 --> 00:05:36,000
Uh, and this usually happens with respect to a smart sparse.

79
00:05:37,000 --> 00:05:40,000
Or I can also say this as sparse vector search.

80
00:05:41,000 --> 00:05:47,000
So here you could see in semantic search I have a vector store right.

81
00:05:47,000 --> 00:05:50,000
Which has all the vectors embedded over here.

82
00:05:50,000 --> 00:05:50,000
Right.

83
00:05:50,000 --> 00:05:56,000
So here all my text is basically converted into dense vectors.

84
00:05:57,000 --> 00:06:04,000
And this dense vector is basically store it in my stored in my, um, uh, my uh, vector store DB.

85
00:06:04,000 --> 00:06:04,000
Right.

86
00:06:05,000 --> 00:06:10,000
In the case of synthetic search, you will be able to see that we will be having sparse matrix.

87
00:06:10,000 --> 00:06:15,000
So all this text will basically be converted into sparse matrix.

88
00:06:15,000 --> 00:06:19,000
Now if you remember or if you know right what is sparse matrix.

89
00:06:19,000 --> 00:06:21,000
And how do we get a sparse matrix.

90
00:06:21,000 --> 00:06:31,000
Uh, if you remember if you know about this techniques like one hot encoding bag of words or TF-IDF,

91
00:06:31,000 --> 00:06:31,000
right?

92
00:06:32,000 --> 00:06:38,000
Whenever we try to use this techniques, we will be able to convert our text into sparse matrix.

93
00:06:38,000 --> 00:06:45,000
Now, when we say sparse matrix, that basically means there will be maximum numbers of zeros and ones,

94
00:06:45,000 --> 00:06:51,000
right wherever a specific word will be available in a sentence that will be one and remaining all will

95
00:06:51,000 --> 00:06:52,000
be zeros.

96
00:06:52,000 --> 00:06:53,000
Right.

97
00:06:53,000 --> 00:06:56,000
So this this is basically called as a sparse matrix.

98
00:06:57,000 --> 00:07:04,000
And with the help of this sparse matrix here we are just focusing on a specific keyword like this keyword.

99
00:07:04,000 --> 00:07:04,000
Right.

100
00:07:04,000 --> 00:07:07,000
Whatever keyword we are actually trying to search.

101
00:07:07,000 --> 00:07:15,000
And uh, this sparse matrix also has some very important properties because from this we will be able

102
00:07:15,000 --> 00:07:18,000
to probably do the keyword search itself.

103
00:07:18,000 --> 00:07:22,000
and this is nothing, but it can also be considered as an exact search.

104
00:07:22,000 --> 00:07:28,000
So in the case of hybrid search, it is just going to combine both this particular techniques, that

105
00:07:28,000 --> 00:07:30,000
is semantic search and syntactic search.

106
00:07:30,000 --> 00:07:33,000
Or I can also say this as exact search or keyword search.

107
00:07:33,000 --> 00:07:34,000
Right.

108
00:07:34,000 --> 00:07:38,000
It is just not going to be dependent on one of the search mechanism.

109
00:07:38,000 --> 00:07:44,000
And similarly there are different different kind of databases that are available Just to show you as

110
00:07:44,000 --> 00:07:50,000
an example, uh, uh, like there's a database like Cassandra, right?

111
00:07:50,000 --> 00:07:53,000
Um, we will be using a library called as cashier.

112
00:07:53,000 --> 00:07:53,000
Right.

113
00:07:53,000 --> 00:07:55,000
And with the help of this cashier.

114
00:07:55,000 --> 00:07:58,000
You'll be able to see that we'll be able to access the Cassandra database.

115
00:07:58,000 --> 00:08:00,000
I will show this as we go ahead.

116
00:08:00,000 --> 00:08:00,000
Okay.

117
00:08:00,000 --> 00:08:05,000
But right now, just try to understand over here what exactly Hybrid Search is doing.

118
00:08:05,000 --> 00:08:06,000
Right?

119
00:08:06,000 --> 00:08:09,000
It is providing it is combining multiple search methodologies.

120
00:08:10,000 --> 00:08:16,000
Um, and uh, if I probably talk about in most of the e-commerce website and all hybrid search is definitely

121
00:08:16,000 --> 00:08:18,000
implemented in the back end.

122
00:08:18,000 --> 00:08:18,000
Okay.

123
00:08:19,000 --> 00:08:20,000
Now let's go ahead and see that.

124
00:08:20,000 --> 00:08:23,000
How does and hybrid search work okay.

125
00:08:23,000 --> 00:08:24,000
So here.

126
00:08:26,000 --> 00:08:33,000
We will go ahead and write how does hybrid search work?

127
00:08:37,000 --> 00:08:44,000
So let's say, uh, in the case of hybrid search, what happens?

128
00:08:44,000 --> 00:08:55,000
Let's say if I have a specific documents for all these documents when we are storing this, uh, when

129
00:08:55,000 --> 00:09:00,000
we are storing this in our vector database, let's say this is my vector database, okay?

130
00:09:00,000 --> 00:09:03,000
And this vector database supports both.

131
00:09:03,000 --> 00:09:04,000
Right.

132
00:09:04,000 --> 00:09:05,000
What does it basically supports.

133
00:09:05,000 --> 00:09:08,000
It supports vector search.

134
00:09:09,000 --> 00:09:10,000
Dense vector search.

135
00:09:10,000 --> 00:09:12,000
Along with this it also supports keyword search.

136
00:09:14,000 --> 00:09:21,000
It has both this particular property right now whenever I have this particular document inside this

137
00:09:21,000 --> 00:09:26,000
inside this particular database or inside this particular vector store DB, this documents will also

138
00:09:26,000 --> 00:09:35,000
be converted into a sparse matrix, right, by using techniques like TF-IDF or bag of words.

139
00:09:36,000 --> 00:09:40,000
So this kind of embeddings, it will be converted and it will be stored inside this particular vector

140
00:09:40,000 --> 00:09:40,000
database.

141
00:09:40,000 --> 00:09:46,000
Along with that, for the same documents, we will also try to convert this into vectors, which will

142
00:09:46,000 --> 00:09:47,000
be my dense vectors.

143
00:09:47,000 --> 00:09:50,000
Dense vectors does not have maximum number of zeros or ones.

144
00:09:50,000 --> 00:09:52,000
Okay, that is the difference.

145
00:09:52,000 --> 00:09:56,000
And we will try to store this particular vectors inside this particular vector store db.

146
00:09:56,000 --> 00:09:58,000
And again here you can use different different techniques.

147
00:09:59,000 --> 00:10:01,000
Or the techniques can be anything as you want.

148
00:10:01,000 --> 00:10:02,000
Uh like we have seen.

149
00:10:02,000 --> 00:10:03,000
Right.

150
00:10:03,000 --> 00:10:05,000
We have used different different embedding techniques.

151
00:10:05,000 --> 00:10:09,000
We have used OpenAI embedding techniques, we have used llama embedding techniques.

152
00:10:09,000 --> 00:10:13,000
Or if you want hugging face embedding techniques.

153
00:10:13,000 --> 00:10:18,000
So by all these embedding techniques we can convert our text into dense vectors.

154
00:10:18,000 --> 00:10:21,000
And we can store this in this particular vector database.

155
00:10:22,000 --> 00:10:28,000
Now the idea is very simple since we are storing this in both this particular format sparse matrix and

156
00:10:28,000 --> 00:10:36,000
dense vector matrix whenever a user puts up any kind of queries now with respect to this particular

157
00:10:36,000 --> 00:10:40,000
query, I will be able to retrieve two different kind of result.

158
00:10:41,000 --> 00:10:41,000
Okay.

159
00:10:42,000 --> 00:10:49,000
One result, uh, will be something related to my exact match or exact search or keyword search.

160
00:10:50,000 --> 00:10:55,000
And with respect to this keyword search, let's say whatever results I am getting, it will be my top

161
00:10:55,000 --> 00:10:57,000
K results, right?

162
00:10:57,000 --> 00:11:00,000
This will be based on this particular keyword search or exact search.

163
00:11:00,000 --> 00:11:04,000
So here uh sorry keyword search.

164
00:11:04,000 --> 00:11:05,000
I have written it downwards.

165
00:11:05,000 --> 00:11:06,000
Right.

166
00:11:06,000 --> 00:11:14,000
So here you'll be able to see that I'll be whenever a user probably queries any information over here

167
00:11:14,000 --> 00:11:15,000
from this particular vector database.

168
00:11:15,000 --> 00:11:21,000
First of all, for the user query again their query will also be divided into two types.

169
00:11:21,000 --> 00:11:23,000
It will be converted into two types.

170
00:11:23,000 --> 00:11:29,000
One is the sparse vectors and the other one is something like dense vectors.

171
00:11:29,000 --> 00:11:30,000
Right.

172
00:11:30,000 --> 00:11:33,000
So both these vectors it will be converted.

173
00:11:33,000 --> 00:11:35,000
Then this part based on the sparse vectors.

174
00:11:35,000 --> 00:11:38,000
Uh, here we are specifically going to get the keyword search.

175
00:11:38,000 --> 00:11:41,000
And here we are going to get the results right.

176
00:11:41,000 --> 00:11:46,000
Let's say the top k results I'll be getting over here based on this dense vectors, I am actually going

177
00:11:46,000 --> 00:11:47,000
to do the vector search.

178
00:11:47,000 --> 00:11:50,000
And again I'll be getting another another results over here.

179
00:11:50,000 --> 00:11:50,000
Right.

180
00:11:50,000 --> 00:11:56,000
Let's say this is my result one result one top k result one.

181
00:11:56,000 --> 00:12:00,000
And this is my top k result two okay.

182
00:12:00,000 --> 00:12:04,000
Now top k result one and top k result two are specifically there.

183
00:12:04,000 --> 00:12:09,000
This is basically done on the vector search which is nothing, but it is a semantic search.

184
00:12:09,000 --> 00:12:12,000
This is basically a semantic search, similar kind of elements.

185
00:12:13,000 --> 00:12:15,000
And this is nothing but this is a keyword search, right?

186
00:12:15,000 --> 00:12:17,000
Based on the sparse vectors that we have actually taken.

187
00:12:18,000 --> 00:12:24,000
Now in hybrid search, what we do is that we combine both of them based on weightage okay.

188
00:12:25,000 --> 00:12:28,000
And then we finally get the final response.

189
00:12:28,000 --> 00:12:35,000
Now the question arises how we are going to probably combine this particular result one and result two

190
00:12:35,000 --> 00:12:41,000
to weightage based on this information that we have got and for.

191
00:12:41,000 --> 00:12:45,000
And finally, when we get the response, we can then further combine this with our LM model and summarize

192
00:12:45,000 --> 00:12:46,000
this entire data.

193
00:12:46,000 --> 00:12:48,000
Okay, that is something separate.

194
00:12:48,000 --> 00:12:55,000
But still the question arises how do we combine result one and result two, and how based on weightage

195
00:12:55,000 --> 00:12:59,000
we take our final result, right.

196
00:12:59,000 --> 00:13:01,000
Final result.

197
00:13:01,000 --> 00:13:04,000
And that is what we will be seeing in our next video.

198
00:13:05,000 --> 00:13:11,000
Um, the technique is very simple and the technique name is something called as reciprocal reciprocal

199
00:13:12,000 --> 00:13:18,000
rank fusion in hybrid search.

200
00:13:20,000 --> 00:13:20,000
Okay.

201
00:13:20,000 --> 00:13:21,000
In hybrid search.

202
00:13:21,000 --> 00:13:29,000
So this technique we will be discussing in our next video which will be the core behind this particular

203
00:13:29,000 --> 00:13:30,000
hybrid search.

204
00:13:30,000 --> 00:13:31,000
But the idea is very simple.

205
00:13:31,000 --> 00:13:34,000
In hybrid search we take a combination of two different things.

206
00:13:35,000 --> 00:13:41,000
We try to do the search based on semantic synthetic, which will be the combination of dense vector

207
00:13:41,000 --> 00:13:43,000
search and exact search or keyword search.

208
00:13:43,000 --> 00:13:43,000
Okay.

209
00:13:43,000 --> 00:13:45,000
So both this particular technique we basically use.

210
00:13:46,000 --> 00:13:47,000
So yes, this was it.

211
00:13:47,000 --> 00:13:52,000
In the next video we are going to discuss about the reciprocal rank fusion and hybrid search and how

212
00:13:52,000 --> 00:13:56,000
we can basically combine both this particular results based on some kind of weightage.

213
00:13:56,000 --> 00:13:57,000
Right.

214
00:13:57,000 --> 00:14:01,000
When I say top k result one, top k result two, that I may get five results right?

215
00:14:01,000 --> 00:14:05,000
Based on the keyword search, I may get five results based on the vector search.

216
00:14:05,000 --> 00:14:05,000
Something like that.

217
00:14:05,000 --> 00:14:06,000
Right.

218
00:14:06,000 --> 00:14:07,000
So yes, this was it.

219
00:14:07,000 --> 00:14:08,000
I will see you in the next video.

220
00:14:08,000 --> 00:14:09,000
Thank you.