Hello guys.
So we are going to continue the discussion with respect to the hybrid search.
And uh, here now we are going to implement a complete amazing project with the help of hybrid search
using Lang Chain.
Now, uh, for this, uh, we will be using a vector database, which is called as pine code.
Now, pine cone, uh, vector database provides you this both.
Right.
Like, uh, you will be able to do the keyword search.
You will be able to do the semantic search.
So let's go ahead.
And first of all we will set up our account.
And then with the help of code we will try to write each and every thing.
And we'll try to probably go ahead and create our, you know, the vector database itself.
So first of all just go ahead and sign up over here.
It is completely for free.
Uh, first two first 2 to 3 instances you can create completely for free.
So I have already signed up.
So I will go ahead and log in.
You have to just go to pinecone.io.
Now once you go inside this, you will be able to see that I have already created one pinecone hybrid
search.
Right?
And if you go inside this, there are some kind of data that is already available over here.
Right?
But as usual, uh, we will not probably work with this, but instead we'll go ahead and create our
new index.
Okay, but since I really want to do everything from with the help of code.
So, uh, we will try to create the instance even with the help of code.
So first of all that you actually require is your API key okay.
So just click on API key.
Just make sure that this will be the default API key that will be provided over here.
And uh you can also rotate the key values like you can probably go ahead and change it uh, if somebody
probably sees it.
Right.
So here I will copy this particular key value.
And then I will start my coding over here.
Right.
So uh, here I will be creating my instance.
Everything as such.
Okay.
Now just quickly to go ahead with I have some good requirement dot txt here.
You'll be able to see so many different libraries.
But uh for lang chain obviously you require a lot of libraries.
Uh, but if I probably talk about with respect to pinecone, I will be installing some other libraries
over here.
So let me quickly go ahead and, you know, just paste it over here.
So I'll be installing pinecone client, pinecone text and pinecone notebooks.
Okay.
So uh, these are the initial libraries that we will be requiring.
So I have already done the installation since I don't want to make my video very long.
So here you'll be able to see a green tick and already the entire, uh, you know, the installation
has basically happened.
Then I will go ahead and write my API underscore key.
This API underscore key will basically be replaced with the values over here.
Right.
So I'm just going to create this particular variable.
You can also create an environment variable if you want okay.
Perfect.
Now uh since we are using Lang chain.
So please make sure that we will go ahead and import this from lang chain underscore community dot retrievers
I'm going to import I'm going to import a library which is nothing but pinecone hybrid search retriever.
Now this this class is responsible in doing both the semantic search and syntactic search.
Right.
So keyword search like it.
It has the combination of both the sparse matrix and dense vector search.
And for that we have to create a retriever.
And this is the class that we are going to use.
So let's go ahead and execute this.
And after executing this uh, the next thing will be that we need to make sure that we first of all
create our index in our pinecone.
Right.
Without this, it will be of no use.
So here, uh, quickly, what I will do, I will just go ahead and import import OS, since I have already
imported pinecone.
So I will go ahead and write from pinecone import pinecone comma server list spec.
Okay.
And then I will just go ahead and create my index name.
My index name over here I will say, hey, this will be my hybrid search example.
And please make sure that you give all this in a smaller letter.
Otherwise you'll be getting an error.
So I will say hey hybrid search Lang long chain pine cone.
Okay, so this will basically be my index name.
Okay.
Uh, once I probably create the index name, we have to initialize the pinecone client.
Okay.
Then for initializing the pinecone client, I'll write PC is equal to pinecone, and we will initialize
with this API key that I have actually created.
Right.
So over here is my API key.
You can create your own API keys.
No need to use mine okay.
Because as soon as I upload this video uh, this will get deleted.
Okay.
All I'll rotate the key.
Then, uh, we will go ahead and create the index.
And here I'm going to basically write if index underscore name is not in PC dot list indexes.
Right.
We are listing all the indexes.
If this particular name is not there, then only we should probably go ahead and create this.
So I will write PC dot create index create underscore index.
And here I'm going to give my name is equal to index underscore name whatever index underscore name
we have specifically taken.
Uh, whenever we try to create an index we also have to give our dimension.
Okay.
So here uh, like what will be the vector dimension.
So here I'm going to use the vector dimension is equal to 384.
The reason is very simple.
Uh, the reason why I'm using and this is nothing but this is my dimension of dense model okay.
Dense vector.
The reason is very simple why I'm using 384.
Because, uh, the embedding technique that I'm actually going to use is a hugging face embedding technique,
which is nothing but sentence transformer that by default converts any text into 384 dimensions.
So that is the reason why I'm creating this particular index with this dimension.
Okay.
Uh, along with this, the next thing that I'm going to use is metric.
So for my metric here I will be using a dot product.
So this is all information you specifically get when you are creating an index.
Let's say if I go to over here and create my index so it's asking you for the dimension right over here.
By default you can see metrics.
Also it asks you whether you want to go with dot product cosine.
And all right you can select any of this particular metric.
So that is the reason why I am giving a metric over here.
Right.
So I'm saying hey just go ahead and do the dot product okay.
And uh this dot product specifically I am going to do it for sparse values.
Right.
Sparse values supported only for dot product because we are creating this hybrid search for both of
them.
Right.
And then finally you have the spec is equal to uh.
For this particular spec I will be using serverless spec.
And I have to probably give my cloud information like AWS it is and which region this also you'll be
able to get inside that particular uh, so I will just go ahead and write us East one.
Okay.
So if you remember, uh, just now we saw this particular index also.
So if I go ahead and see this here you have the serverless okay.
All the information is over here let's say.
And here you can select this US East one and all this is locked.
So that is the reason I have written this.
I have selected AWS.
If you go ahead with Google you have to go ahead and write this right.
So this is just a starter plan to go ahead with.
So I'm going to use this free one okay.
Perfect.
So let's see whether this gets executed.
So here I think everything looks good enough I have my index name uh PC dot create.
And here I will just write comma okay.
Oh okay.
Just a second I will just update this entire thing.
Okay, perfect.
Now, this looks good.
Exactly right.
Uh, I have all the details.
Now, let's go ahead and create this index.
So here you will be able to see it will take some amount of time to create this index.
But this is going to get created directly inside our vector database.
Now if I go ahead and click on index, if I reload this I think it should get created okay.
So here is my hybrid search link chain pinecone.
And right now the dimension is 384.
I don't have anything.
The metric is dot product.
The reason of using metric is that I really want to support both.
Uh, that is for my sparse matrix and my vector index.
Right.
So here I have my all details.
Now let's go ahead and further do the more coding over here.
I have already created my API.
Now let's click on my index and I will say hey PC dot index.
Okay PC dot index.
And here I'm just going to give my index underscore name.
So let's go ahead and create see this index over here.
So this is my pinecone index at this particular memory location.
And obviously my index name that you'll be able to see over here that index name based on this index
name I'm trying to take out that details.
Okay.
Um, now first thing first, uh, you know that, right?
Our entire project is a combination of two different.
One, I'll try to create a vector embedding, and the other one, I will try to create a sparse matrix.
Okay.
And that is what hybrid search basically tells you to do it right.
So first of all what I will do I will go ahead and import OS.
And uh here I will write from dot EMV import load underscore dot env okay.
And we will be loading all our environment variables right now just to give you an idea about the environment
variables, since I'm going to use an open source embedding technique from lang chain from Huggingface
itself.
So I will go ahead and import from lang chain underscore Huggingface.
Okay, I'm going to import my hugging face embeddings okay.
And here one of the embedding technique that we regularly use and with respect to the same dimensions,
I'll be using this hugging face embeddings.
And here I'm going to use my model underscore name is equal to all Dash mini.
This is a sentence transformer lm l six and version v2 okay.
So this will specifically be my embeddings.
So here let's go ahead and see the embeddings over here quickly.
Uh yeah.
So my embedding is getting downloaded.
And this will probably take some amount of time.
And here you can see the word embedding is nothing but 384.
Please note uh before doing this, if you directly execute it, it will not work.
Because the reason it is working because I have already imported OS dot environ.
You have to create this huggingface token right?
H f underscore token.
And this is already present in my environment variable.
So I will go ahead and write OS dot get env is nothing but h f underscore token.
You have to execute along with this if you have not.
If you do not have the h f underscore token, this will not get loaded.
Okay.
So please make sure that you have this particular code.
Perfect.
So this was it.
Um now let's go ahead and execute it once again.
And now I will be having my entire huggingface embedding.
Now, this embedding technique is used to create your dense vectors.
Uh, if I go ahead and proceed further, uh, for my sparse matrix, I'm going to use one more.
That is pinecone dot uh, and pinecone underscore text for my sparse.
There is a, there is, there is an encoder which is called as BM two five encoder.
Now this particular encoder uses tf IDF by default okay tf IDF techniques.
So here I'm going to write PAF two five underscore encoder okay is equal to nothing.
But I will just go ahead and initialize this encoder.
And I will say hey use the default one which is nothing but tf IDF okay.
And this basically becomes my BF two five encoder.
Okay.
And this is responsible in doing the sparse matrix conversion for any queries or for any text.
Okay.
Now, uh, I hope uh, you are very much clear with all these things now, and you are able to probably
do all, all these amazing stuffs.
And all right now I will try to create some sentences.
Let's go ahead and create some sentences.
And this sentences will be a list okay.
And just to show you one example, I will just be creating some sentence over here with respect to this
particular list.
I will have this three sentences.
Let's say uh, in 2023 I visited Paris, in 2022, I visited New York, and in 20 and 2021 I visited
Ariana.
Okay, now I will take all this information and quickly I will go ahead and write Bm25 underscore encoder
dot.
We are just going to fit on this entire sentences okay.
So once we do this uh, in short we are just applying tf IDF values on this sentences.
Okay.
Perfect.
So once I do this fit along with this, what I will do I will store the values to a JSON file.
Okay.
So here, uh, I will just go ahead and write p f bm BM 25 encoder dot dump okay.
And I'm going to basically dump my BM 25 underscore values in the form of dot JSON.
Okay.
And here you'll be able to see that I'm actually dumping this entire information.
You can also go ahead and load this just to make sure that you have that entire dump okay.
And you can also load it so has no attribute dumpy okay.
It was dump.
No worries.
Okay.
So now here you can see that I have my encoder.
And this is specifically for sparse matrix.
Okay.
okay.
Um now one more thing that you will be able to see over here is that, uh, once I do this particular
sparse matrix dump, uh, so I've actually created the sparse matrix for this, right.
Now let's go ahead and create my retriever.
And this retriever will be important because with the help of this specific retriever, only I will
have the combination of both, uh, the embedding techniques and, uh, obviously the sparse embedding
techniques.
Right.
So for that, I will go ahead and create this pinecone hybrid search retriever.
And here I'm going to use my embeddings is equal to whatever embedding technique I have used.
That is nothing but huggingface embeddings.
And then my sparse encoder which will be assigned to my b b b b m to five encoder.
And I have to also specify my index right.
So this will basically be my index.
Now once I execute this this is my retriever.
Now with the help of this particular retriever it supports both embedding the vector dense embedding
technique and along with that the sparse matrix.
Okay.
So this is my retriever.
So I will just go ahead and display my retriever okay.
So this is my retriever.
And this is the entire information that you will be able to see.
Right.
So finally uh let's do one thing.
Let's quickly go ahead and add all the text that I want.
Right?
So I will say, hey, we will go ahead and add the text and the text that we are going to add.
Is all my sentences inside my index.
Right.
So what are my sentences?
Here you go.
I will copy this and I will paste it over here okay.
So let's go ahead and execute this.
Now here we'll be able to see that it will get inserted inside that particular index.
Right.
The index.
uh, the pinecone index that we have created.
Now it's time we can probably retrieve any results.
First of all, let's see whether it has got inserted or not.
So if I go over here, here you can see, see it's loading.
And all these three sentences are now available.
And this is all 384 dimensions.
Right.
So this 1 in 2021 I visited context uh Orleans.
In 2023 I visited New York Paris and all right, now let's go ahead and ask some questions.
And this time it is going to use the combination of both right.
Your vector dense search and the other search.
So here I'll say with what city did I visit last okay.
So here you'll be able to see visit last.
So here you can see I in 2021 I visited New Orleans, then New York, then visited Paris.
All this page content is basically able to come over here.
It is used both, uh, because this entire hybrid search retrieval is using both embeddings and the
sparse matrix.
Okay.
If I say recent.
So in 2021 all this information is basically coming up.
Okay.
Which city did I visit first?
Okay.
So here you can see in 2021 I visited neon.
So all the information are basically you are able to see it over here.
But internally it is taking a combination of both hybrid search.
And uh, this is what is an amazing example which you can also go ahead and do it okay.
So I hope you like this particular video.
Uh, I will see you all in the next video.
Thank you.
Take care.
Bye bye.