Hello guys.
So we are going to continue the discussion with respect to embedding techniques.
And in this video we are going to see about embedding techniques using hugging face.
Now hugging face also has a lot of open source models uh, specifically used for embedding and also
different different types of LM models through which you can actually use them and create your generative
AI application.
So let us go ahead.
And first of all, what you really need to do is that just go to the Hugging face.co website okay.
So in this particular website you'll be able to see that there will be models.
There will be so many things that are available over here with respect to all the tasks.
So first of all just go ahead and create your account.
Because later on we are going to create a lot of end to end projects again using hugging face.
And the best thing about hugging face and Lang Chen is that they have integrate.
They have created a library, which is a integration of both hugging face and language in itself.
So you can call any LLM models that you like over here.
Okay.
So first of all, uh, in order to work this, uh, you really need to have to create your own API key,
right?
So once you go to the settings over here, right, once you click on the settings here, you will be
able to see uh, something called as access token.
So just go ahead and create your access token okay.
So here you can see I've created so many different different access token.
You can just go ahead and click on new token.
Give the token name right what type you want.
You can basically select read because you're just going to read the token uh to access any LM models
okay.
So here I can probably go ahead and write uh lang chain okay.
And here I will just go ahead and select read and I'll generate a token.
Now this is the token that I'm actually going to use in order to work with hugging face.
Uh, right.
All the embedding techniques that are available in hugging face.
So now I'm just going to go back to my code.
Uh, first thing first, what I really need to do, if you remember.
Right.
So here we use this dot env.
So same thing I will be copy and pasting it over here because we need to call that same token.
Right.
So what I will be doing is that I will be creating a token, uh, specific token in my environment variable,
and the format will be something like this.
Okay.
So I will show you the format.
So here you can see this is the format that it will be in your environment variable.
So just go ahead and probably open your environment variable dot env.
And there you can just copy and paste this particular token in a new line.
Right.
Like how we did it for OpenAI.
Again, I'm not going to show you the dot env file.
The reason is very simple because this particular token, uh, I need to keep it hidden.
Right.
So I'll just remove this.
Now what I'm actually going to do I'm just going to write OS dot environ.
Okay.
And let me just go ahead and call my hugging face token h f underscore token.
Okay.
So I'll be setting this up token over here.
And I'll go ahead and write OS dot get env with respect to the same token.
Right.
If I execute it, you'll be able to see that this will get executed successfully.
That basically means in my environment variable, whatever token I have actually created, I've assigned
it to this particular hugging face underscore token.
Okay.
So this is the first step that we need to do.
Okay.
Now the second step is that I will go to my requirement dot txt.
And here I'm going to install one library okay.
The library is nothing but it is basically called as sentence underscore transformers.
Okay.
I have to use this particular library because uh in this libraries the embedding technique is basically
available.
So now I will just go ahead and open my terminal and just go ahead and write pip install minus r requirement
dot txt.
Okay so here you will be able to see this and all.
My requirement dot txt will get downloaded.
Now for sentence transformer.
The default that library that we require is nothing but torch.
So here you can see within 30s it will be getting downloaded.
It is somewhere around 5150 9.8 MB.
Once this is getting downloaded we can use this sentence transformer embedding technique for embeddings,
right?
So till then the installation is basically taking place.
I will just go ahead and uh, also we need to also install one more library.
Since I said that uh, recently Lang Chen and Hugging Face, they have combined the hands together and
they have created another library which is called as lang chain underscore hugging face.
Okay, so we need to also install this library because inside this library only, we'll be finding various
uh, techniques of calling LM models and all, which I will be discussing.
But in this video I'll be focusing more on the embedding technique.
Okay.
So here you can see I have my lecture underscore hugging face.
Um, now what I'm actually going to do is that as soon as this particular installation takes place,
I will also go ahead and install this from now till the installation is taking place.
Let's go back over here.
So I'm just going to go ahead and write my code.
I'll say hey from lang chain underscore hugging face okay I'm going to import.
Hugging face embeddings okay.
So we are going to use this hugging face embeddings over here okay.
Okay.
Now this hugging face embeddings this same hugging face embeddings I can copy and paste it.
And the first, uh, embedding technique that we are specifically going to use is something called a
sentence transformer.
So let me just go ahead and make a field over here.
I'll create a markdown.
And I will also provide some information okay.
Okay.
Over here.
I'll just go ahead and copy and paste it so that you'll also get some kind of description information.
Okay.
So here you can see sentence transformer on hugging face hugging face.
Uh, sentence transformer is a Python framework for state of art sentence text and image embeddings.
One of the embedding model is used in the hugging face embedding class.
We have also added uh analysis called as sentence transformer embedding for users who are familiar with
directly using that package.
Okay.
So this basically uses something called a sentence Bert.
Okay.
Uh, with the help of that, you will be able to call any embedding techniques.
So here I will just go ahead and uh, use this Huggingface embeddings.
Now here also you can see the installation has taken place.
Now let me quickly again do the pip install requirement dot txt because I also need to install this
link chain underscore huggingface.
Okay again installation is something that you will always be requiring.
So I'll just go ahead and execute this.
Now inside this I will give my model name okay.
So here I'm going to say hey this will basically be my model name.
And let's go ahead and give my model name over here.
The model name will be something called as uh, if you go ahead and search in hugging face, this is
the model that we are going to use all mini lm, L6 version two okay.
This will basically be used for embeddings.
So now I'll go and create my embeddings equal to this one.
So it shows syntax okay I have to write from okay from.
So if I go ahead and execute it I think this should be working fine and I should not get any error.
You can also go ahead and install Tqdm so that you will be able to see that how much time it is probably
taking the model to to get downloaded.
Okay, so there are some warnings.
You can ignore this particular warning.
Okay.
But it is going to take some seconds to probably load the model in our system.
Okay.
And then once we do this I will just go ahead and create my text document.
So this will be my text.
Let's say I'll go ahead and write.
This is a test document.
And we will try to convert this into vectors okay.
Remember the reason why this is working?
Because we have already, uh, called this particular h f underscore token over here.
Right.
So here is this is a test document over here.
I will just go ahead and write my query.
Underscore result is equal to embedding embeddings dot.
And here we can go ahead and write our embed underscore query.
Okay.
And once I go ahead and execute this give my text over here.
Then I will be able to see my.
Query underscore results.
So if I go ahead and execute this you'll be able to see this.
And uh, if you want to go ahead and see the dimension of this particular vectors, this particular
sentence, how much vectors it has got, uh, what is the dimension of that.
So I can go ahead and write length of query underscore results.
Okay.
So it's result okay.
Now here you can see it is basically converted into a 384 dimension.
Similarly uh uh, you know you can also do it for embed documents wherein you can give multiple lists
of statements.
And here is one example where I'm writing embedding dot embed documents with respect to this particular
value.
And this is not a test document okay.
So here if I go ahead and see my doc underscore results of zero.
So the first embedding is this one.
And the second embedding also you'll be able to see it.
So this in short actually shows us like how to properly perform with the help of hugging face sentence
transformers.
Similarly you can also do it with hugging face itself.
Okay.
And uh, one is through sentence transformers, which we have already seen it.
Okay.
The other thing is that we can also check something called as hugging face inference API embedding,
but that we will see it in the later stage.
This is just to give you an idea like what all techniques we can specifically use over here.
So, uh, I hope, uh, you were able to understand about the embedding techniques.
Uh, now, see, uh, step by step, we are able to cover each and every thing right from data ingestion
to text splitter to, you know, probably converting vectors, uh, text into vectors.
Then you also saw one example of vector store DB.
Right now there are two important things again after this.
One is vector store DB different different vector store DB will try to see.
And then after that we will also go ahead and explore uh retrievers and chains, which are super important.
And then we can finally go ahead and implement lot of end to end projects.
So yes, this was it for my side.
Uh, I'll see you all in the next video.
Thank you.