Hello guys.
So we are going to continue our discussion with respect to Lang Chin.
Uh, till now, uh, we have discussed about step one that is loading from various data sources with
the help of document loader.
This was loading different kind of data itself.
Then we went to the second step wherein we converted all this documents that we loaded into text chunks
or document chunks.
And we saw again various different methods over here.
Now coming to the third step here, what we are doing going to do is that we are going to probably take
this text chunks or document chunks and convert this into vectors.
Now converting, uh, you know, a text chunk or a document chunk into vectors.
There are various ways.
Okay.
Uh, in our session, what we are going to focus on, we are going to focus on three important techniques.
One is with the help of OpenAI library.
Second, uh, we will be seeing something called as llama.
Okay.
We'll discuss about what exactly is Olama.
And the third technique that we are going to use is with the help of hugging face.
Okay, now out of this, it's not like only this three embedding techniques are there.
There are a lot of different techniques that are available, but if you know all these three, that
will be more than sufficient because in terms of accuracy, with respect to the embedding techniques,
I think this three will be more than sufficient.
Yes, there are different different LM models like say you have Google Gemini.
For Google Gemini you have different embedding techniques.
If you are using other models like cloudy right cloudy three from anthropic there they have a different
embedding techniques.
But what I feel for our course this three embedding techniques will be more than sufficient.
If I probably consider with respect to OpenAI this is paid okay.
With llama you can run this embedding techniques from local.
So this will be using open source okay.
And hugging face.
Also we will try to use an open source embedding technique okay.
So we are going to come up with three videos.
First of all uh we'll discuss about OpenAI how you can probably go ahead and uh, use an open AI API
and how you can use OpenAI embedding techniques to convert text into vectors.
Then we'll go with llama and then finally hugging face.
So this is the plan of upcoming videos.
In this video, let's focus on OpenAI.
Now quickly I'm going to go to my browser over here okay.
If you go ahead and search for open AI okay API key.
Okay.
Now I hope everybody knows about OpenAI, right.
It has brought up this entire chat GPT right.
And I hope everybody is may be using ChatGPT.
Now inside that you will be having a lot of LLM models right now.
The recent model is uh, uh, GPT four.
Right.
And again, with OpenAI there is a OpenAI embedding techniques uh, also available.
Right.
So for this I will be requiring my OpenAI API key since I said that this is paid.
So if you go ahead and go to this particular URL that is platform.openai.com/playground, please go
ahead and create your account if you have not created it.
And here you just need to invest $5.
Okay.
The reason is that go ahead and check it out because the the reason why I'm showing you OpenAI embedding.
Because the accuracy is amazing in this, the accuracy with respect to implementing with the help of
APIs, any chatbot solution.
It's amazing right?
Many, many companies are using it.
So that is the reason, um, you need to probably invest $5 to check out how the API key actually works.
So in the API you will be able to see if I go to my um, settings option over here, there'll be something
called as billing.
Right.
So right now you can see I have $2.82.
And let's say if I want more I will probably keep on uploading because I have added some payment methods.
Okay.
Now what I will do.
Uh, I'll go to my dashboard and I'll go to my API key.
So here you can see I I've created two API keys.
You can go ahead and create your own API key okay.
Give any name over here and just click on Create Secret Key.
So once it is created it will give you the entire API key.
Just go ahead and copy and paste it okay I'm going to keep this as hidden because if I share here again
you'll be using it.
And again I may finish up all my balance right.
So your API key will start with s k dash okay.
And the remaining key will be available over there.
So once you create it please make sure that you copy it.
And then we will go ahead and create an environment variable for this.
Now I will go to my code okay.
Now in my environment variable.
See over here.
If you see in this folder I have this dot env file.
Now this environment variable is like I will be storing my API key over there.
I'll not show you the environment variable because I've already stored my API key over there.
So for that you just need to write open I underscore API underscore key and just go ahead and use equal
to.
And your key should be pasted in this format right.
So whatever key will be present you just go ahead and copy and paste it over there.
Right now this will be the environment key that will be available in your environment variable.
So if I go ahead and open this probably you'll be seeing this key value pair over there okay.
You just need to write open API key and this particular information.
So please do that particular step.
Uh so I'm not going to show you because it is crucial not to show you OpenAI API key.
Now if I really want to call that API key of the OpenAI eyes.
I have to load that environment variables in my coding environment.
Okay.
How do I do that.
So for this you will be requiring one library.
So first library is python dash dot env okay.
We will be using this specific library.
Why do we use it.
Because with the help of this particular library you'll be able to see that I will be importing something
like this from dot EMV import load underscore dot env.
And when I initialize this, it will basically call all your environment variables in your coding environment.
Okay.
So first of all we will go ahead and install this okay python dash dot env okay.
Now since we are also working with lang chain with OpenAI.
Right.
So there is also one more library that we will be requiring.
That is nothing but lang chain dash open I.
Okay, so this two libraries I will be requiring first is to load the environment variable.
And second is nothing but Langston OpenAI.
So I will go ahead and open my terminal over here I will clear on my screen.
So let me clear all my screen.
And here I'll say hey pip install minus our requirement dot txt.
And please make sure that it is there in the environment variable okay.
So I'll go ahead and execute it okay.
So finally here you can see that the requirement is already satisfied.
And here I've already done the installation of this.
So this both the libraries has been installed.
Now let's go back to our embedding technique.
And let's see how can we specifically work with Liang Chen.
Uh Dash OpenAI okay.
And we'll go step by step in order to understand this entire thing.
Okay, so first of all, I am importing OS and then I'm importing this.
This is for my environment variables to load all my environment variables load all the environment variables
okay.
Perfect.
Now you'll be able to see that I'm getting true.
So obviously it is saying that hey it has loaded.
Now I'll go ahead and write OS dot environ, okay.
And you know that I've created my open AI API key.
So I'll write open AI underscore API key.
And I'll set this environment variable by calling my environment variable that I've saved in my env
file.
Right.
So this will be get underscore uh get env get environment variable with open AI underscore API underscore
key.
Okay.
So this is what I'm actually able to get.
So once I execute this that basically means my environment variable has been loaded okay.
Now in order to use the embedding technique.
Now what exactly is embedding techniques.
Here we are converting our text converting text into vectors okay.
That is what we are specifically doing over here okay.
We are converting the text into vectors.
So for using the OpenAI embedding I will go to lang chain from lang chain underscore OpenAI I'm going
to import OpenAI embeddings.
Okay.
So this sorry OpenAI embeddings.
Now this is what I'm actually going to use okay.
Now in OpenAI embeddings you have different different models in order to check it out.
What I will do, I will just go back to my documentation.
And here you will be able to see all the models, what all models are specifically there and all.
So here you can see documentation okay.
So in the documentation here lot of different different functionalities available right.
And we will be also using OpenAI later on to create lot many uh, gen AI projects.
So here you can see you have models right.
With respect to model overview, you have all the specific models like GPT four or GPT four turbo,
GPT four or GPT 3.5.
If you want to work with images, that is Dall-E, then you have audio for whisper and all.
But right now I'm focusing more on embeddings, so I'll click on this.
So see a set of models that converts text into a numerical form.
Or I can also say it as vectors okay.
And that is what I have written over here.
If I want to convert the text into vectors okay.
So I'll go ahead and click on this right later on we'll be seeing all this specific models once we develop
an application.
So here you can see in embeddings you have this many number of models.
One is uh text embedding large right.
Uh text embedding three small text embedding 88 002.
So here you can see this is the most capable embedding model for both English and non-English task.
The output dimension is 3072.
Increased performance over second generation.
Other embedding model.
Most capable second generation embedding model.
So what we will do is that we'll try to just use this, uh, and we'll call this particular model for
our embedding purpose.
Okay.
So look quickly, let's go over here.
And here I have initialized OpenAI embedding.
Now I'll call OpenAI embeddings.
And let's go ahead and call my model.
The model is nothing but it will be text embedding three dash large okay.
So this is my embedding model that I'm actually going to use it.
Okay.
So I will just go ahead and execute this.
So if you go ahead and execute this embedding it will show, hey it is nothing but it is a OpenAI embeddings
okay.
And here you can see all the other information at this particular memory location and some of the default
parameters that has been assigned over here.
Now let's test this OpenAI embedding.
So let's say that I am going to consider a text.
And this text will be like this is a tutorial on, uh, embeddings on OpenAI embedding.
All right.
OpenAI embedding okay.
Now remember this OpenAI embedding is not completely for free because, uh, there will be some charges
that will be applied for this particular embedding.
Okay.
But since and it will be a minimal charge anyhow for to test our application and all.
Okay, so here what I'm actually going to do, I'll take this particular uh, uh, text and I'm going
to say hey embeddings dot.
And I'm going to probably convert this text into vectors.
So here I'm actually going to go ahead and write text okay.
So this basically becomes my text okay.
Uh, my uh vectors.
So in order to see my vectors I will just go ahead and see my query underscore result okay.
And I'll go ahead and execute it.
So let's go ahead and see this query underscore result over here.
So it is going to take this particular text.
And it has converted this okay.
Converted this into a vector.
Now let's see the dimension for this okay.
So if I go ahead and write query underscore result of zero.
So this is my first vector that is visible.
Let me do one thing.
Let me just go ahead and write length of query of results.
So here you can see 3072 dimension is there.
And that is what we got from the uh from the uh documentation.
Right.
So what it does is that it takes this entire sentence and it converts into a vector of 3072 dimension.
With all these values, you'll be able to see all these values over here.
Okay.
And that is what is all about.
Uh, you know how it probably converts a text into vectors, right?
Uh, you see, you can definitely do multiple in different, different ways also.
Okay.
And what I can also do is that I can go ahead and set my own dimensions if you want.
Let's say that I don't want to go with 3072, uh, which is the default.
I can just copy the same thing.
Okay.
Let's say I will go ahead and call this embeddings again.
Okay.
I'll paste it over here.
Now inside this embeddings what I will write I will go ahead and say, hey, this is my dimension,
okay?
This is my dimension.
And this time I will say, hey, just convert this into one zero, two four dimension.
Right.
And uh, this dimension, I will just go ahead and say embedding underscore 1024 okay.
So once I execute it you will be able to see this over here.
And uh embeddings underscore 1024.
So this is my, uh, embedding.
Okay.
Now I'll do or execute the same code over here.
So let me just go ahead and paste it over here.
Right.
So this is the sentence that I'm actually going to do.
Right.
1024.
And here let me just go ahead and write query result.
So here I'll just go ahead and write length of query result because it needs to show me in 1024 dimension.
So here you can see I'm able to get into 1024 dimension.
Right.
And if you want to see the result.
So this will be your vectors for that same text.
Okay.
So uh, this was a simple way of converting an embeddings uh, or of any text into vectors.
Now let me do one thing.
Let me just go to my one of the example over here.
Recursive character text splitter.
Okay.
So uh, let's say I will be taking one example okay.
And I'll try to convert that entire thing into.
So let's take this speech dot text okay I will copy this.
I'll put it in my embedding okay.
And here I'm going to just copy this entire thing paste it okay.
So I'll just show you for an entire document how you can actually do so.
Once this basically comes, this will basically be my document.
Okay.
Then what do we do?
Uh, after we get the document, uh, we, uh uh, here I can say loader dot load.
Then, uh, with respect to this loader dot load, uh, I will just go ahead and convert that into recursive.
I'll split that based on my recursive character splitter.
Okay.
So I'm just going to copy and paste it over here.
Okay.
I'm going to take this particular documents.
And this basically becomes my 500 documents.
So so here is my all the text that I have.
Now once I use this code over here, what I am actually doing, I'm taking this entire documents, and
I've done the recursive character text splitter and converted this into chunks of documents.
This is perfect.
Okay.
Now, uh, what I really need to do is that I need to take all these documents and convert this into
vectors, and finally store it in a vector db.
Right.
Vector store db.
Now, uh, this is really, really important.
Please focus on this.
So what I'm actually doing is that I will combine both of these steps.
First, I will take this entire chunk of documents, convert it into a vector and then store it into
the vector DB.
That is what I'm planning to do.
Okay.
now how do I actually do it?
I'll just give you one example.
Okay.
And one of the vector database that I'm actually going to use.
So here what I'm actually going to do I'm going to combine vector embedding since we have a list of
documents okay.
Since I have a list of documents I will combine vector embedding and vector store.
Okay.
So uh, let's say for this particular example I'm going to use a vector store DB which is called as
chroma.
Okay.
We'll discuss about vector store DB uh, more in depth in the upcoming videos.
First of all, I just want to give you a simple, basic example.
Let's consider that I'm going to use a vector store db which is called as vector stores, which is called
as chroma.
Okay, so this will be my vector store DB okay.
Um, again it is an open source.
You can use this okay.
Now my main aim is that with the help of vector embedding, first of all I need to convert all this
documents text into vectors.
Okay that is the first step.
And then finally store it in the chroma db.
So here I will go ahead and write db is equal to.
Let's crawl this particular chroma okay dot I will just go ahead and call from documents since we are
working with documents.
Okay.
Inside this vector store db, the first parameter that goes is nothing.
Your final documents okay.
And the second parameter is nothing but your embedding technique that you are going to use.
Let's say I want to go ahead and use this embedding technique of 1024.
Okay.
Where did it go.
Here.
Let's say I'm going to use this embedding technique.
So I'm just going to copy it I'm going to just going to paste it over here.
So this basically becomes my vector store DB okay.
So if I go ahead and see this DB now it will get executed.
And I'm getting an error.
Could not import chroma DB python package because I need to go ahead and install chroma DB because this
is a vector store DB.
So I'll go ahead and open my terminal.
And first of all I'll just go ahead and update my requirement dot txt And let's go ahead and quickly
install this chrome.
ADB is another vector store and it is in the form of library.
Uh, we will first of all go ahead and install this okay.
And then only we'll be able to use this.
But here what we are doing is that we are combining vector store DB along with vector embedding techniques.
And that is the code.
We have written it over here.
Right.
So it will take some time to do the installation.
So here again, let me repeat it.
First of all we are importing chroma.
We have already created our embedding underscore 1024.
This is nothing but this is OpenAI embeddings.
And in chroma DB what I'm actually going to do I'm going to use this function under dot from underscore
documents.
I'll give all my final documents.
And on all this final documents I need to apply this particular embedding technique.
And finally we get this particular vector store DB okay.
Again based on your internet connection it will take some time.
And again this will be a open source vector store DB which you can also use it.
Yes.
For deployment I will also be showing you I'll be using some of the vector database where I will be
deploying it in the cloud, and from that also you will be able to call it okay.
But just get an idea here.
Uh, instead of just using just a vector embedding, converting that document into text, it is also
good that you get to know about how to store it in the vector database as we go ahead.
Right now, I'm just going to discuss about one vector database called chroma.
But as we go ahead we'll be discussing more about different different vector databases.
So yes this installation has taken place.
Now I think it should get executed.
So now if I'm executing it you'll be able to see that I'll get the DB and it says that, hey, it is
a vector store and this is my chrome at this particular memory location.
Now how do I query from this particular vector the database okay.
So first of all I'll just go ahead and open this particular speech text okay.
Let's say I will be copying this entire content and I'll be searching okay.
Over here.
So here I will go ahead and write.
This will basically be my query.
Okay.
I've just used some text over there and I'll say hey use this DB and do some similarity search.
Okay.
Similarity search.
And here I'm going to basically use this particular query okay.
And uh here I'm going to save this entire in one variable which is called as retrieve.
retrieve retrieved results is equal to this one.
Okay.
And then I will go ahead and print it retrieve.
So I'll just go and write retrieved underscore result okay.
And I'll just go ahead and execute it.
So here you can see I'm able to get the response.
I'm doing a similarity search right on a specific query.
I want to retrieve some information from this particular vector store DB I've taken a text from that
and I'm searching by using the similarity search uh method.
And finally you'll be able to see I'm able to get that entire information right.
It will be all easier to conduct.
And finally you'll be seeing that entire text.
The context has got explored and it is basically search this and it is basically displaying over here.
Right.
And that is the power of vector study right now.
I hope you are able to understand how step by step we have taken a text, we have converted that into
vectors.
And finally with a document over here like this dot txt file we we used all the methods.
First of all we did the data injection.
Second we did the text splitter.
Then third uh we applied vector embedding along with vector store db.
And fourth we have also retrieved the results right.
So here you'll be able to see what did we do.
Over here.
We retrieve the results from querying vector to vector store db vector store db.
And here what technique we have applied.
It is nothing, but we have applied similarity search.
Okay, so I hope you are able to understand all these things.
Uh, now in my next video I am going to show you with some open source libraries, uh, like how you
can actually do it.
And then we'll also be discussing about hugging face.
Okay.
So yeah, uh, this was it.
I hope you are quite excited to understand this entire session.
I will see you all in the next video.
Thank you.