labs/03-orchestration/02-Embeddings/embeddings.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 02 Embeddings\n",
    "\n",
    "In this lab, we'll explore how we can bring our own data into the models used by Azure OpenAI.\n",
    "\n",
    "We'll start as usual by defining our authenticating with Azure AD to obtain a token and providing the endpoint details, specifying the model deployment we want to use and then we'll initiate a connection to the Azure OpenAI service.\n",
    "\n",
    "**NOTE**: As with previous labs, we'll use the values from the `.env` file in the root of this repository."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from langchain.llms import AzureOpenAI\n",
    "from dotenv import load_dotenv\n",
    "from azure.identity import DeviceCodeCredential\n",
    "\n",
    "# Load environment variables\n",
    "if load_dotenv(override=True):\n",
    "    print(\"Found OpenAPI Base Endpoint: \" + os.getenv(\"OPENAI_API_BASE\"))\n",
    "else: \n",
    "    print(\"No file .env found\")\n",
    "\n",
    "openai_api_type = \"azure_ad\"\n",
    "openai_api_base = os.getenv(\"OPENAI_API_BASE\")\n",
    "openai_api_version = os.getenv(\"OPENAI_API_VERSION\")\n",
    "deployment_name = os.getenv(\"AZURE_OPENAI_COMPLETION_DEPLOYMENT_NAME\")\n",
    "embedding_name = os.getenv(\"AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME\")\n",
    "\n",
    "default_credential = DeviceCodeCredential(tenant_id=os.getenv(\"AZURE_TENANT_ID\"))\n",
    "token = default_credential.get_token(\"https://cognitiveservices.azure.com/.default\")\n",
    "openai_api_key = token.token\n",
    "print (\"Using Azure AD authentication\")\n",
    "\n",
    "# Create an instance of Azure OpenAI\n",
    "llm = AzureOpenAI(\n",
    "    openai_api_type = openai_api_type,\n",
    "    openai_api_version = openai_api_version,\n",
    "    openai_api_base = openai_api_base,\n",
    "    openai_api_key = openai_api_key,\n",
    "    deployment_name = deployment_name,\n",
    "    temperature = 0\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's begin by asking the AI a simple question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Call the API\n",
    "r = llm(\"Tell me about the latest Ant-Man movie. When was it released? What is it about?\")\n",
    "\n",
    "# Print the response\n",
    "print(r)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do you notice about the response?\n",
    "\n",
    "The AI thinks the latest \"Ant-Man\" movie was \"Ant-Man and the Wasp\" and it was released in July 2018. \n",
    "\n",
    "OpenAI models are trained on a large set of data, but that happened at a specific point in time depending on the model. So, many of the models have no information about events that took place in recent months or years.\n",
    "\n",
    "To help the AI out, we can provide additional information. This is the same process you would follow if you want the AI to work with your own company data. The AI won't know about information that isn't publically available, so if you want the AI to work with that information, then you'll need to get that information into the model.\n",
    "\n",
    "The thing is, you can't actually do that. The models are pre-trained, so the only way to get more information in is to retrain the model, which is an expensive and time consuming process.\n",
    "\n",
    "However, there *are* ways to get the AI models to work with new data. The most popular of these methods is to use *embeddings*, which we'll explore in the next sections."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bring Your Own Data\n",
    "\n",
    "Langchain provides a number of useful tools, which include tools to simplify the process of working with external documents. Below, we'll use the `DirectoryLoader` which can read multiple files from a directory and the `UnstructuredMarkdownLoader` which can process files in Markdown format. We'll use these to process a bunch of markdown formatted files that contain details of movies that were released in the year 2023."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.document_loaders import DirectoryLoader, UnstructuredMarkdownLoader\n",
    "\n",
    "data_dir = \"data/movies\"\n",
    "\n",
    "documents = DirectoryLoader(path=data_dir, glob=\"*.md\", show_progress=True, loader_cls=UnstructuredMarkdownLoader).load()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have a `documents` object which contains all of the information from our markdown documents about movies.\n",
    "\n",
    "We can use the `question_answering` chain to provide the AI with access to our documents and then ask the same question about Ant-Man movies again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Question answering chain\n",
    "from langchain.chains.question_answering import load_qa_chain\n",
    "\n",
    "# Prepare the chain and the query\n",
    "chain = load_qa_chain(llm)\n",
    "query = \"Tell me about the latest Ant Man movie. When was it released? What is it about?\"\n",
    "\n",
    "chain.run(input_documents=documents, question=query)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great! The model now knows about the latest Ant-Man movie.\n",
    "\n",
    "However, there's something lurking! Let's take a look at what happened behind the scenes.\n",
    "\n",
    "We'll do two things here. First we'll add the `verbose=True` parameter to the chain, and we'll wrap the chain execution in a callback, which will allow us to capture the number of tokens consumed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Support for callbacks\n",
    "from langchain.callbacks import get_openai_callback\n",
    "\n",
    "# Prepare the chain and the query\n",
    "chain = load_qa_chain(llm, verbose=True)\n",
    "query = \"Tell me about the latest Ant Man movie. When was it released? What is it about?\"\n",
    "\n",
    "# Run the chain, using the callback to capture the number of tokens used\n",
    "with get_openai_callback() as callback:\n",
    "    chain.run(input_documents=documents, question=query)\n",
    "    total_tokens = callback.total_tokens\n",
    "\n",
    "print(f\"Total tokens used: {total_tokens}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the output from the last code section, you should see a lot of information. At the end, you should see a count of the number of tokens used. You might be surprised to see that the query used around 2,900 tokens! That's a lot of tokens.\n",
    "\n",
    "With the verbose option enabled, the rest of the output shows the prompt that was constructed for the query. If you scroll back through the output, you'll see that the prompt included **all** of the information from our documents, so this is why the query used so many tokens.\n",
    "\n",
    "As we've discussed previously, AI models have a maximum number of tokens you can use and a charging model based on the number of tokens consumed. In this example, the documents are relatively small in size and there's only 20 of them, so clearly this is not going to scale when we want to work with larger documents and more of them."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Embeddings\n",
    "\n",
    "The solution to working with large amounts of external information is to use *embeddings*. OpenAI provide embedding models which allow human readable information to be analysed for meaning and intent. The output from an embedding model is data in a numeric format, known as *vectors*. These allow computers to group pieces of similar information together. The vectors are then kept in a *vector store*. When you want to ask a question, an embedding model is again used to convert the query text into vectors and the vector data that represents your query can then be searched in the vector store. Any similar vectors that are found in the database are likely to be a good response to your query.\n",
    "\n",
    "To prevent overloading a prompt with a large number of tokens, instead of sending all of our documents to the AI, we can perform a vector search first to narrow down to a set of interesting results, and then use that smaller subset of information as part of a prompt."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's walk through the process of using embeddings to give the AI some details about our movies. We'll start by initiating an instance of an embeddings model. You'll notice this is similar to when we initialise one of our model deployments to run a query, but in this case we specify and embedding model. Typically the embedding model used is `text-embedding-ada-002`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.embeddings import OpenAIEmbeddings\n",
    "\n",
    "embeddings_model = OpenAIEmbeddings(\n",
    "    openai_api_type = openai_api_type,\n",
    "    openai_api_version = openai_api_version,\n",
    "    openai_api_base = openai_api_base,\n",
    "    openai_api_key = openai_api_key,\n",
    "    deployment = embedding_name,\n",
    "    chunk_size = 1\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE:** The `chunk_size = 1` parameter is used to workaround a temporary limitation in the Azure OpenAI API which only allows one embedding to be processed at a time with each call to the API.\n",
    "\n",
    "Now that we've initialised a model to create embeddings, let's go ahead and embed some documents.\n",
    "\n",
    "As we did in the previous example, we'll use Langchain's builit in loaders to read the documents from a directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "documents = DirectoryLoader(path=data_dir, glob=\"*.md\", show_progress=True, loader_cls=UnstructuredMarkdownLoader).load()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next step is to use a *splitter*. A splitter enables us to break up larger documents into chunks, so that we don't risk hitting the token limit when submitting our data to the embedding model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.text_splitter import CharacterTextSplitter\n",
    "text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
    "document_chunks = text_splitter.split_documents(documents)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next stage is to convert the chunks of split documents into vectors which we do by passing the data through an embedding model. The resultant vectors are then stored in a vector database. In this example, we're using the **Qdrant** (pronounced 'quadrant') database. We initialise it using the `location=\":memory:\"` option, so that the database will be stored in memory rather than persisted to disk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.vectorstores import Qdrant\n",
    "\n",
    "qdrant = Qdrant.from_documents(\n",
    "    document_chunks,\n",
    "    embeddings_model,\n",
    "    location=\":memory:\",\n",
    "    collection_name=\"movies\",\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above code segment handles the process of initialising the Qdrant database, passing our documents through the embedding model and storing the resulting vectors in the database.\n",
    "\n",
    "Next, we define a *retriever*. In Langchain, retrievers are an interface that allow results to be returned from vector stores. So, we establish a retriever for our Qdrant database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "retriever = qdrant.as_retriever()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we define a `RetrievalQA` chain. This chain brings handles the process of answering a question by performing the search on the vector store, then taking the results of that search and passing them to our AI model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.chains import RetrievalQA\n",
    "qa = RetrievalQA.from_chain_type(llm=llm, chain_type=\"stuff\", retriever=retriever)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we'll run our query again. However, we'll make one small change.\n",
    "\n",
    "You may be thinking that it's not surprising that the AI now knows about the latest Ant-Man movie, because we told it about the latest Ant-Man movie! So, let's try and show that the AI is actually doing some work here, after all it is a reasoning engine.\n",
    "\n",
    "If you're not a fan of these movies, Ant-Man originates from Marvel comic books. And the collection of movies that originate from Marvel comic books are said to be part of the Marvel Cinematic Universe, sometimes referred to as the MCU. We haven't mentioned Marvel or MCU in the data we've provided, so if modify the query slightly and ask the AI about the MCU instead of specifically about Ant-Man, let's see if it can use it's models to figure out what we mean."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"Tell me about the latest MCU movie. When was it released? What is it about?\"\n",
    "qa.run(query)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If all went well, the AI should have responded that the latest MCU movie is Ant-Man and the Wasp: Quantumania which was released in February 2023.\n",
    "\n",
    "So, we're getting the response we expected, but let's check in on one of the reasons why we've done all of this. Has the number of tokens used been reduced?  Let's use the same technique as before and employ a callback to find out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with get_openai_callback() as callback:\n",
    "    qa.run(query)\n",
    "    total_tokens = callback.total_tokens\n",
    "\n",
    "print(f\"Total tokens used: {total_tokens}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The exact number of tokens used may vary, but it should be clear that this query now uses far fewer tokens than our original query, typically around 2,000 fewer."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "AI Orchestrators like Langchain and Semantic Kernel can help simplify the process of embedding, vectorization and search. In the preceding section, we stepped through the process of document splitting, embedding, vectorisation, storing vectors in a database and creating a retriever. In the next section, we use Langchain's document loader as we did previously to load and process our Markdown formatted documents, but this time we use a `VectorstoreIndexCreator` which you can see only requires a couple of parameters - the embedding model that we want to use and the source data (`loader`) to use. However, behind the scenes, the `VectorstoreIndexCreator` is carrying out all of the steps we did previously."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.indexes import VectorstoreIndexCreator\n",
    "\n",
    "loader = DirectoryLoader(path=data_dir, glob=\"*.md\", show_progress=True, loader_cls=UnstructuredMarkdownLoader)\n",
    "\n",
    "index = VectorstoreIndexCreator(\n",
    "    embedding=embeddings_model\n",
    "    ).from_loaders([loader])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, to run a query against our data, we just need to specify the prompt and then call the index we've created above and pass in the model (`llm`) we want to use and the question we want to ask."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"Tell me about the latest Ant Man movie. When was it released? What is it about?\"\n",
    "index.query(llm=llm, question=query)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see this is a really simple way to implement embeddings and vectors as part of an AI application. It's great for getting up and running quickly.\n",
    "\n",
    "We can use the callback method again to confirm that we're still seeing a reduced number of tokens being consumed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run the chain, using the callback to capture the number of tokens used\n",
    "with get_openai_callback() as callback:\n",
    "    index.query(llm=llm, question=query)\n",
    "    total_tokens = callback.total_tokens\n",
    "\n",
    "print(f\"Total tokens used: {total_tokens}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next Section\n",
    "\n",
    "📣 [Qdrant](../03-Qdrant/qdrant.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}