llama_index_selector_gh_models.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluating different models from the GitHub Models catalog with LlamaIndex and Phoenix\n",
    "\n",
    "The following notebook demonstrate how users can use multiple models from GitHub Model depending on the scenario and use the right model for the right job. In this case, we will use LLamaIndex to build a RAG system and select different models from different providers, maximizing the capabilities they have on each case."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing\n",
    "\n",
    "In this example, we will use multiple models deployed in this project, including Phi-3.5, Cohere Command R+, Cohere Embed V3, Mistral Large, and OpenAI GPT-4o. Endpoints URLs and keys are stored in the `.env` file. Please update it accordingly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install python-dotenv \n",
    "!pip install openinference-instrumentation-llama-index \n",
    "!pip install azure.ai.inference \n",
    "!pip install arize-phoenix\n",
    "!pip install llama-index-core\n",
    "!pip install llama_index.llms\n",
    "!pip install llama-index-llms-openai\n",
    "!pip install llama-index-llms-azure-inference\n",
    "!pip install llama-index-embeddings-azure-inference\n",
    "!pip install llama-index-readers-file\n",
    "!pip install spacy\n",
    "!pip install llama-index-llms-azure-openai\n",
    "\n",
    "\n",
    "\n",
    "import os\n",
    "from dotenv import load_dotenv\n",
    "from azure.ai.inference import ChatCompletionsClient\n",
    "from azure.ai.inference.models import SystemMessage, UserMessage\n",
    "from azure.core.credentials import AzureKeyCredential\n",
    "\n",
    "# load_dotenv(\".env\", override=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's configure asynchronous operations on the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "import nest_asyncio\n",
    "\n",
    "nest_asyncio.apply()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configure instrumentation\n",
    "\n",
    "We will use LlamaIndex to build a RAG system to answer different questions from the Paul Grahm dataset. To identify opportunities of improvement, we are using Phonix for tracing and monitoring. The following section configures automatic instrumentation of LlamaIndex and connects it with a Phoenix instance running locally:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import phoenix as px\n",
    "from openinference.instrumentation.llama_index import LlamaIndexInstrumentor\n",
    "from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter\n",
    "from opentelemetry.sdk.trace import SpanLimits, TracerProvider\n",
    "from opentelemetry.sdk.trace.export import SimpleSpanProcessor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "session = px.launch_app()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's configure tracing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "endpoint = session.url + \"v1/traces\"\n",
    "tracer_provider = TracerProvider(span_limits=SpanLimits(max_attributes=100_000))\n",
    "tracer_provider.add_span_processor(SimpleSpanProcessor(OTLPSpanExporter(endpoint)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's configure instrumentation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "LlamaIndexInstrumentor().instrument(tracer_provider=tracer_provider)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building a RAG system with models from the catalog\n",
    "\n",
    "Let's use the Cohere model ecosystem to implement our RAG solution. Cohere models are optimized for RAG patterns and they can work in a large range of languages, specially when using the Cohere Embed V3 Multilingual:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core import Settings\n",
    "from llama_index.core import SimpleDirectoryReader\n",
    "from llama_index.core import StorageContext\n",
    "from llama_index.core import SummaryIndex\n",
    "from llama_index.core import VectorStoreIndex\n",
    "from llama_index.core.selectors import LLMSingleSelector\n",
    "\n",
    "from llama_index.llms.azure_inference import AzureAICompletionsModel\n",
    "from llama_index.embeddings.azure_inference import AzureAIEmbeddingsModel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cohere Command R+:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "llm = AzureAICompletionsModel(\n",
    "                endpoint=\"https://models.inference.ai.azure.com\",\n",
    "                credential=os.environ[\"GITHUB_TOKEN\"],\n",
    "                temperature=0.1,\n",
    "                max_tokens=1024,\n",
    "                streaming=True,\n",
    "                model_name=\"gpt-4o-mini\",\n",
    ")\n",
    "llm._model_name = \"gpt-4o-mini\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Test the model\n",
    "response = llm.complete(\"The sky is a beautiful blue and\")\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cohere Embed V3 - Multilingual:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "embed_model = AzureAIEmbeddingsModel(\n",
    "                endpoint=\"https://models.inference.ai.azure.com\",\n",
    "                credential=os.environ[\"GITHUB_TOKEN\"],\n",
    "                model_name=\"cohere-embed-v3-english\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building the index\n",
    "\n",
    "To demostrate how to use different models, let's first create an index using Cohere models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Settings.llm = llm\n",
    "Settings.embed_model = embed_model\n",
    "Settings.chunk_size = 1024"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example, we will use the kinfey dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "documents = SimpleDirectoryReader(\"data/kinfey\").load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have documents, we create nodes by applying chunking into it as it is configured:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nodes = Settings.node_parser.get_nodes_from_documents(documents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's initialize storage context, by default it's in-memory so we don't have to worry about persisting them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nodes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "storage_context = StorageContext.from_defaults()\n",
    "storage_context.docstore.add_documents(nodes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Search tools"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our RAG system will be able to answer questions that look to summarize multiple sources of information vs a more simple retrieval strategy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Tree summarize"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The summary index is a simple data structure where nodes are stored in a sequence. During index construction, the document texts are chunked up, converted to nodes, and stored in a list. During query time, the summary index iterates through the nodes with some optional filter parameters, and synthesizes an answer from all the nodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "storage_context"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "summary_index = SummaryIndex(nodes, storage_context=storage_context)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "summarize_query_engine = summary_index.as_query_engine(\n",
    "    llm=llm,\n",
    "    response_mode=\"tree_summarize\",\n",
    "    use_async=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Vector index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`VectorStoreIndex` only stores nodes in document store if vector store does not store text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nodes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vector_index = VectorStoreIndex(nodes, storage_context=storage_context)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vector_query_engine = vector_index.as_query_engine()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vector_index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Constructing the query engine with the search tools"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core.tools import QueryEngineTool\n",
    "\n",
    "summary_tool = QueryEngineTool.from_defaults(\n",
    "    query_engine=summarize_query_engine\n",
    ")\n",
    "\n",
    "vector_tool = QueryEngineTool.from_defaults(\n",
    "    query_engine=vector_query_engine\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core.query_engine import RouterQueryEngine\n",
    "\n",
    "query_engine = RouterQueryEngine(\n",
    "    selector=LLMSingleSelector.from_defaults(),\n",
    "    query_engine_tools=[\n",
    "        summary_tool,\n",
    "        vector_tool,\n",
    "    ],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how this works:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = query_engine.query(\"Kinfey's job\")\n",
    "print(str(response))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = query_engine.query(\"what Kinfey worked on\")\n",
    "print(str(response))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using an smaller model for simpler tasks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using exactly the same class `AzureAIModelInferenceLLM` we can instantiate another model, in this case a Phi-3-mini-4K model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "slm = AzureAICompletionsModel(\n",
    "                endpoint=\"https://models.inference.ai.azure.com\",\n",
    "                credential=os.environ[\"GITHUB_TOKEN\"],\n",
    "                temperature=0.1,\n",
    "                max_tokens=1024,\n",
    "                streaming=True,\n",
    "                model_name=\"Phi-3.5-mini-instruct\",\n",
    ")\n",
    "slm._model_name = \"Phi-3.5-mini-instruct\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's configure the `RouterQueryEngine` to use Phi-3 for the routing task instead of the larger model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query_engine = RouterQueryEngine(\n",
    "    selector=LLMSingleSelector.from_defaults(llm=slm),\n",
    "    query_engine_tools=[\n",
    "        summary_tool,\n",
    "        vector_tool,\n",
    "    ],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = query_engine.query(\"What Kinfey is good at\")\n",
    "print(str(response))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build an evaluation dataset\n",
    "\n",
    "Let's build an evaluation dataset to see the effect of the change in the model. We will use another LLM to generate examples, in this case Mistral Large which is a good model for RAG:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "generator_llm = AzureAICompletionsModel(\n",
    "                endpoint=\"https://models.inference.ai.azure.com\",\n",
    "                credential=os.environ[\"GITHUB_TOKEN\"],\n",
    "                temperature=0,\n",
    "                model_name=\"Mistral-large\",\n",
    ")\n",
    "generator_llm._model_name = \"Mistral-large\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core.llama_dataset import LabelledRagDataset\n",
    "from llama_index.core.llama_dataset.generator import RagDatasetGenerator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create the generator:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_generator = RagDatasetGenerator.from_documents(\n",
    "    documents=documents,\n",
    "    llm=generator_llm,\n",
    "    num_questions_per_chunk=2,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rag_dataset = dataset_generator.generate_questions_from_nodes()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Query:\", rag_dataset[1].query)\n",
    "print(\"Context:\", rag_dataset[1].reference_contexts[0][:50], \"...\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's save the examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rag_dataset.save_json(\"evals/pg_rag_dataset.json\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can reload them as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# rag_dataset = LabelledRagDataset.from_json(\"evals/pg_rag_dataset.json\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Use evaluations for retrieval\n",
    "\n",
    "__FaithfulnessEvaluator__\n",
    "\n",
    "`FaithfulnessEvaluator` is used to measure if the response from a query engine matches any response nodes. This is useful for measuring if the response has hallucinated.\n",
    "\n",
    "__RelevancyEvaluator__\n",
    "\n",
    "`RelevancyEvaluator` is used to measure if the response and the source nodes match the query. This is useful for measuring if the query was actually answered by the response."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.llms.azure_openai import AzureOpenAI\n",
    "from llama_index.core.evaluation import BatchEvalRunner\n",
    "from llama_index.core.evaluation import RelevancyEvaluator, FaithfulnessEvaluator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case let's use a more powerful model as a judge, being GPT-4:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gpt4judge = AzureOpenAI(\n",
    "    deployment=\"gpt-4o\",\n",
    "    azure_endpoint=\"https://models.inference.ai.azure.com\",\n",
    "    api_key=os.environ[\"GITHUB_TOKEN\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Configure the evaluators with this LLM:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "relevancy_evaluator = RelevancyEvaluator(llm=gpt4judge)\n",
    "faithfulness_evaluator = FaithfulnessEvaluator(llm=gpt4judge)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create a dataset of the `query` property only."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_eval_queries = [sample.query for sample in rag_dataset[1::2]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> `rag_dataset[1::2]` retries the odd indexes only, since the dataset contains \"question 1:\" as part of the generation. It probably requires to change the generation template."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A `BatchEvalRunner` will allow us to run evalutions over all the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "runner = BatchEvalRunner(\n",
    "    {\"faithfulness\": faithfulness_evaluator, \"relevancy\": relevancy_evaluator},\n",
    "    workers=8,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compute the evaluations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_results = await runner.aevaluate_queries(query_engine, queries=batch_eval_queries)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's write the evaluation results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "eval_results_dict = {}\n",
    "eval_results_dict[\"faithfulness\"] = [\n",
    "    dict(result) for result in eval_results[\"faithfulness\"]\n",
    "]\n",
    "eval_results_dict[\"relevancy\"] = [dict(result) for result in eval_results[\"relevancy\"]]\n",
    "\n",
    "with open(\"evals/pg_rag_eval_results_phi3.json\", \"w\") as f:\n",
    "    json.dump(eval_results_dict, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compute the scores:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "faithfulness_score = sum(\n",
    "    result.passing for result in eval_results[\"faithfulness\"]\n",
    ") / len(eval_results[\"faithfulness\"])\n",
    "relevancy_score = sum(result.passing for result in eval_results[\"relevancy\"]) / len(\n",
    "    eval_results[\"relevancy\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see the results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Faithfulness Score: {faithfulness_score}\")\n",
    "print(f\"Relevancy Score: {relevancy_score}\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}