feat(LAB-3135): add kili.llm.create_conversation (#1775)

Co-authored-by: paulruelle <[email protected]>

docs/sdk/llm.md

@@ -0,0 +1,3 @@
+# LLM module
+
+::: kili.llm.presentation.client.llm.LlmClientMethods

docs/tutorials.md

@@ -71,6 +71,11 @@ For a more specific use case, follow [this tutorial](https://python-sdk-docs.kil
 
 Webhooks are really similar to plugins, except they are self-hosted, and require a web service deployed at your end, callable by Kili. To learn how to use webhooks, follow [this tutorial](https://python-sdk-docs.kili-technology.com/latest/sdk/tutorials/webhooks_example/).
 
+## LLM
+
+[This tutorial](https://python-sdk-docs.kili-technology.com/latest/sdk/tutorials/llm_project_setup/) will show you how to set up a Kili project that uses a Large Language Model (LLM), create and associate the LLM model with the project, and initiate a conversation using the Kili Python SDK.
+
+
 ## Integrations
 
 [This tutorial](https://python-sdk-docs.kili-technology.com/latest/sdk/tutorials/vertex_ai_automl_od/) will show you how train an object detection model with Vertex AI AutoML and Kili for faster annotation

mkdocs.yml

@@ -20,6 +20,7 @@ nav:
           - Label: sdk/label.md
           - Label Utils: sdk/label_utils.md
           - Label Parsing: sdk/label_parsing.md
+          - LLM: sdk/llm.md
           - Notification: sdk/notification.md
           - Organization: sdk/organization.md
           - Plugins: sdk/plugins.md
@@ -57,6 +58,7 @@ nav:
       - Exporting Project Data:
           - Exporting a Project: sdk/tutorials/export_a_kili_project.md
           - Parsing Labels: sdk/tutorials/label_parsing.md
+      - LLM Projects: sdk/tutorials/llm_project_setup.md
       - Setting Up Plugins:
           - Developing Plugins: sdk/tutorials/plugins_development.md
           - Plugin Example - Programmatic QA: sdk/tutorials/plugins_example.md

recipes/llm_project_setup.ipynb

@@ -0,0 +1,280 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a href=\"https://colab.research.google.com/github/kili-technology/kili-python-sdk/blob/main/recipes/llm_project_setup.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# How to Set Up a Kili Project with a LLM Model and Create a Conversation"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In this tutorial, you'll learn how to set up a project in Kili Technology that integrates a Large Language Model (LLM), associate the LLM with your project, and create a conversation using the Kili Python SDK. By the end of this guide, you'll have a functional project ready to collect and label LLM outputs for comparison and evaluation.\n",
+    "\n",
+    "\n",
+    "Here are the steps we will follow:\n",
+    "\n",
+    "1. Creating a Kili project with a custom interface\n",
+    "2. Creating an LLM model\n",
+    "3. Associating the model with the project\n",
+    "4. Creating a conversation"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Creating a Kili Project with a Custom Interface"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We will create a Kili project with a custom interface that includes a comparison job and a classification job. This interface will be used for labeling and comparing LLM outputs.\n",
+    "\n",
+    "Here's the JSON interface we will use:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "interface = {\n",
+    "    \"jobs\": {\n",
+    "        \"COMPARISON_JOB\": {\n",
+    "            \"content\": {\n",
+    "                \"options\": {\n",
+    "                    \"IS_MUCH_BETTER\": {\"children\": [], \"name\": \"Is much better\", \"id\": \"option1\"},\n",
+    "                    \"IS_BETTER\": {\"children\": [], \"name\": \"Is better\", \"id\": \"option2\"},\n",
+    "                    \"IS_SLIGHTLY_BETTER\": {\n",
+    "                        \"children\": [],\n",
+    "                        \"name\": \"Is slightly better\",\n",
+    "                        \"id\": \"option3\",\n",
+    "                    },\n",
+    "                    \"TIE\": {\"children\": [], \"name\": \"Tie\", \"id\": \"option4\", \"mutual\": True},\n",
+    "                },\n",
+    "                \"input\": \"radio\",\n",
+    "            },\n",
+    "            \"instruction\": \"Pick the best answer\",\n",
+    "            \"mlTask\": \"COMPARISON\",\n",
+    "            \"required\": 1,\n",
+    "            \"isChild\": False,\n",
+    "            \"isNew\": False,\n",
+    "        },\n",
+    "        \"CLASSIFICATION_JOB\": {\n",
+    "            \"content\": {\n",
+    "                \"categories\": {\n",
+    "                    \"BOTH_ARE_GOOD\": {\"children\": [], \"name\": \"Both are good\", \"id\": \"category1\"},\n",
+    "                    \"BOTH_ARE_BAD\": {\"children\": [], \"name\": \"Both are bad\", \"id\": \"category2\"},\n",
+    "                },\n",
+    "                \"input\": \"radio\",\n",
+    "            },\n",
+    "            \"instruction\": \"Overall quality\",\n",
+    "            \"mlTask\": \"CLASSIFICATION\",\n",
+    "            \"required\": 0,\n",
+    "            \"isChild\": False,\n",
+    "            \"isNew\": False,\n",
+    "        },\n",
+    "    }\n",
+    "}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now, we create the project using the `create_project` method, with type `LLM_INSTR_FOLLOWING`:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from kili.client import Kili\n",
+    "\n",
+    "kili = Kili(\n",
+    "    # api_endpoint=\"https://cloud.kili-technology.com/api/label/v2/graphql\",\n",
+    ")\n",
+    "project = kili.create_project(\n",
+    "    title=\"[Kili SDK Notebook]: LLM Project\",\n",
+    "    description=\"Project Description\",\n",
+    "    input_type=\"LLM_INSTR_FOLLOWING\",\n",
+    "    json_interface=interface,\n",
+    ")\n",
+    "project_id = project[\"id\"]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Creating an LLM Model"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We will now create an LLM model in Kili, by specifying the model's credentials and connector type. In this example, we will use the OpenAI SDK as the connector type.\n",
+    "\n",
+    "**Note**: Replace `api_key` and `endpoint` with your model's actual credentials."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "model_response = kili.llm.create_model(\n",
+    "    organization_id=\"<YOUR_ORGANIZATION_ID>\",\n",
+    "    model={\n",
+    "        \"credentials\": {\n",
+    "            \"api_key\": \"<YOUR_OPEN_AI_API_KEY>\",\n",
+    "            \"endpoint\": \"https://api.openai.com/v1/\",\n",
+    "        },\n",
+    "        \"name\": \"My Model\",\n",
+    "        \"type\": \"OPEN_AI_SDK\",\n",
+    "    },\n",
+    ")\n",
+    "\n",
+    "model_id = model_response[\"id\"]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You can now see the model integration by clicking **Manage organization** :\n",
+    "\n",
+    "![Model Integration](./img/llm_models.png)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Associating the Model with the Project"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Next, we will associate the created model with our project by creating project models with different configurations. Each time you create a prompt, two models will be chosen from the project models in the project \n",
+    "\n",
+    "In this example, we compare **GPT 4o** and **GPT 4o Mini**, with different temperature settings :"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# First project model with a fixed temperature\n",
+    "first_project_model = kili.llm.create_project_model(\n",
+    "    project_id=project_id,\n",
+    "    model_id=model_id,\n",
+    "    configuration={\n",
+    "        \"model\": \"gpt-4o\",\n",
+    "        \"temperature\": 0.5,\n",
+    "    },\n",
+    ")\n",
+    "\n",
+    "# Second project model with a temperature range\n",
+    "second_project_model = kili.llm.create_project_model(\n",
+    "    project_id=project_id,\n",
+    "    model_id=model_id,\n",
+    "    configuration={\n",
+    "        \"model\": \"gpt-4o-mini\",\n",
+    "        \"temperature\": {\"min\": 0.2, \"max\": 0.8},\n",
+    "    },\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You can now see the project models in the project settings :\n",
+    "\n",
+    "![Project Models](./img/llm_project_models.png)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Creating a Conversation"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now, we'll generate a conversation by providing a prompt.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "conversation = kili.llm.create_conversation(\n",
+    "    project_id=project_id, prompt=\"Give me Schrödinger equation.\"\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "It will add an asset to your project, and you'll be ready to start labeling the conversation :\n",
+    "\n",
+    "![Conversation](./img/llm_conversation.png)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Summary"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In this tutorial, we've:\n",
+    "\n",
+    "- **Created a Kili project** with a custom interface for LLM output comparison.\n",
+    "- **Registered an LLM model** in Kili with the necessary credentials.\n",
+    "- **Associated the model** with the project by creating project models with different configurations.\n",
+    "- **Generated a conversation** using a prompt, adding it to the project for labeling.\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "name": "python"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

src/kili/adapters/kili_api_gateway/llm/mappers.py

@@ -109,3 +109,30 @@ def map_delete_project_model_input(project_model_id: str) -> Dict:
     return {
         "deleteProjectModelId": project_model_id,
     }
+
+
+def map_create_llm_asset_input(data: Dict) -> Dict:
+    """Map the input for the createLLMAsset mutation."""
+    result = {
+        "authorId": data["author_id"],
+    }
+    if "status" in data:
+        result["status"] = data["status"]
+    if "label_type" in data:
+        result["labelType"] = data["label_type"]
+    return result
+
+
+def map_project_where(project_id: str) -> Dict:
+    """Map the 'where' parameter for mutations that require a ProjectWhere."""
+    return {"id": project_id}
+
+
+def map_create_chat_item_input(label_id: str, prompt: str) -> Dict:
+    """Map the input for the createChatItem mutation."""
+    return {"content": prompt, "role": "USER", "labelId": label_id}
+
+
+def map_asset_where(asset_id: str) -> Dict:
+    """Map the 'where' parameter for the createChatItem mutation."""
+    return {"id": asset_id}

src/kili/adapters/kili_api_gateway/llm/operations.py

@@ -94,3 +94,25 @@ def get_project_models_query(fragment: str) -> str:
           }}
         }}
         """
+
+
+def get_create_llm_asset_mutation(fragment: str) -> str:
+    """Return the GraphQL createLLMAsset mutation."""
+    return f"""
+        mutation CreateLLMAsset($where: ProjectWhere!, $data: CreateLLMAssetData!) {{
+            createLLMAsset(where: $where, data: $data) {{
+                {fragment}
+            }}
+        }}
+    """
+
+
+def get_create_chat_item_mutation(fragment: str) -> str:
+    """Return the GraphQL createChatItem mutation."""
+    return f"""
+        mutation CreateChatItem($data: CreateChatItemData!, $where: AssetWhere!) {{
+            createChatItem(data: $data, where: $where) {{
+                {fragment}
+            }}
+        }}
+    """