Skip to content

Commit

Permalink
docs(lab-3135): add tutorial and recipe for LLM project setup
Browse files Browse the repository at this point in the history
  • Loading branch information
@paulruelle
paulruelle committed Sep 24, 2024
1 parent edb0179 commit 2740ba0
Show file tree
Hide file tree
Showing 5 changed files with 641 additions and 1 deletion.
2 changes: 1 addition & 1 deletion docs/sdk/llm.md
View file
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
# LLM module

::: kili.presentation.client.llm.LlmClientMethods
::: kili.llm.presentation.client.llm.LlmClientMethods
334 changes: 334 additions & 0 deletions docs/sdk/tutorials/llm_project_setup.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,334 @@
<!-- FILE AUTO GENERATED BY docs/utils.py DO NOT EDIT DIRECTLY -->
<a href="https://colab.research.google.com/github/kili-technology/kili-python-sdk/blob/main/recipes/llm_project_setup.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>

# How to Set Up a Kili Project with an LLM Model and Create a Conversation

In this tutorial, we will learn how to set up a Kili project that uses a Large Language Model (LLM), associate an LLM model with the project, and create a conversation using the Kili Python SDK.

Here are the steps we will follow:

1. Creating a Kili project with a custom interface
2. Creating an LLM model
3. Associating the model with the project
4. Creating a conversation

## Creating a Kili Project with a Custom Interface

We will create a Kili project with a custom interface that includes a comparison job and a classification job. This interface will be used for labeling and comparing LLM outputs.

Here's the JSON interface we will use:


```python
interface = {
"jobs": {
"COMPARISON_JOB": {
"content": {
"options": {
"IS_MUCH_BETTER": {"children": [], "name": "Is much better", "id": "option1"},
"IS_BETTER": {"children": [], "name": "Is better", "id": "option2"},
"IS_SLIGHTLY_BETTER": {
"children": [],
"name": "Is slightly better",
"id": "option3",
},
"TIE": {"children": [], "name": "Tie", "id": "option4", "mutual": True},
},
"input": "radio",
},
"instruction": "Pick the best answer",
"mlTask": "COMPARISON",
"required": 1,
"isChild": False,
"isNew": False,
},
"CLASSIFICATION_JOB": {
"content": {
"categories": {
"BOTH_ARE_GOOD": {"children": [], "name": "Both are good", "id": "category1"},
"BOTH_ARE_BAD": {"children": [], "name": "Both are bad", "id": "category2"},
},
"input": "radio",
},
"instruction": "Overall quality",
"mlTask": "CLASSIFICATION",
"required": 0,
"isChild": False,
"isNew": False,
},
}
}
```

Now, we create the project using the `create_project` method:


```python
from kili.client import Kili

kili = Kili(
# api_endpoint="https://cloud.kili-technology.com/api/label/v2/graphql",
)
project = kili.create_project(
title="[Kili SDK Notebook]: LLM Project",
description="Project Description",
input_type="LLM_INSTR_FOLLOWING",
json_interface=interface,
)

project_id = project["id"]
print(f"Project created with ID: {project_id}")
```

Project created with ID: cm1gdz02o00hwbe0wb6ch1w23


## Creating an LLM Model

We will now create an LLM model in Kili. For this, we need to specify the model's credentials, name, and type.

**Note**: Replace `'YOUR_MODEL_API_KEY'` and `'YOUR_MODEL_ENDPOINT'` with your model's actual credentials.


```python
model_response = kili.llm.create_model(
organization_id="your_organization_id", # Replace with your organization ID
model={
"credentials": {"api_key": "YOUR_MODEL_API_KEY", "endpoint": "YOUR_MODEL_ENDPOINT"},
"name": "Your Model Name",
"type": "OPEN_AI_SDK", # Or the appropriate model type
},
)

model_id = model_response["id"]
print(f"Model created with ID: {model_id}")
```


---------------------------------------------------------------------------

TransportQueryError Traceback (most recent call last)

File ~/kili-python-sdk/src/kili/core/graphql/graphql_client.py:255, in GraphQLClient.execute(self, query, variables, **kwargs)
254 try:
--> 255 return self._execute_with_retries(document, variables, **kwargs)
257 except graphql.GraphQLError: # local validation error
258 # the local schema might be outdated
259 # we refresh the schema and retry once


File ~/kili-python-sdk/venv/lib/python3.12/site-packages/tenacity/__init__.py:336, in BaseRetrying.wraps.<locals>.wrapped_f(*args, **kw)
335 wrapped_f.statistics = copy.statistics # type: ignore[attr-defined]
--> 336 return copy(f, *args, **kw)


File ~/kili-python-sdk/venv/lib/python3.12/site-packages/tenacity/__init__.py:475, in Retrying.__call__(self, fn, *args, **kwargs)
474 while True:
--> 475 do = self.iter(retry_state=retry_state)
476 if isinstance(do, DoAttempt):


File ~/kili-python-sdk/venv/lib/python3.12/site-packages/tenacity/__init__.py:376, in BaseRetrying.iter(self, retry_state)
375 for action in self.iter_state.actions:
--> 376 result = action(retry_state)
377 return result


File ~/kili-python-sdk/venv/lib/python3.12/site-packages/tenacity/__init__.py:398, in BaseRetrying._post_retry_check_actions.<locals>.<lambda>(rs)
397 if not (self.iter_state.is_explicit_retry or self.iter_state.retry_run_result):
--> 398 self._add_action_func(lambda rs: rs.outcome.result())
399 return


File /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/_base.py:449, in Future.result(self, timeout)
448 elif self._state == FINISHED:
--> 449 return self.__get_result()
451 self._condition.wait(timeout)


File /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/_base.py:401, in Future.__get_result(self)
400 try:
--> 401 raise self._exception
402 finally:
403 # Break a reference cycle with the exception in self._exception


File ~/kili-python-sdk/venv/lib/python3.12/site-packages/tenacity/__init__.py:478, in Retrying.__call__(self, fn, *args, **kwargs)
477 try:
--> 478 result = fn(*args, **kwargs)
479 except BaseException: # noqa: B902


File ~/kili-python-sdk/src/kili/core/graphql/graphql_client.py:302, in GraphQLClient._execute_with_retries(self, document, variables, **kwargs)
277 @retry(
278 reraise=True, # re-raise the last exception
279 retry=retry_all(
(...)
300 self, document: DocumentNode, variables: Optional[Dict], **kwargs
301 ) -> Dict[str, Any]:
--> 302 return self._raw_execute(document, variables, **kwargs)


File ~/kili-python-sdk/src/kili/core/graphql/graphql_client.py:309, in GraphQLClient._raw_execute(self, document, variables, **kwargs)
308 with _execute_lock:
--> 309 return self._gql_client.execute(
310 document=document,
311 variable_values=variables,
312 extra_args={
313 "headers": {
314 **(self._gql_transport.headers or {}),
315 **LogContext(),
316 }
317 },
318 **kwargs,
319 )


File ~/kili-python-sdk/venv/lib/python3.12/site-packages/gql/client.py:484, in Client.execute(self, document, variable_values, operation_name, serialize_variables, parse_result, get_execution_result, **kwargs)
483 else: # Sync transports
--> 484 return self.execute_sync(
485 document,
486 variable_values=variable_values,
487 operation_name=operation_name,
488 serialize_variables=serialize_variables,
489 parse_result=parse_result,
490 get_execution_result=get_execution_result,
491 **kwargs,
492 )


File ~/kili-python-sdk/venv/lib/python3.12/site-packages/gql/client.py:248, in Client.execute_sync(self, document, variable_values, operation_name, serialize_variables, parse_result, get_execution_result, **kwargs)
247 with self as session:
--> 248 return session.execute(
249 document,
250 variable_values=variable_values,
251 operation_name=operation_name,
252 serialize_variables=serialize_variables,
253 parse_result=parse_result,
254 get_execution_result=get_execution_result,
255 **kwargs,
256 )


File ~/kili-python-sdk/venv/lib/python3.12/site-packages/gql/client.py:1028, in SyncClientSession.execute(self, document, variable_values, operation_name, serialize_variables, parse_result, get_execution_result, **kwargs)
1027 if result.errors:
-> 1028 raise TransportQueryError(
1029 str_first_element(result.errors),
1030 errors=result.errors,
1031 data=result.data,
1032 extensions=result.extensions,
1033 )
1035 assert (
1036 result.data is not None
1037 ), "Transport returned an ExecutionResult without data or errors"


TransportQueryError: {'message': 'insert or update on table "Model" violates foreign key constraint "Model_organizationId_fkey"', 'locations': [{'line': 2, 'column': 3}], 'path': ['createModel'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'stacktrace': ['Error', ' at Query.run (/Users/paulruelle/kili/services/label-backend-v2/node_modules/sequelize/src/dialects/postgres/query.js:76:25)', ' at /Users/paulruelle/kili/services/label-backend-v2/node_modules/sequelize/src/sequelize.js:650:28', ' at async PostgresQueryInterface.insert (/Users/paulruelle/kili/services/label-backend-v2/node_modules/sequelize/src/dialects/abstract/query-interface.js:795:21)', ' at async Model.save (/Users/paulruelle/kili/services/label-backend-v2/node_modules/sequelize/src/model.js:4154:35)', ' at async Function.create (/Users/paulruelle/kili/services/label-backend-v2/node_modules/sequelize/src/model.js:2305:12)', ' at async SequelizeRepositoryModel.createModel (/Users/paulruelle/kili/services/label-backend-v2/src/domains/llm/repositories/sequelize/SequelizeRepositoryModel.ts:44:26)', ' at async UseCasesModel.createModel (/Users/paulruelle/kili/services/label-backend-v2/src/domains/llm/useCases/UseCasesModel.ts:46:26)', ' at async Object.createModel (/Users/paulruelle/kili/services/label-backend-v2/src/domains/llm/presentation/resolvers/model/mutations.ts:15:17)']}}


The above exception was the direct cause of the following exception:


GraphQLError Traceback (most recent call last)

Cell In[6], line 1
----> 1 model_response = kili.llm.create_model(
2 organization_id="your_organization_id", # Replace with your organization ID
3 model={
4 "credentials": {
5 'api_key': "YOUR_MODEL_API_KEY",
6 'endpoint': "YOUR_MODEL_ENDPOINT"
7 },
8 "name": "Your Model Name",
9 "type": "OPEN_AI_SDK" # Or the appropriate model type
10 }
11 )
13 model_id = model_response["id"]
14 print(f"Model created with ID: {model_id}")


File ~/kili-python-sdk/src/kili/utils/logcontext.py:60, in log_call.<locals>.wrapper(*args, **kwargs)
56 context["kili-client-call-time"] = (
57 datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
58 )
59 context["kili-client-call-uuid"] = str(uuid.uuid4())
---> 60 return func(*args, **kwargs)


File ~/kili-python-sdk/src/kili/llm/presentation/client/llm.py:158, in LlmClientMethods.create_model(self, organization_id, model)
150 raise ValueError(f"Unsupported model type: {model['type']}")
152 model_input = ModelToCreateInput(
153 credentials=credentials,
154 name=model["name"],
155 type=model_type,
156 organization_id=organization_id,
157 )
--> 158 return self.kili_api_gateway.create_model(model=model_input)


File ~/kili-python-sdk/src/kili/adapters/kili_api_gateway/llm/operations_mixin.py:83, in ModelConfigurationOperationMixin.create_model(self, model)
81 fragment = fragment_builder(["id"])
82 mutation = get_create_model_mutation(fragment)
---> 83 result = self.graphql_client.execute(mutation, payload)
84 return result["createModel"]


File ~/kili-python-sdk/src/kili/core/graphql/graphql_client.py:275, in GraphQLClient.execute(self, query, variables, **kwargs)
271 raise kili.exceptions.GraphQLError(error=err.errors) from err
273 except exceptions.TransportQueryError as err: # remove validation error
274 # the server refused the query after some retries, we crash
--> 275 raise kili.exceptions.GraphQLError(error=err.errors) from err


GraphQLError: GraphQL error: "insert or update on table "Model" violates foreign key constraint "Model_organizationId_fkey""


## Associating the Model with the Project

Next, we will associate the created model with our project by creating project models with different configurations.

We will create two project models with different temperature settings.


```python
# First project model with a fixed temperature
first_project_model = kili.llm.create_project_model(
project_id=project_id,
model_id=model_id,
configuration={
"model": "Your Model Name",
"temperature": 0.5,
},
)

first_project_model_id = first_project_model["id"]
print(f"First project model created with ID: {first_project_model_id}")

# Second project model with a temperature range
second_project_model = kili.llm.create_project_model(
project_id=project_id,
model_id=model_id,
configuration={
"model": "Your Model Name",
"temperature": {"min": 0.2, "max": 0.8},
},
)

second_project_model_id = second_project_model["id"]
print(f"Second project model created with ID: {second_project_model_id}")
```

## Creating a Conversation

Now, we will create a conversation in the project using a prompt.


```python
conversation = kili.llm.create_conversation(project_id=project_id, prompt="Hello, world!")

print("Conversation created.")
```

## Summary

In this tutorial, we've successfully set up a Kili project with a custom interface, created an LLM model, associated the model with the project by creating project models, and created a conversation using the Kili Python SDK.
12 changes: 12 additions & 0 deletions docs/tutorials.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,18 @@ For a more specific use case, follow [this tutorial](https://python-sdk-docs.kil

Webhooks are really similar to plugins, except they are self-hosted, and require a web service deployed at your end, callable by Kili. To learn how to use webhooks, follow [this tutorial](https://python-sdk-docs.kili-technology.com/latest/sdk/tutorials/webhooks_example/).

## LLM

[This tutorial](https://python-sdk-docs.kili-technology.com/latest/sdk/tutorials/llm_project_setup/) will show you how to set up a Kili project that uses a Large Language Model (LLM), create and associate the LLM model with the project, and initiate a conversation using the Kili Python SDK.









## Integrations

[This tutorial](https://python-sdk-docs.kili-technology.com/latest/sdk/tutorials/vertex_ai_automl_od/) will show you how train an object detection model with Vertex AI AutoML and Kili for faster annotation
Expand Down
Loading

0 comments on commit 2740ba0

Please sign in to comment.