feat(LAB-3244): on LLM dynamic projects export annotations at convers… #1812
Conversation
FannyGaudin
force-pushed
the
feature/lab-3244-aau-on-llm-dynamic-projects-i-export-annotations-at
branch
2 times, most recently
from
a39328b
to
272a072
Compare
| @@ -192,8 +195,8 @@ def _format_comparison_annotation(annotation, completions, job, obfuscated_model | |||
|
|
|||
| def _format_json_response( | |||
| jobs_config: Dict, annotations: List[Dict], completions: List[Dict], obfuscated_models: Dict | |||
| ) -> Dict[str, Union[str, List[str]]]: | |||
| result = {} | |||
| ) -> Dict[str, Dict]: | |||
There was a problem hiding this comment.
Don't you loose some of the typing here ?
There was a problem hiding this comment.
Yes I will update the type
| "label": formatted_response["label"], | ||
| } | ||
| if step == total_rounds - 1 and formatted_response["conversation_label"]: | ||
| label_data["conversation_label"] = formatted_response["conversation_label"] |
There was a problem hiding this comment.
I discussed with Stéphanie about that, if we want to differentiate the type of the annotation at the label level, we need to do it for every levels ; for example :
...
"label": {
"completion": {
"RANKING": [
"B_2"
]
},
"conversation": {
"TRANSCRIPTION_1": "some transcription at asset level"
},
"chat_item": {
"012399314932R4": {
"TRANSCRIPTION_2": "some transcription at chat item level"
}
}
}
There was a problem hiding this comment.
I'm not really ok with this format, as we duplicate some information already present in the json_interface. We can discuss it with the team if you want
There was a problem hiding this comment.
And doing this change mean applying it also on static export for consistency, creating a breaking change in our export format
There was a problem hiding this comment.
No problem but could you tell me what format you have in mind ?
FannyGaudin
force-pushed
the
feature/lab-3244-aau-on-llm-dynamic-projects-i-export-annotations-at
branch
from
272a072
to
5411753
Compare
…ation level