unitxt
unitxt copied to clipboard
IBM
🦄 Unitxt: a python library for getting data fired up and set for training and evaluation
https://github.com/IBM/unitxt/blob/231fd293f53baac6be94133c606d7ddaf66eacd4/src/unitxt/artifact.py#L50 I’d like to propose a minor optimization related to key intersection operations between two dictionaries. Original Code： `keys_in_both = dict1.keys() & dict2.keys()` Suggested Replacement： `keys_in_both = set(dict1).intersection(dict2)` Although both...
We encountered an issue where we are getting negative values when evaluating `metrics.rag.context_relevance.sentence_bert_mini_lm`. The expectation here is to get a value between a positive value between 0 and 1. This...
Hi, I am looking athe Multiple-choice QA example provided [here](https://github.com/IBM/unitxt/blob/main/examples/multiple_choice_qa_evaluation.py) . I wanted to know how the accuracy is calculated. Do we 1) ask the model to generate the answer,...
print("Instance Results:") print(results.instance_scores.summary) ``` text_a text_a_type \ 0 The drain is clogged with hair. It has to be cleaned. premise 1 Jane knocked on Susan's door but she did not...
In my view, there is a need for a common utility in unitxt, that allows specifying multiple environment variable names and ensuring either only one is set, or all the...
LiteLLMInferenceEngine, as it is implemented on unitxt, creates an event loop. FastAPI/Jupyter Notebooks also create one. `asyncio` doesn't allow nested loop, so we need to rely on `nest_asyncio` to handle...
1) Normal small model on a benchmark 2) Big models - 2 GPU 3) Lora or similar models 4) Run small models with long context like open rag on both...
