unitxt
unitxt copied to clipboard
IBM
🦄 Unitxt: a python library for getting data fired up and set for training and evaluation
In complex tasks, like RAG, there prediction_type is a Dictionary with a fixed fields. Today, it is modeled as prediction_type: "Dict" We want to be able to write: “prediction_type”: “Dict[{“answer”:...
Today tasks have "inputs" and "outputs". This is confusing many users - who expect output fields to relate to the model output. Actually it should be named: "input_fields" - fields...
Unitxt has two concepts: (Raw/Texutal) Predictions and Processed Predictions (after applying post processing operators) Today, the raw predictions which are added to the dataset during evaluation are transformed into the...
Localized an temporarily disabled test in test_loaders. @unittest.skip("Currently this fails from datasets 2.20") def test_load_from_HF_multiple_innvocation_with_filter(self): loader = LoadHF( path="CohereForAI/aya_evaluation_suite", name="aya_human_annotated", filtering_lambda='lambda instance: instance["language"]=="eng"', ) ms = loader.process() dataset = ms.to_dataset()...
These splitters in unitxt/prepare/splitters/missing_split.py are do not have informative names, so when people read them in a file , it's not clear what's happening. I think it's just better to...
https://unitxt.readthedocs.io/en/latest/docs/operators.html Some issues with this page: 1.Not all operators appear in the page (e.g. Get) , probably because they come from other files 2. It's very hard to look for...
`Artifact.from_dict` will try to convert any dictionary to artifact even when the dictionary is a mapping of columns from datasets. **There cannot be a column named `type`:** ```python import tempfile...
Add to the recipe an option to use the templates list from the task and use the first one by default. This simplify the construction of new cards assuming they...
Descriptions for tasks can allow easier synthetic template generation, as well as seamless integration with libraries like instruct lab and DSPy.