unitxt
unitxt copied to clipboard
IBM
🦄 Unitxt: a python library for getting data fired up and set for training and evaluation
Fusion classes were suppose to add field named "group" to every instance of the fusion streams, stating the name of its origin. In turn in metric computation time the metric...
Currently the demos for every instance are sampled based on the inital seed and its order in the stream, instaed of based solely on the content of the instance. This...
DiverseLabelsSampler is used to select few shot examples. 1) The name is not clear. It should be something DiverseDemosSampler. 2) DiverseLabelsSampler requires adding a choices field - which requires adding...
When loading a metric with overwrite args (for example: `metrics.char_edit_distance[reference_field=original_text]`), this is not reflected in the result dict returned from the metric, which will have the original metric name. This...
data_dir=“tuning-data-cleared/ceramic/mixtures_02.26.2024/code/dolphin_coder”, data_files={“train”: “train/part0.jsonl”, “test”: “val/part0.jsonl”}, This causes failures. Possible solution: 1. Raise an error if '/' in data_files, but allow fusing of datasource (see #707) 2. Handle '/' in paths....
Bug in the MMLU dataset production process: ``` from unitxt.templates import MultipleChoiceTemplate # self.template = MultipleChoiceTemplate(type='multiple_choice_template', artifact_identifier='template_0', _requirements_list=[], caching=None, apply_to_streams=None, dont_apply_to_streams=None, skip_rendered_instance=True, postprocessors=['processors.first_character'], instruction='', target_prefix='', title_fields=[], input_format='Question: [question] Choices: [choices]...
Currently, we use pd.read_csv() for loading csv file. It causes that empty cells in a csv file are transformed to nan (which is not suited) The fix is simple: pd.read_csv('test.csv',...
that allows using output of other cards in new cards
Today, tasks fields have no types ``` FormTask( inputs=["text", "text_type", "class"], outputs={"class" , "label"} , metrics=[ "metrics.f1_micro_multi_label", "metrics.f1_macro_multi_label", "metrics.accuracy", ], ) ``` This makes it hard for the user to...