Yoav Katz

matthews_correlation returning 0 on perfect correlation

11

Why is this the accepted behavior (strict=False was set a long time ago)? ******************************************************************************** The results of running the main metric in used in the card (matthews_correlation) over simulated predictions...

rougeL returns 0 score on perfect prediction in some languages

1

Change xlsum.py to run on all languages (remove if `lang == langs[0]:`) Run `python prepare/cards/xlsum.py` Traceback (most recent call last): File "/home/runner/work/unitxt/unitxt/tests/test_preperation.py", line 47, in test_preprations import_module_from_file(file) File "/home/runner/work/unitxt/unitxt/tests/test_preperation.py", line...

This is the root cause: https://github.com/IBM/unitxt/blob/main/src/unitxt/test_utils/metrics.py#L75 I think we need to test if the inner metric in MetricPipeline is GlobalMetri object and if so, set the n_resamples=3 to it.

Remaining issues with additional datasets

2

There are few open issues: There is no multi_label template (fix required to unfair_tos and reuters) Can I use text_type : argument? I wonder if dbpedia_14 is of type text...

Loaders need it, do other operators? https://github.com/IBM/unitxt/pull/339

Subtle issue in ExtractFieldValues

1

The tests of ExtractFieldValues work well but when I use it, I saw the the field was not added. The reason , I think is that process method modified the...

We have a request for another augmentor that adds whitespace at both the start and end of string: "let's add up to 5 consecutive whitespaces at the beginning and end...

Today templates can be defined that use field that don't exist or not use any fields at all. 1. test_card() should check all templates and not just one. 2. it...

I changed a card (added a preprocessing step), but the dataset was loaded from cache: 07/16/2023 13:49:32 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /Users/yoavkatz/cache/huggingface/datasets/unitxt___data/card=cards.sst2_sentiment,template_item=0/1.1.1/161c975966d35694e0db488ca61993c4a4cfb44975f0fa25e6aac6dc3806b97f/cache-d2a30425e116067b.arrow Need to...

Some data is available in Huggingface spaces and not HF datasets. We'd like a custom loader from Huggingface spaces. ``` class LoadFromHFSpace(Loader): user_name: str space_name:str data_files: Mapping[str, str]] _requirements_list: List[str]...