unitxt

unitxt copied to clipboard

Using cached metrics in consecutive test_card runs can be faulty

4

I noticed that a consecutive `test_card` is using the same instance of the metric used by the previous call (for a different card). This is problematic for me, because the...

antonioan

performance profiler with visualization

10

dafnapension

Fix missing deep copy in MapInstanceValues

3

If the value we map to is a list, we must deepcopy the value, otherwise external changes to the list effect the value of the mapper .

yoavkatz

Added calls to prepare() to ensure requirement checks are done early

2

yoavkatz

Integrating the metrics-llmaj pipeline with the current unitxt llmaj. This required some changes outside the scope of our new catalog: 1. moving code from fm-eval to unitxt, including changes from...

lilacheden

Right now, we don't report the number of instances (only score, score_name, and confidence interval). This is always useful, but more so when we use the group_by options, which splits...

yoavkatz

assaftibm

Some inference engines use environment variables and it is not documented how to set them up.

elronbandel

Changing v11 instruction to test LLMAJ grounded ensemble

1

KMirshahi

Example: dataset = load_dataset( card="cards.wnli", template="templates.classification.multi_class.relation.default", num_demos=2, demos_pool_size=10, loader_limit=20, ) print(dataset) print(dataset["demos_pool"][0]['source']) DatasetDict({ validation: Dataset({ features: [‘metrics’, ‘data_classification_policy’, ‘target’, ‘references’, ‘postprocessors’, ‘source’, ‘task_data’, ‘group’], num_rows: 3 }) test: Dataset({ features:...

yoavkatz