unitxt

unitxt copied to clipboard

Currently demo examples are selected from the demo pool (which is typically extracted from the train set). The selection of demos per instance is random by default, or more complex...

yoavkatz

Current Formats do support different target_prefix in demos

2

Today the code in formats.py assumes the target prefix for the demos is taken from the instance. This may cause subtle issue when the target_prefix uses input field for formatting...

yoavkatz

Exponential time complexity when confidence interval calculation is enabled

13

The code below will call a full-dataset metric computation for each row - i.e. the work is duplicated num_row times. This is a big issue for huggingface metrics, such as...

jezekra1

add chat rag datasets and dialog operator

1

OfirArviv

Bluebench updates

1

Adding Bluebench, including the arena-hard

perlitz

elronbandel

MMLU Template Instruction Inconsistency

1

Hi, I noticed that in some templates of MMLU, it seems that the part of the prompt "The following are multiple choice questions (with answers) about {topic}." is being placed...

RonaldBXu

Issue with Backward Compatibility in Version Updates

1

I recently encountered an issue where a change in the code, specifically modifying the check from `if key == "type"` to `if key == "__type__"`, caused my locally saved data...

eliyahabba

1) Seperate the structured representation and make it available at a dedicated field for people want to use it externally (e.g for using open ai api) (2) change existing formats...

yoavkatz

Add speed ups guide

1

elronbandel