unitxt

unitxt copied to clipboard

Update engine ibm_wml model

1

In _prepare/engines/ibm_wml/llama3.py_ in _model_list_ there is a model that is outdated and should be replaced with more recent `meta-llama/llama-3-3-70b-instruct` model. With this outdated model some fm-eval users might have problems...

MikolajCharchut

A replacement is needed for dataset lmsys/arena-hard-browser that was gone from HF

2

`'lmsys/arena-hard-browser'` was gone from HF. `test_preparation` does not catch faulty `prepare/cards/arena_hard/generation/english_gpt-4-0314_reference.py` that tries to access this (gone) dataset, because a missing dataset is considered by `test_preparation` an error to be...

dafnapension

elronbandel

Unify llm judges into a single prepare file

3

This PR moves judges in `prepare/metrics/llm_as_judge/direct/llama_3_3_70b_instruct_adherence_completeness.py` to `prepare/metrics/llm_as_judge/llm_as_judge.py` so that: - judges and the underlying inference engine are created using the same inference engine/judge parameter set (for example: temperature =...

martinscooper

eladven

For issue 1575: Eliminating Manual Class Registration in Unitxt, replaced by Import Paths

4

`__type__` in catalog is expressed as a dict {`module`: module, `name`: class_name}, therefrom classes are instantiated through python's import utils. This means that if a class `c` is defined in...

dafnapension

elronbandel

Remove IBM GenAI support following sunset

2

genai service was sunset, we should remove all references to it from Unitxt code base

yoavkatz

Eliminating Manual Class Registration in Unitxt with Import Paths

14

### **Problem Statement** In Unitxt, every artifact in the catalog includes a `__type__` field in its JSON representation. This field stores the class that was used to instantiate the artifact,...

elronbandel

Document all env variables

3

Today we don't have clear documentation of all the env variables.

yoavkatz