lighteval icon indicating copy to clipboard operation
lighteval copied to clipboard

Documenting how to launch multilingual tasks

Open clefourrier opened this issue 6 months ago • 2 comments

Atm, need to use custom tasks to launch them, must be documented

clefourrier avatar Jun 23 '25 11:06 clefourrier

Came here to open this issue. It took me some time to figure out that I need to a pass a custom task file and not a folder:

lighteval accelerate "model_name=gpt2" "community|tumlu:azerbaijani|0|0" --custom-tasks .\community_tasks\turkic_evals.py

It would be nice if there was an example like this in README:

ceferisbarov avatar Jun 24 '25 13:06 ceferisbarov

+1

mertbozkir avatar Sep 03 '25 15:09 mertbozkir