lighteval
lighteval copied to clipboard
Documenting how to launch multilingual tasks
Atm, need to use custom tasks to launch them, must be documented
Came here to open this issue. It took me some time to figure out that I need to a pass a custom task file and not a folder:
lighteval accelerate "model_name=gpt2" "community|tumlu:azerbaijani|0|0" --custom-tasks .\community_tasks\turkic_evals.py
It would be nice if there was an example like this in README:
+1