lighteval Loading local data for custom tasks

Hi team, I am running some eval with generate custom data locally. Is it possible to dynamically add custom tasks with a local file directory? Ideally I would like to be able to do something like: load_datasets("parquet", data_files="path_to_my_data") I do not believe data_files is an argument to LightevalTaskConfig ? Thank you!

Apr 21 '25 04:04 ScottHoang

Hi ! No it's not possible atm but you can definetely add it and we would be happy to review and merge it :)

Apr 22 '25 11:04 NathanHB

I resolved it. will post PR eod

Apr 24 '25 19:04 ScottHoang

Hi @NathanHB. I'm also interested in this feature. Were you happy with the changes originally proposed in #687? If so, happy to resubmit those in a new PR with style checks. Thank you!

Nov 18 '25 16:11 davebiagioni

hey @davebiagioni ! Yes absolutely, would be happy to review and merge this.

I believe simply adding a data_files in the lighteval task config that defaults to None but loads files when defined should work.

Ping if you need any help on this :)

Nov 20 '25 10:11 NathanHB

@NathanHB here you go, happy to iterate as needed. thanks!! #1083

Nov 24 '25 19:11 davebiagioni

thanks ! added a review :)

Dec 04 '25 14:12 NathanHB