How to submit lightening model for evaluation

[Amshaker]

Hi,

I have a question please, assume that I fine-tuned the Falcon model with adapters on a specific dataset and want to upload it on openLLM leaderboard for evaluation. The steps are clear here for transformer models. How can we do same model upload using lit-gpt repo?

Best regards, Abdelrahman.

[carmocca]

The HF leaderboard only supports HF model definitions. Conversion from our format to HF is tracked in #183.

Alternatively, you could run the https://github.com/EleutherAI/lm-evaluation-harness which is what the leaderboard uses under the hood. @lantiga had a working version for lit-llama. Maybe he can share it

[Amshaker]

@lantiga It would be awesome if you shared the working version for lit-llama, please.

[carmocca]

The https://github.com/Lightning-AI/lit-gpt/blob/main/tutorials/evaluation.md tutorial goes over this