LLM benchmarking on Oobabooga?

[ghjardim]

Description

Hello. I have a lot of GGUF LLM models in my computer, all of them are 7B with different types of quantizations. I run all of them on CPU. I'd like to benchmark them, comparing:

• Prompt evaluation speed;
• Inference speed (token generation);
• Quality of output (generation of factual answers, and similar scenarios, in a similar way to OpenLLM Leaderboard).

I'm wondering if it's possible to do such kind of benchmarking on Oobabooga. If not, it'd be nice for the users to be able to do it, for them to decide for the best model they have.