Yann Dubois

Makes sense, then maybe a better way would be to just support `max_length` like in HuggingFace [GenerationConfig](https://huggingface.co/docs/transformers/main_classes/text_generation) so that we can get a fixed total length without it having to...

+1 that would really be amazing! 💯

> Hey guys! Would be nice if somebody would collect such models in a single place. I think a simple .md file at this repository would be enough. I really...

Hi Jon! Sorry for the misunderstanding but the right leaderboard to update is [this one](https://github.com/tatsu-lab/alpaca_eval/blob/main/src/alpaca_eval/leaderboards/data_AlpacaEval/alpaca_eval_gpt4_leaderboard.csv). I updated the doc to make it more clear. Please pull from `main` and run...

Hi @Peanuttttttttt can you show the model_configs you are using? My guess is that all the time is spent in the generation and not eval, which would explain the slowness...

Hi @liutianlin0121, this is not currently supported. Also the inference that we provide is generally pretty slow compared to current standards. If it's too slow for you, I would actually...

thanks for flagging this, the issue seems to be https://github.com/imageio/imageio/issues/973 if you send a PR I'd be happy to merge it. Another possibility would be downgrading imageio

Thanks @rwitten, something like the Chinchilla results was what I was asking about but I was hoping to see the actual training curves and final evaluation results to be able...

Great, thanks!

Yeah, I like that! Are you suggesting to save that in the csv leaderboard? Note that all os "m:win|e:weighted_ae_g4turbo |b:g4turbo|ae:2" is actually equivalent to "AlpacaEval 2" and thus constant inside...