Aloysius Chow

The links in the readme were changed, but the new repos are missing the model checkpoints as you said. Old repos with checkpoints are still up at: ChanceFocus/finma-7b-full ChanceFocus/finma-7b-nlp

No worries! I'm not a contributor, I think someone on the team will update the new repositories soon. Would be good to leave this open so the team is aware,...

I encountered the same issue last night with `--model hf-causal-llama`. I got it to run by loading the model as a VLLM model with the below arguments, but benchmark results...

@stas00 You're right, I think the poor performance is largely because prompts are truncated to the maximum context length, and there is no response as a result. This shows up...

@stas00 The latest paper expands the set of tasks that models are benchmarked on, I wanted a subset of tasks that covered most of the categories that I could run...

@jiminHuang Your response is much appreciated, I hope to hear from the team regarding the release of instructions for reproducing your results when possible. I have reached out via email...