benchllama icon indicating copy to clipboard operation
benchllama copied to clipboard

Published 20 hours ago • verified publisher icon srikanth235

llama3:instruct nan

Open falmanna opened this issue 10 months ago • 2 comments

Running the benchmark against llama3 is returning nan for all rates

falmanna avatar Apr 20 '24 02:04 falmanna

Currently, benchllama supports evaluating FIM models. Will try to add eval support for instruct fine-tuned models soon. Hope this clarifies the confusion.

srikanth235 avatar Apr 23 '24 05:04 srikanth235

@srikanth235 thanks for the work!

I did actually run it successfully against multiple instruct fine-tunes models, is there a difference between them and llama3:instruct?

  • codellama:13b-instruct
  • mistral:7b-instruct
  • solar:10.7b-instruct-v1-q4_1

falmanna avatar Apr 23 '24 11:04 falmanna

okay seems like things are working fine now, might have been an ollama problem. Closing.

falmanna avatar May 12 '24 14:05 falmanna