mobile_app_open icon indicating copy to clipboard operation
mobile_app_open copied to clipboard

LLM TinyMMLU Dataset Implementation

Open farook-edev opened this issue 5 months ago • 0 comments

As described in the parent issue, implementation of TinyMMLU is quite complete. The only things I think might need to be done are:

  • ~Load data sample by sample as needed instead of all at once.~ (not necessary)
  • ~Save output tokens in processing step and do all accuracy related calculations on the accuracy side. (needs discussion)~
  • ~handle instances where no answer letter is provided. (likely fail query)~ (currently being done implicitly)
  • ~properly build zero-shot prompt rather than cutting input_formatted~

farook-edev avatar Oct 02 '25 00:10 farook-edev