llm-foundry icon indicating copy to clipboard operation
llm-foundry copied to clipboard

Add big bench hard

Open bmosaicml opened this issue 1 year ago • 0 comments

Adding Big Bench Hard subset as a set of combined CoT tasks, formatted according to the specification in this repo.

These tasks are quite large and quite slow. I don't recommend including in all gauntlet runs, though it might be useful to include for some ad-hoc IFT runs.

We can calibrate them later to add to gauntlet.

Testing MPT7B: mpt-eval-cStFsb

bmosaicml avatar Feb 08 '24 21:02 bmosaicml