elk-generalization icon indicating copy to clipboard operation
elk-generalization copied to clipboard

Support logging of accuracy for non-constant answer choices

Open AlexTMallen opened this issue 1 year ago • 0 comments

Huggingface evaluation doesn't allow passing of the answer choices to the evaluation metric function, so we are currently asserting that the answer choices for all examples are the same.

AlexTMallen avatar Feb 16 '24 07:02 AlexTMallen