promptbase icon indicating copy to clipboard operation
promptbase copied to clipboard

Shuffling Ensemble: unshuffling ensemble variants for voting is missing in the code

Open HyunggyuJang opened this issue 1 year ago • 0 comments

Issue Description

The code is missing the implementation for reverting shuffling in the voting process. Currently, the shuffling ensemble is implemented in https://github.com/microsoft/promptbase/blob/dcafda47c2440d52a7e2332e1ce6801f1c644a84/src/promptbase/mmlu/problem_utils.py#L63 However, I couldn't find a section in https://github.com/microsoft/promptbase/blob/dcafda47c2440d52a7e2332e1ce6801f1c644a84/src/promptbase/mmlu/analyze.py#L77-L98 where the original order is recovered to account for the voting result.

Expected Behavior

The code should include the necessary logic to revert the shuffling and account for the voting result.

HyunggyuJang avatar Dec 16 '23 12:12 HyunggyuJang