promptbase

promptbase copied to clipboard

pirsoz

Missing mmlu as valid_dataset even though readme encourages download

1

(gpt5) [jfreeman@mir-81 promptbase]$ python -m promptbase mmlu usage: __main__.py [-h] [--subject SUBJECT] [--list_subjects] {gsm8k,humaneval,math,drop,bigbench} __main__.py: error: argument dataset: invalid choice: 'mmlu' (choose from 'gsm8k', 'humaneval', 'math', 'drop', 'bigbench') Head of...

jackfreeman88

MMLU dataset

1

Hi, How to convert the MMLU csv data to json data (e.g: mmlu_questions_dev.json) Thanks

pribadihcr

…and unify the code

eukub

gpt4.jsonl with open("gpt4.jsonl") as f: FileNotFoundError: [Errno 2] No such file or directory: 'gpt4.jsonl'

lydiayyang

Add colab

2

Would be helpful to have a colab for prompting with medprompt on a custom question and answer pair

franz101

Replace Chain of Thought with Thread of Thought

2

https://arxiv.org/abs/2311.08734 is Thread of Thought new SOTA over Chain of Thought?

bennmann

Clarification needed in evaluation numbers

5

Hello, Thanks for the repo and awesome work. I am requesting clarification on the evaluation results shown in the repo. For humanEval Zero shot, GPT-4's score is reported here as...

saurabhkumar8112

## Issue Description The code is missing the implementation for reverting shuffling in the voting process. Currently, the shuffling ensemble is implemented in https://github.com/microsoft/promptbase/blob/dcafda47c2440d52a7e2332e1ce6801f1c644a84/src/promptbase/mmlu/problem_utils.py#L63 However, I couldn't find a section...

HyunggyuJang

Working on a GSM8K benchmark using Guidance and some local models. Inspired in part by https://arxiv.org/abs/2403.06988

riedgar-ms