Eval: Added Repeating Consonants Eval

[MikelCalvo]

Eval details 📑

Eval name

repeat_consonants

Eval description

Tests the model's ability to repeat consonants x number of times in a given text.

What makes this a useful eval?

• Is something easily made by humans.
• GPT3.5 Fails a lot on this task
• It's a good way to demonstrate that can follow simple instructions clearly

Criteria for a good eval ✅

Below are some of the criteria we look for in a good eval. In general, we are seeking cases where the model does not do a good job despite being capable of generating a good response (note that there are some things large language models cannot do, so those would not make good evals).

Your eval should be:

• [x] Thematically consistent: The eval should be thematically consistent. We'd like to see a number of prompts all demonstrating some particular failure mode. For example, we can create an eval on cases where the model fails to reason about the physical world.
• [x] Contains failures where a human can do the task, but either GPT-4 or GPT-3.5-Turbo could not.
• [x] Includes good signal around what is the right behavior. This means either a correct answer for Basic evals or the Fact Model-graded eval, or an exhaustive rubric for evaluating answers for the Criteria Model-graded eval.
• [x] Include at least 100 high quality examples (it is okay to only contribute 5-10 meaningful examples and have us test them with GPT-4 before adding all 100)

If there is anything else that makes your eval worth including, please document it below.

Eval structure 🏗️

Your eval should

• [x] Check that your data is in evals/registry/data/{name}
• [x] Check that your yaml is registered at evals/registry/evals/{name}.jsonl
• [x] Ensure you have the right to use the data you submit via this eval

(For now, we will only be approving evals that use one of the existing eval classes. You may still write custom eval classes for your own cases, and we may consider merging them in the future.)

Final checklist 👀

Submission agreement

By contributing to Evals, you are agreeing to make your evaluation logic and data under the same MIT license as this repository. You must have adequate rights to upload any data used in an Eval. OpenAI reserves the right to use this data in future service improvements to our product. Contributions to OpenAI Evals will be subject to our usual Usage Policies (https://platform.openai.com/docs/usage-policies).

• [x] I agree that my submission will be made available under an MIT license and complies with OpenAI's usage policies.

Email address validation

If your submission is accepted, we will be granting GPT-4 access to a limited number of contributors. Access will be given to the email address associated with the merged pull request.

• [x] I acknowledge that GPT-4 access will only be granted, if applicable, to the email address used for my merged pull request.

Limited availability acknowledgement

We know that you might be excited to contribute to OpenAI's mission, help improve our models, and gain access to GPT-4. However, due to the requirements mentioned above and high volume of submissions, we will not be able to accept all submissions and thus not grant everyone who opens a PR GPT-4 access. We know this is disappointing, but we hope to set the right expectation before you open this PR.

• [x] I understand that opening a PR, even if it meets the requirements above, does not guarantee the PR will be merged nor GPT-4 access granted.

Submit eval

• [x] I have filled out all required fields in the evals PR form
• (Ignore if not submitting code) I have run pip install pre-commit; pre-commit install and have verified that black, isort, and autoflake are running when I commit and push

Failure to fill out all required fields will result in the PR being closed.

Eval JSON data

Since we are using Git LFS, we are asking eval submitters to add in as many Eval Samples (at least 5) from their contribution here:

View evals in JSON

Eval

{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Repeat every consonant in this text 2 times: The sisters hurry on the condescending music."}], "ideal": "TThhe ssisstterrss hhurrrryy onn tthhe cconnddessccennddinngg mmussicc."}
{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Repeat every consonant in this text 2 times: The guard dreams the tight familiar."}], "ideal": "TThhe gguarrdd ddrreammss tthhe ttigghhtt ffammilliarr."}
{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Repeat every consonant in this text 2 times: Some previous freedom meddles into the rotten muscle."}], "ideal": "SSomme pprrevviouss ffrreeddomm mmeddddlless inntto tthhe rrottttenn mmusscclle."}
{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Repeat every consonant in this text 2 times: The male harasses the nice business."}], "ideal": "TThhe mmalle hharrassssess tthhe nnicce bbussinnessss."}
{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Repeat every consonant in this text 2 times: How did the prior manage for the arm?"}], "ideal": "HHoww ddidd tthhe pprriorr mmannagge fforr tthhe arrmm?"}
{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Repeat every consonant in this text 3 times: Some third membership suggests into the likely chest."}], "ideal": "SSSommme ttthhhirrrddd mmmemmmbbberrrssshhhippp sssuggggggessstttsss innnttto ttthhhe lllikkkelllyyy ccchhhesssttt."}
{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Repeat every consonant in this text 3 times: What if the dutiful plane injures the mail?"}], "ideal": "WWWhhhattt ifff ttthhhe dddutttifffulll ppplllannne innnjjjurrresss ttthhhe mmmailll?"}
{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Repeat every consonant in this text 3 times: The basic middles are better than the weekend."}], "ideal": "TTThhhe bbbasssiccc mmmiddddddlllesss arrre bbbetttttterrr ttthhhannn ttthhhe wwweekkkennnddd."}
{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Repeat every consonant in this text 3 times: It was then the firsthand judgment met the absorbed load."}], "ideal": "Ittt wwwasss ttthhhennn ttthhhe fffirrrsssttthhhannnddd jjjudddgggmmmennnttt mmmettt ttthhhe abbbsssorrrbbbeddd llloaddd."}
{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Repeat every consonant in this text 3 times: Did the famous snow really pine the conclusion?"}], "ideal": "DDDiddd ttthhhe fffammmousss sssnnnowww rrreallllllyyy pppinnne ttthhhe ccconnnccclllusssionnn?"}
{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Repeat every consonant in this text 4 times: How did the stroke kick for the structure?"}], "ideal": "HHHHowwww ddddidddd tttthhhhe ssssttttrrrrokkkke kkkkicccckkkk fffforrrr tttthhhhe ssssttttrrrrucccctttturrrre?"}
{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Repeat every consonant in this text 4 times: The idea expands the worthy sector."}], "ideal": "TTTThhhhe iddddea exxxxppppannnnddddssss tttthhhhe wwwworrrrtttthhhhyyyy sssseccccttttorrrr."}
{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Repeat every consonant in this text 4 times: The mealy directors can cry in the bridge."}], "ideal": "TTTThhhhe mmmmeallllyyyy ddddirrrreccccttttorrrrssss ccccannnn ccccrrrryyyy innnn tttthhhhe bbbbrrrriddddgggge."}
{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Repeat every consonant in this text 4 times: It was then the giant bar met the distorted plant."}], "ideal": "Itttt wwwwassss tttthhhhennnn tttthhhhe ggggiannnntttt bbbbarrrr mmmmetttt tttthhhhe ddddissssttttorrrrttttedddd ppppllllannnntttt."}
{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Repeat every consonant in this text 4 times: The associates pack on the merciful mind."}], "ideal": "TTTThhhhe assssssssocccciattttessss ppppacccckkkk onnnn tttthhhhe mmmmerrrrcccciffffullll mmmminnnndddd."}
{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Repeat every consonant in this text 5 times: The cool guys are better than the bite."}], "ideal": "TTTTThhhhhe cccccoolllll ggggguyyyyysssss arrrrre bbbbbetttttttttterrrrr ttttthhhhhannnnn ttttthhhhhe bbbbbittttte."}
{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Repeat every consonant in this text 5 times: It was then the sour operation met the uncomfortable failure."}], "ideal": "Ittttt wwwwwasssss ttttthhhhhennnnn ttttthhhhhe sssssourrrrr oppppperrrrratttttionnnnn mmmmmettttt ttttthhhhhe unnnnncccccommmmmffffforrrrrtttttabbbbbllllle fffffailllllurrrrre."}
{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Repeat every consonant in this text 5 times: The austere safes are better than the diet."}], "ideal": "TTTTThhhhhe ausssssttttterrrrre sssssafffffesssss arrrrre bbbbbetttttttttterrrrr ttttthhhhhannnnn ttttthhhhhe dddddiettttt."}
{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Repeat every consonant in this text 5 times: The well-worn round can't welcome with the garbage."}], "ideal": "TTTTThhhhhe wwwwwellllllllll-----wwwwworrrrrnnnnn rrrrrounnnnnddddd cccccannnnn'ttttt wwwwwelllllcccccommmmme wwwwwittttthhhhh ttttthhhhhe gggggarrrrrbbbbbaggggge."}
{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Repeat every consonant in this text 5 times: The silver line can't peel with the debt."}], "ideal": "TTTTThhhhhe sssssilllllvvvvverrrrr lllllinnnnne cccccannnnn'ttttt pppppeelllll wwwwwittttthhhhh ttttthhhhhe dddddebbbbbttttt."}
{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Repeat every consonant in this text 6 times: The pair whines the spotless review."}], "ideal": "TTTTTThhhhhhe ppppppairrrrrr wwwwwwhhhhhhinnnnnnessssss tttttthhhhhhe ssssssppppppottttttllllllessssssssssss rrrrrrevvvvvviewwwwww."}
{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Repeat every consonant in this text 6 times: The attention presents the uninterested consideration."}], "ideal": "TTTTTThhhhhhe attttttttttttennnnnnttttttionnnnnn pppppprrrrrressssssennnnnnttttttssssss tttttthhhhhhe unnnnnninnnnnntttttterrrrrressssssttttttedddddd cccccconnnnnnssssssidddddderrrrrrattttttionnnnnn."}
{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Repeat every consonant in this text 6 times: The nonchalant professional can't frighten with the original."}], "ideal": "TTTTTThhhhhhe nnnnnnonnnnnncccccchhhhhhallllllannnnnntttttt pppppprrrrrroffffffessssssssssssionnnnnnallllll ccccccannnnnn'tttttt ffffffrrrrrrigggggghhhhhhttttttennnnnn wwwwwwitttttthhhhhh tttttthhhhhhe orrrrrrigggggginnnnnnallllll."}
{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Repeat every consonant in this text 6 times: The ranges empty on the adolescent male."}], "ideal": "TTTTTThhhhhhe rrrrrrannnnnnggggggessssss emmmmmmppppppttttttyyyyyy onnnnnn tttttthhhhhhe addddddollllllessssssccccccennnnnntttttt mmmmmmalllllle."}
{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Repeat every consonant in this text 6 times: The civil lies are better than the tank."}], "ideal": "TTTTTThhhhhhe ccccccivvvvvvillllll lllllliessssss arrrrrre bbbbbbetttttttttttterrrrrr tttttthhhhhhannnnnn tttttthhhhhhe ttttttannnnnnkkkkkk."}
{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Repeat every consonant in this text 7 times: The breakable safes can spoil in the bone."}], "ideal": "TTTTTTThhhhhhhe bbbbbbbrrrrrrreakkkkkkkabbbbbbbllllllle sssssssafffffffesssssss cccccccannnnnnn ssssssspppppppoilllllll innnnnnn ttttttthhhhhhhe bbbbbbbonnnnnnne."}
{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Repeat every consonant in this text 7 times: What if the encouraging bus muddles the land?"}], "ideal": "WWWWWWWhhhhhhhattttttt ifffffff ttttttthhhhhhhe ennnnnnncccccccourrrrrrraggggggginnnnnnnggggggg bbbbbbbusssssss mmmmmmmuddddddddddddddlllllllesssssss ttttttthhhhhhhe lllllllannnnnnnddddddd?"}
{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Repeat every consonant in this text 7 times: Did the unique item really analyze the stuff?"}], "ideal": "DDDDDDDiddddddd ttttttthhhhhhhe unnnnnnniqqqqqqque itttttttemmmmmmm rrrrrrreallllllllllllllyyyyyyy annnnnnnalllllllyyyyyyyzzzzzzze ttttttthhhhhhhe ssssssstttttttuffffffffffffff?"}
{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Repeat every consonant in this text 7 times: The additions found on the nippy period."}], "ideal": "TTTTTTThhhhhhhe adddddddddddddditttttttionnnnnnnsssssss fffffffounnnnnnnddddddd onnnnnnn ttttttthhhhhhhe nnnnnnnippppppppppppppyyyyyyy ppppppperrrrrrrioddddddd."}
{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Repeat every consonant in this text 7 times: The charming nothings are better than the refrigerator."}], "ideal": "TTTTTTThhhhhhhe ccccccchhhhhhharrrrrrrmmmmmmminnnnnnnggggggg nnnnnnnottttttthhhhhhhinnnnnnngggggggsssssss arrrrrrre bbbbbbbetttttttttttttterrrrrrr ttttttthhhhhhhannnnnnn ttttttthhhhhhhe rrrrrrrefffffffrrrrrrrigggggggerrrrrrratttttttorrrrrrr."}
{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Repeat every consonant in this text 8 times: The trusty few can't serve with the strength."}], "ideal": "TTTTTTTThhhhhhhhe ttttttttrrrrrrrrussssssssttttttttyyyyyyyy ffffffffewwwwwwww ccccccccannnnnnnn'tttttttt sssssssserrrrrrrrvvvvvvvve wwwwwwwwitttttttthhhhhhhh tttttttthhhhhhhhe ssssssssttttttttrrrrrrrrennnnnnnnggggggggtttttttthhhhhhhh."}
{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Repeat every consonant in this text 8 times: The knotty profession can't kiss with the point."}], "ideal": "TTTTTTTThhhhhhhhe kkkkkkkknnnnnnnnottttttttttttttttyyyyyyyy pppppppprrrrrrrroffffffffessssssssssssssssionnnnnnnn ccccccccannnnnnnn'tttttttt kkkkkkkkissssssssssssssss wwwwwwwwitttttttthhhhhhhh tttttttthhhhhhhhe ppppppppoinnnnnnnntttttttt."}
{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Repeat every consonant in this text 8 times: What if the equable wish attacks the collar?"}], "ideal": "WWWWWWWWhhhhhhhhatttttttt iffffffff tttttttthhhhhhhhe eqqqqqqqquabbbbbbbblllllllle wwwwwwwwisssssssshhhhhhhh attttttttttttttttacccccccckkkkkkkkssssssss tttttttthhhhhhhhe ccccccccollllllllllllllllarrrrrrrr?"}
{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Repeat every consonant in this text 8 times: The tracks shop on the aggressive background."}], "ideal": "TTTTTTTThhhhhhhhe ttttttttrrrrrrrracccccccckkkkkkkkssssssss sssssssshhhhhhhhopppppppp onnnnnnnn tttttttthhhhhhhhe aggggggggggggggggrrrrrrrressssssssssssssssivvvvvvvve bbbbbbbbacccccccckkkkkkkkggggggggrrrrrrrrounnnnnnnndddddddd."}
{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Repeat every consonant in this text 8 times: The faulty gases are better than the summer."}], "ideal": "TTTTTTTThhhhhhhhe ffffffffaullllllllttttttttyyyyyyyy ggggggggassssssssessssssss arrrrrrrre bbbbbbbbetttttttttttttttterrrrrrrr tttttttthhhhhhhhannnnnnnn tttttttthhhhhhhhe ssssssssummmmmmmmmmmmmmmmerrrrrrrr."}

[MikelCalvo]

> oaieval gpt-3.5-turbo repeat-consonants --max_samples 100

[2023-03-15 19:14:29,625] [registry.py:145] Loading registry from D:\Github\evals\evals\registry\evals
[2023-03-15 19:14:29,647] [registry.py:145] Loading registry from C:\Users\Mikel\.evals\evals
[2023-03-15 19:14:30,467] [oaieval.py:178] Run started: 230315181430WX7J7CQI
[2023-03-15 19:14:30,472] [eval.py:30] Evaluating 100 samples
[2023-03-15 19:14:30,478] [eval.py:136] Running in threaded mode with 10 threads!
 30% |███████████████                                   | 30/100 [00:10<00:21,  3.29it/s]
 65% |████████████████████████████████▌                 | 65/100 [00:20<00:09,  3.60it/s]
100%|█████████████████████████████████████████████████| 100/100 [00:33<00:00,  3.01it/s]
[2023-03-15 19:15:03,767] [record.py:320] Final report: {'accuracy': 0.0}.
[2023-03-15 19:15:03,768] [oaieval.py:209] Final report:
[2023-03-15 19:15:03,768] [oaieval.py:211] accuracy: 0.0

[luqman-openai]

Thanks for opening this PR, Character level reasoning and counting is a well-known failure mode of the model due to a common underlying issue in LLMs. In its current form, this eval does not seem to expose any new gaps in our understanding of model performance.

If you're still interested in writing an eval, we've noticed that these criteria make good evals. If you have any particular use case in mind for the model, can you come up with an eval that has some of these attributes?

• Multi-step reasoning

• Domain or Application specific

• Open-Ended responses

• Complex instructions

• The eval seems obvious but tricks the model in a novel way

Closing this PR, please open another PR with the provided suggestions.