Thank you for contributing an eval! ♥️

🚨 Please make sure your PR follows these guidelines, failure to follow the guidelines below will result in the PR being closed automatically. Note that even if the criteria are met, that does not guarantee the PR will be merged nor GPT-4 access granted. 🚨

PLEASE READ THIS:

In order for a PR to be merged, it must fail on GPT-4. We are aware that right now, users do not have access, so you will not be able to tell if the eval fails or not. Please run your eval with GPT-3.5-Turbo, but keep in mind as we run the eval, if GPT-4 gets higher than 90% on the eval, we will likely reject since GPT-4 is already capable of completing the task.

We plan to roll out a way for users submitting evals to see the eval performance on GPT-4 soon. Stay tuned! Until then, you will not be able to see the eval performance on GPT-4. We encourage partial PR's with ~5-10 example that we can then run the evals on and share the results with you so you know how your eval does with GPT-4 before writing all 100 examples.

Eval details 📑

Eval name

Unique Combinations with Constraints (unique_combinations)

Eval description

unique_combinations takes a classic combinatorial coding problem from the whiteboard of a coding interview and presents it to GPT-4. The basic is idea is this: given an unlimited supply of coins (X) in denominations of 3, 5, ..., what is the highest/lowest (constraint) number of X to achieve 35 cents (Y)? In my testing, GPT-4 could not produce the correct answer, so I adjusted the problem slightly to promote explainability. Instead of asking for just the highest or lowest number of X, I instead ask for the unique combination of Xs that satisfies the constraint to achieve exactly Y. (Note that GPT-4 does yield consistent results between those two distinct problems). To abstract it away from the classic "coin" problem, I include several different scenarios that poke at the same base combinatorial nature of the problem (e.g. weight of boxes, denominations of coins, objects in a bag).

I originally noticed that GPT-4 was bad at permutations/combinations when playing around with ChatGPT Plus, so I decided to format that problem in this way. The current data does not guarantee all denominations will be used, but I've written the code to generate the prompts in an extensible way that allows for additional scenarios, constraints, and combination methods to be easily added. I can make that available or work on extending the scope of this combinatorial problem myself, if desired.

What makes this a useful eval?

It tests combinatorial reasoning in the real world as well as arithmetic based on that reasoning. In my testing, neither GPT-3.5-Turbo nor GPT-4 could produce the correct unique combination, and their solutions do not even add up to the target value.

Exposes flaws in the following categories:

Math / logical / physical reasoning
- It tests combinatorial reasoning in the real world as well as arithmetic based on that reasoning. In my testing, neither GPT-3.5-Turbo nor GPT-4 could produce the correct unique combination, and their solutions do not even add up to the target value.
Real-world use case
- Attempting to find the maximum number of packages that could fit on a truck given a list of weights.
- Finding fewest holes to patch in a boat to prevent it from sinking given various flow rates (could make for some fun real-world testing)
- Finding the highest number of deliveries that can be made with a given amount of gas

Criteria for a good eval ✅

Below are some of the criteria we look for in a good eval. In general, we are seeking cases where the model does not do a good job despite being capable of generating a good response (note that there are some things large language models cannot do, so those would not make good evals).

Your eval should be:

[x] Thematically consistent: The eval should be thematically consistent. We'd like to see a number of prompts all demonstrating some particular failure mode. For example, we can create an eval on cases where the model fails to reason about the physical world.
[x] Contains failures where a human can do the task, but either GPT-4 or GPT-3.5-Turbo could not.
[x] Includes good signal around what is the right behavior. This means either a correct answer for Basic evals or the Fact Model-graded eval, or an exhaustive rubric for evaluating answers for the Criteria Model-graded eval.
[x] Include at least 100 high quality examples (it is okay to only contribute 5-10 meaningful examples and have us test them with GPT-4 before adding all 100)

If there is anything else that makes your eval worth including, please document it below.

Unique eval value

What makes this problem particularly interesting is that ChatGPT will consistently get this problem wrong but will provide code that solves the problem correctly if prompted to do so (see the image attached below). If asked to produce the output of the code, it does so incorrectly, yielding its original guess. This points to an entirely separate eval: being able to return the output of Python code. Additionally, the answers provided by GPT-4 and GPT-3.5-Turbo do not add up to the target value.

Eval structure 🏗️

Your eval should

[x] Check that your data is in evals/registry/data/{name}
[x] Check that your yaml is registered at evals/registry/evals/{name}.yaml
[x] Ensure you have the right to use the data you submit via this eval

(For now, we will only be approving evals that use one of the existing eval classes. You may still write custom eval classes for your own cases, and we may consider merging them in the future.)

Final checklist 👀

Submission agreement

By contributing to Evals, you are agreeing to make your evaluation logic and data under the same MIT license as this repository. You must have adequate rights to upload any data used in an Eval. OpenAI reserves the right to use this data in future service improvements to our product. Contributions to OpenAI Evals will be subject to our usual Usage Policies (https://platform.openai.com/docs/usage-policies).

[x] I agree that my submission will be made available under an MIT license and complies with OpenAI's usage policies.

Email address validation

If your submission is accepted, we will be granting GPT-4 access to a limited number of contributors. Access will be given to the email address associated with the merged pull request.

[x] I acknowledge that GPT-4 access will only be granted, if applicable, to the email address used for my merged pull request.

Limited availability acknowledgement

We know that you might be excited to contribute to OpenAI's mission, help improve our models, and gain access to GPT-4. However, due to the requirements mentioned above and high volume of submissions, we will not be able to accept all submissions and thus not grant everyone who opens a PR GPT-4 access. We know this is disappointing, but we hope to set the right expectation before you open this PR.

[x] I understand that opening a PR, even if it meets the requirements above, does not guarantee the PR will be merged nor GPT-4 access granted.

Submit eval

[x] I have filled out all required fields in the evals PR form
[x] (Ignore if not submitting code) I have run pip install pre-commit; pre-commit install and have verified that black, isort, and autoflake are running when I commit and push

Failure to fill out all required fields will result in the PR being closed.

Eval JSON data

Since we are using Git LFS, we are asking eval submitters to add in as many Eval Samples (at least 5) from their contribution here:

View evals in JSON

Eval

{"input": [{"role": "system", "content": "Given an unlimited supply of boxes with weights of 7, 10, and 11, what is the unique combination of boxes that uses the minimum number of boxes to have a weight of exactly 65? Respond ONLY in the format <box-weight>: <number-of-box> with NO OTHER INFORMATION and DO NOT include unused boxes. Provide each value on a new line, sorted by box-weight."}], "ideal": "10: 1\n11: 5"}
{"input": [{"role": "system", "content": "Given an unlimited supply of bean bags that contain 5, 6, 7, or 18 beans, what is the unique combination of bean bags that uses the maximum number of bags to have exactly 51 beans? Respond ONLY in the format <beans-in-bag>: <number-of-bag> with NO OTHER INFORMATION and DO NOT include unused bean bags. Provide each value on a new line, sorted by beans-in-bag."}], "ideal": "5: 9\n6: 1"}
{"input": [{"role": "system", "content": "Given an unlimited supply of coins in denominations of 3, 4, 8, 9, 11, and 16, what is the unique combination of coins that uses the maximum number of coins to have a value of exactly 34? Respond ONLY in the format <denomination>: <number-of-coin> with NO OTHER INFORMATION and DO NOT include unused coins. Provide each value on a new line, sorted by denomination."}], "ideal": "3: 10\n4: 1"}
{"input": [{"role": "system", "content": "Given an unlimited supply of bean bags that contain 3, 4, 5, 6, 7, 10, 13, 14, 18, or 19 beans, what is the unique combination of bean bags that uses the minimum number of bags to have exactly 189 beans? Respond ONLY in the format <beans-in-bag>: <number-of-bag> with NO OTHER INFORMATION and DO NOT include unused bean bags. Provide each value on a new line, sorted by beans-in-bag."}], "ideal": "18: 1\n19: 9"}
{"input": [{"role": "system", "content": "Given an unlimited supply of coins in denominations of 5, and 19, what is the unique combination of coins that uses the maximum number of coins to have a value of exactly 44? Respond ONLY in the format <denomination>: <number-of-coin> with NO OTHER INFORMATION and DO NOT include unused coins. Provide each value on a new line, sorted by denomination."}], "ideal": "5: 5\n19: 1"}
{"input": [{"role": "system", "content": "Given an unlimited supply of bean bags that contain 10, 13, 15, 16, 17, or 18 beans, what is the unique combination of bean bags that uses the maximum number of bags to have exactly 93 beans? Respond ONLY in the format <beans-in-bag>: <number-of-bag> with NO OTHER INFORMATION and DO NOT include unused bean bags. Provide each value on a new line, sorted by beans-in-bag."}], "ideal": "10: 8\n13: 1"}
{"input": [{"role": "system", "content": "Given an unlimited supply of boxes with weights of 7, 8, 11, 12, 14, 15, 16, 17, 18, and 19, what is the unique combination of boxes that uses the minimum number of boxes to have a weight of exactly 113? Respond ONLY in the format <box-weight>: <number-of-box> with NO OTHER INFORMATION and DO NOT include unused boxes. Provide each value on a new line, sorted by box-weight."}], "ideal": "18: 1\n19: 5"}
{"input": [{"role": "system", "content": "Given an unlimited supply of boxes with weights of 5, 6, 9, 10, 11, 13, 14, and 15, what is the unique combination of boxes that uses the minimum number of boxes to have a weight of exactly 164? Respond ONLY in the format <box-weight>: <number-of-box> with NO OTHER INFORMATION and DO NOT include unused boxes. Provide each value on a new line, sorted by box-weight."}], "ideal": "14: 1\n15: 10"}
{"input": [{"role": "system", "content": "Given an unlimited supply of bean bags that contain 4, 6, 10, or 14 beans, what is the unique combination of bean bags that uses the maximum number of bags to have exactly 42 beans? Respond ONLY in the format <beans-in-bag>: <number-of-bag> with NO OTHER INFORMATION and DO NOT include unused bean bags. Provide each value on a new line, sorted by beans-in-bag."}], "ideal": "4: 9\n6: 1"}
{"input": [{"role": "system", "content": "Given an unlimited supply of coins in denominations of 5, 9, 11, and 17, what is the unique combination of coins that uses the minimum number of coins to have a value of exactly 79? Respond ONLY in the format <denomination>: <number-of-coin> with NO OTHER INFORMATION and DO NOT include unused coins. Provide each value on a new line, sorted by denomination."}], "ideal": "11: 1\n17: 4"}
{"input": [{"role": "system", "content": "Given an unlimited supply of coins in denominations of 3, 4, 7, 9, 15, 18, and 19, what is the unique combination of coins that uses the maximum number of coins to have a value of exactly 19? Respond ONLY in the format <denomination>: <number-of-coin> with NO OTHER INFORMATION and DO NOT include unused coins. Provide each value on a new line, sorted by denomination."}], "ideal": "3: 5\n4: 1"}
{"input": [{"role": "system", "content": "Given an unlimited supply of bean bags that contain 6, 18, or 19 beans, what is the unique combination of bean bags that uses the minimum number of bags to have exactly 189 beans? Respond ONLY in the format <beans-in-bag>: <number-of-bag> with NO OTHER INFORMATION and DO NOT include unused bean bags. Provide each value on a new line, sorted by beans-in-bag."}], "ideal": "18: 1\n19: 9"}
{"input": [{"role": "system", "content": "Given an unlimited supply of coins in denominations of 4, 6, 14, and 17, what is the unique combination of coins that uses the maximum number of coins to have a value of exactly 46? Respond ONLY in the format <denomination>: <number-of-coin> with NO OTHER INFORMATION and DO NOT include unused coins. Provide each value on a new line, sorted by denomination."}], "ideal": "4: 10\n6: 1"}
{"input": [{"role": "system", "content": "Given an unlimited supply of bean bags that contain 3, 5, 9, 10, 12, 13, 14, or 18 beans, what is the unique combination of bean bags that uses the maximum number of bags to have exactly 20 beans? Respond ONLY in the format <beans-in-bag>: <number-of-bag> with NO OTHER INFORMATION and DO NOT include unused bean bags. Provide each value on a new line, sorted by beans-in-bag."}], "ideal": "3: 5\n5: 1"}
{"input": [{"role": "system", "content": "Given an unlimited supply of bean bags that contain 5, 6, 7, 8, 10, 13, 18, or 19 beans, what is the unique combination of bean bags that uses the maximum number of bags to have exactly 51 beans? Respond ONLY in the format <beans-in-bag>: <number-of-bag> with NO OTHER INFORMATION and DO NOT include unused bean bags. Provide each value on a new line, sorted by beans-in-bag."}], "ideal": "5: 9\n6: 1"}
{"input": [{"role": "system", "content": "Given an unlimited supply of boxes with weights of 5, 9, and 13, what is the unique combination of boxes that uses the maximum number of boxes to have a weight of exactly 49? Respond ONLY in the format <box-weight>: <number-of-box> with NO OTHER INFORMATION and DO NOT include unused boxes. Provide each value on a new line, sorted by box-weight."}], "ideal": "5: 8\n9: 1"}
{"input": [{"role": "system", "content": "Given an unlimited supply of boxes with weights of 4, 6, and 11, what is the unique combination of boxes that uses the minimum number of boxes to have a weight of exactly 83? Respond ONLY in the format <box-weight>: <number-of-box> with NO OTHER INFORMATION and DO NOT include unused boxes. Provide each value on a new line, sorted by box-weight."}], "ideal": "6: 1\n11: 7"}

Mar 23 '23 20:03 Sypherd

Jakie mogą być efekty połączenia tego z grami o niskiej wartości w problemie typu real - fikcją + szachy + żywienie

czw., 23 mar 2023, 21:37 użytkownik Chris Sypherd @.***> napisał:

Thank you for contributing an eval! ♥️

🚨 Please make sure your PR follows these guidelines, failure to follow the guidelines below will result in the PR being closed automatically. Note that even if the criteria are met, that does not guarantee the PR will be merged nor GPT-4 access granted. 🚨

PLEASE READ THIS:

In order for a PR to be merged, it must fail on GPT-4. We are aware that right now, users do not have access, so you will not be able to tell if the eval fails or not. Please run your eval with GPT-3.5-Turbo, but keep in mind as we run the eval, if GPT-4 gets higher than 90% on the eval, we will likely reject since GPT-4 is already capable of completing the task.

We plan to roll out a way for users submitting evals to see the eval performance on GPT-4 soon. Stay tuned! Until then, you will not be able to see the eval performance on GPT-4. We encourage partial PR's with ~5-10 example that we can then run the evals on and share the results with you so you know how your eval does with GPT-4 before writing all 100 examples. Eval details 📑 Eval name

Unique Combinations with Constraints (unique_combinations) Eval description

unique_combinations takes a classic combinatorial coding problem from the whiteboard of a coding interview and presents it to GPT-4. The basic is idea is this: given an unlimited supply of Xs in denominations of x1, x2, ..., what is the (constraint) highest/lowest number of X to achieve value Y? In my testing, GPT-4 did not get the correct answer, so I adjusted problem slightly to promote explainability. Instead of asking for just the highest or lowest number of X, I instead ask for the unique combination of X that satisfies the constraint to achieve exactly Y. (Note that GPT-4 does yield consistent results between those two distinct problems). To abstract it away from the classic "coin" problem, I include several different scenarios that poke at the same base combinatorial nature of the problem (e.g. weight of boxes, denominations of coins, objects in a bag).

I originally noticed that GPT-4 was bad at permutations/combinations when playing around with ChatGPT Plus, so I decided to format that problem in this way. The current data does not guarantee all denominations will be used, but I've written the code to generate the prompts in an extensible way that allows for additional scenarios, constraints, and combination methods to be easily added. I can make that available or work on extending the scope of this combinatorial problem myself, if desired. What makes this a useful eval?

It tests combinatorial reasoning in the real world as well as arithmetic based on that reasoning. In my testing, neither GPT-3.5-Turbo nor GPT-4 could produce the correct unique combination, and their solutions do not even add up to the target value. Criteria for a good eval ✅

Below are some of the criteria we look for in a good eval. In general, we are seeking cases where the model does not do a good job despite being capable of generating a good response (note that there are some things large language models cannot do, so those would not make good evals).

Your eval should be:

Thematically consistent: The eval should be thematically consistent. We'd like to see a number of prompts all demonstrating some particular failure mode. For example, we can create an eval on cases where the model fails to reason about the physical world.

Contains failures where a human can do the task, but either GPT-4 or GPT-3.5-Turbo could not.

Includes good signal around what is the right behavior. This means either a correct answer for Basic evals or the Fact Model-graded eval, or an exhaustive rubric for evaluating answers for the Criteria Model-graded eval.

Include at least 100 high quality examples (it is okay to only contribute 5-10 meaningful examples and have us test them with GPT-4 before adding all 100)

If there is anything else that makes your eval worth including, please document it below. Unique eval value

What makes this problem particularly interesting is that ChatGPT will consistently get this problem wrong but will provide code that solves the problem correctly if prompted to do so (see the image attached below). If asked to produce the output of the code, it does so incorrectly, yielding its original guess. This points to an entirely separate eval: being able to return the output of Python code. Additionally, the answers provided by GPT-4 and GPT-3.5-Turbo do not add up to the target value. [image: image] https://user-images.githubusercontent.com/50557586/227346727-2611fa4b-06ba-42d4-b14f-f658f36300e5.png Eval structure 🏗️

Your eval should

Check that your data is in evals/registry/data/{name}

Check that your yaml is registered at evals/registry/evals/{name}.yaml

Ensure you have the right to use the data you submit via this eval

(For now, we will only be approving evals that use one of the existing eval classes. You may still write custom eval classes for your own cases, and we may consider merging them in the future.) Final checklist 👀 Submission agreement

By contributing to Evals, you are agreeing to make your evaluation logic and data under the same MIT license as this repository. You must have adequate rights to upload any data used in an Eval. OpenAI reserves the right to use this data in future service improvements to our product. Contributions to OpenAI Evals will be subject to our usual Usage Policies ( https://platform.openai.com/docs/usage-policies).

I agree that my submission will be made available under an MIT license and complies with OpenAI's usage policies.

Email address validation

If your submission is accepted, we will be granting GPT-4 access to a limited number of contributors. Access will be given to the email address associated with the merged pull request.

I acknowledge that GPT-4 access will only be granted, if applicable, to the email address used for my merged pull request.

Limited availability acknowledgement

We know that you might be excited to contribute to OpenAI's mission, help improve our models, and gain access to GPT-4. However, due to the requirements mentioned above and high volume of submissions, we will not be able to accept all submissions and thus not grant everyone who opens a PR GPT-4 access. We know this is disappointing, but we hope to set the right expectation before you open this PR.

I understand that opening a PR, even if it meets the requirements above, does not guarantee the PR will be merged nor GPT-4 access granted.

Submit eval

I have filled out all required fields in the evals PR form

(Ignore if not submitting code) I have run pip install pre-commit; pre-commit install and have verified that black, isort, and autoflake are running when I commit and push

Failure to fill out all required fields will result in the PR being closed. Eval JSON data

Since we are using Git LFS, we are asking eval submitters to add in as many Eval Samples (at least 5) from their contribution here: View evals in JSON Eval

{"input": [{"role": "system", "content": "Given an unlimited supply of boxes with weights of 7, 10, and 11, what is the unique combination of boxes that uses the minimum number of boxes to have a weight of exactly 65? Respond ONLY in the format : with NO OTHER INFORMATION and DO NOT include unused boxes. Provide each value on a new line, sorted by box-weight."}], "ideal": "10: 1\n11: 5"} {"input": [{"role": "system", "content": "Given an unlimited supply of bean bags that contain 5, 6, 7, or 18 beans, what is the unique combination of bean bags that uses the maximum number of bags to have exactly 51 beans? Respond ONLY in the format : with NO OTHER INFORMATION and DO NOT include unused bean bags. Provide each value on a new line, sorted by beans-in-bag."}], "ideal": "5: 9\n6: 1"} {"input": [{"role": "system", "content": "Given an unlimited supply of coins in denominations of 3, 4, 8, 9, 11, and 16, what is the unique combination of coins that uses the maximum number of coins to have a value of exactly 34? Respond ONLY in the format : with NO OTHER INFORMATION and DO NOT include unused coins. Provide each value on a new line, sorted by denomination."}], "ideal": "3: 10\n4: 1"} {"input": [{"role": "system", "content": "Given an unlimited supply of bean bags that contain 3, 4, 5, 6, 7, 10, 13, 14, 18, or 19 beans, what is the unique combination of bean bags that uses the minimum number of bags to have exactly 189 beans? Respond ONLY in the format : with NO OTHER INFORMATION and DO NOT include unused bean bags. Provide each value on a new line, sorted by beans-in-bag."}], "ideal": "18: 1\n19: 9"} {"input": [{"role": "system", "content": "Given an unlimited supply of coins in denominations of 5, and 19, what is the unique combination of coins that uses the maximum number of coins to have a value of exactly 44? Respond ONLY in the format : with NO OTHER INFORMATION and DO NOT include unused coins. Provide each value on a new line, sorted by denomination."}], "ideal": "5: 5\n19: 1"} {"input": [{"role": "system", "content": "Given an unlimited supply of bean bags that contain 10, 13, 15, 16, 17, or 18 beans, what is the unique combination of bean bags that uses the maximum number of bags to have exactly 93 beans? Respond ONLY in the format : with NO OTHER INFORMATION and DO NOT include unused bean bags. Provide each value on a new line, sorted by beans-in-bag."}], "ideal": "10: 8\n13: 1"} {"input": [{"role": "system", "content": "Given an unlimited supply of boxes with weights of 7, 8, 11, 12, 14, 15, 16, 17, 18, and 19, what is the unique combination of boxes that uses the minimum number of boxes to have a weight of exactly 113? Respond ONLY in the format : with NO OTHER INFORMATION and DO NOT include unused boxes. Provide each value on a new line, sorted by box-weight."}], "ideal": "18: 1\n19: 5"} {"input": [{"role": "system", "content": "Given an unlimited supply of boxes with weights of 5, 6, 9, 10, 11, 13, 14, and 15, what is the unique combination of boxes that uses the minimum number of boxes to have a weight of exactly 164? Respond ONLY in the format : with NO OTHER INFORMATION and DO NOT include unused boxes. Provide each value on a new line, sorted by box-weight."}], "ideal": "14: 1\n15: 10"} {"input": [{"role": "system", "content": "Given an unlimited supply of bean bags that contain 4, 6, 10, or 14 beans, what is the unique combination of bean bags that uses the maximum number of bags to have exactly 42 beans? Respond ONLY in the format : with NO OTHER INFORMATION and DO NOT include unused bean bags. Provide each value on a new line, sorted by beans-in-bag."}], "ideal": "4: 9\n6: 1"} {"input": [{"role": "system", "content": "Given an unlimited supply of coins in denominations of 5, 9, 11, and 17, what is the unique combination of coins that uses the minimum number of coins to have a value of exactly 79? Respond ONLY in the format : with NO OTHER INFORMATION and DO NOT include unused coins. Provide each value on a new line, sorted by denomination."}], "ideal": "11: 1\n17: 4"} {"input": [{"role": "system", "content": "Given an unlimited supply of coins in denominations of 3, 4, 7, 9, 15, 18, and 19, what is the unique combination of coins that uses the maximum number of coins to have a value of exactly 19? Respond ONLY in the format : with NO OTHER INFORMATION and DO NOT include unused coins. Provide each value on a new line, sorted by denomination."}], "ideal": "3: 5\n4: 1"} {"input": [{"role": "system", "content": "Given an unlimited supply of bean bags that contain 6, 18, or 19 beans, what is the unique combination of bean bags that uses the minimum number of bags to have exactly 189 beans? Respond ONLY in the format : with NO OTHER INFORMATION and DO NOT include unused bean bags. Provide each value on a new line, sorted by beans-in-bag."}], "ideal": "18: 1\n19: 9"} {"input": [{"role": "system", "content": "Given an unlimited supply of coins in denominations of 4, 6, 14, and 17, what is the unique combination of coins that uses the maximum number of coins to have a value of exactly 46? Respond ONLY in the format : with NO OTHER INFORMATION and DO NOT include unused coins. Provide each value on a new line, sorted by denomination."}], "ideal": "4: 10\n6: 1"} {"input": [{"role": "system", "content": "Given an unlimited supply of bean bags that contain 3, 5, 9, 10, 12, 13, 14, or 18 beans, what is the unique combination of bean bags that uses the maximum number of bags to have exactly 20 beans? Respond ONLY in the format : with NO OTHER INFORMATION and DO NOT include unused bean bags. Provide each value on a new line, sorted by beans-in-bag."}], "ideal": "3: 5\n5: 1"} {"input": [{"role": "system", "content": "Given an unlimited supply of bean bags that contain 5, 6, 7, 8, 10, 13, 18, or 19 beans, what is the unique combination of bean bags that uses the maximum number of bags to have exactly 51 beans? Respond ONLY in the format : with NO OTHER INFORMATION and DO NOT include unused bean bags. Provide each value on a new line, sorted by beans-in-bag."}], "ideal": "5: 9\n6: 1"} {"input": [{"role": "system", "content": "Given an unlimited supply of boxes with weights of 5, 9, and 13, what is the unique combination of boxes that uses the maximum number of boxes to have a weight of exactly 49? Respond ONLY in the format : with NO OTHER INFORMATION and DO NOT include unused boxes. Provide each value on a new line, sorted by box-weight."}], "ideal": "5: 8\n9: 1"} {"input": [{"role": "system", "content": "Given an unlimited supply of boxes with weights of 4, 6, and 11, what is the unique combination of boxes that uses the minimum number of boxes to have a weight of exactly 83? Respond ONLY in the format : with NO OTHER INFORMATION and DO NOT include unused boxes. Provide each value on a new line, sorted by box-weight."}], "ideal": "6: 1\n11: 7"}

You can view, comment on, or merge this pull request online at:

https://github.com/openai/evals/pull/421 Commit Summary

deeb558 https://github.com/openai/evals/pull/421/commits/deeb558beb61dd3b21e9f6a94c886e8395912135 Add eval to registry

30cbe66 https://github.com/openai/evals/pull/421/commits/30cbe6626d62571c469afff6a5fb2d3d33452c9c Add samples.jsonl

File Changes

(2 files https://github.com/openai/evals/pull/421/files)

A evals/registry/data/unique_combinations/samples.jsonl https://github.com/openai/evals/pull/421/files#diff-a88aed42e38a853a81022512c3069b2dbe66f631af71756a9eaf06cbc53a3ca5 (3)

A evals/registry/evals/unique_combinations.yaml https://github.com/openai/evals/pull/421/files#diff-1d0991d47a431dffb6c3d47996966f72cddd2411ef34cfd29017e888cc40832e (8)

Patch Links:

https://github.com/openai/evals/pull/421.patch

https://github.com/openai/evals/pull/421.diff

— Reply to this email directly, view it on GitHub https://github.com/openai/evals/pull/421, or unsubscribe https://github.com/notifications/unsubscribe-auth/A6PJHHWKWDYM33KGYVQRMY3W5SX7NANCNFSM6AAAAAAWFWU6SM . You are receiving this because you are subscribed to this thread.Message ID: @.***>

Mar 23 '23 22:03 SierotkaM

Thanks for the contribution. I would like to request some changes.

I would recommend using the Includes or ModelGraded evaluation method here because arithmetic and other complex calculations are hard for the model to do zero-shot without a chance to reason through the steps. Asking the model to provide reasoning will give the model a fair chance to solve the question, and a proper evaluation method can be used to evaluate the provided answer. If you want to use Includes, you can ask the model to format the final answer in a specific format, as you are doing in the case of the Match method.

Add a proper description of the eval in the .yaml file.

We would love to review the PR again after the suggested changes.

Just fixed those things, let me know if there's anything else you'd like to see. Thanks for looking at it!

May 19 '23 19:05 Sypherd

I think you forgot the following step:

Asking the model to provide reasoning will give the model a fair chance to solve the question.

The purpose of using Includes is to change the prompt and ask model to give the reasoning along with the answer.

May 19 '23 20:05 usama-openai

Oh, thank you! I did miss that. That makes sense! I updated the prompt to allow it to reason. I reran the eval and still see similarly poor performance from the models on this task.

May 22 '23 13:05 Sypherd

evals evals copied to clipboard

Unique Combinations with Constraints

Thank you for contributing an eval! ♥️

Eval details 📑

Eval name

Eval description

What makes this a useful eval?

Criteria for a good eval ✅

Unique eval value

Eval structure 🏗️

Final checklist 👀

Submission agreement

Email address validation

Limited availability acknowledgement

Submit eval

Eval JSON data

Eval

evals
evals copied to clipboard