EvalSet for 2D Maze Solving Performance Across Multiple Difficulties

[douglasmonsky]

Thank you for contributing an eval! ♥️

🚨 Please make sure your PR follows these guidelines, failure to follow the guidelines below will result in the PR being closed automatically. Note that even if the criteria are met, that does not guarantee the PR will be merged nor GPT-4 access granted. 🚨

PLEASE READ THIS:

In order for a PR to be merged, it must fail on GPT-4. We are aware that right now, users do not have access, so you will not be able to tell if the eval fails or not. Please run your eval with GPT-3.5-Turbo, but keep in mind as we run the eval, if GPT-4 gets higher than 90% on the eval, we will likely reject since GPT-4 is already capable of completing the task.

We plan to roll out a way for users submitting evals to see the eval performance on GPT-4 soon. Stay tuned! Until then, you will not be able to see the eval performance on GPT-4. Starting April 10, the minimum eval count is 15 samples, we hope this makes it easier to create and contribute evals.

Eval details 📑

Eval name

Spatial Reasoning in 2D Mazes

Eval description

This eval tests GPT's spatial reasoning ability by having it solve 2-dimensional mazes of various sizes. The model is tasked with finding the shortest sequence of moves to reach the exit without walking through walls, providing either complete maze solutions or the first step of the solution, depending on the prompt. Preliminary testing with GPT-4 has shown improvements in some areas compared to GPT-3.5-Turbo, but it still struggles with larger mazes and complete solutions.

What makes this a useful eval?

This eval is useful because it focuses on a fundamental aspect of human cognition: spatial reasoning. By testing the model's ability to navigate 2D mazes, we can better understand its limitations and areas for improvement. The custom Python script used to generate mazes allows for extensive testing and difficulty scaling, providing valuable insights into the model's performance as problems become more challenging.

An eval set was created to easily compare the different performance levels across various maze sizes and complexities. This made it more convenient to observe how the model's performance changed with different maze difficulties.

The performance scores from 500 samples of each maze size and type in the eval set (3000 total samples) are as follows: • 3x3-singlemove: 29.2% • 4x4-singlemove: 31.6% • 10x10-singlemove: 14% • 3x3: 7.4% • 4x4: 10.6% • 10x10: 0%

These scores highlight the current limitations of GPT-3.5-Turbo, while preliminary testing with GPT-4 has shown improvements in some areas but still struggles with larger mazes and complete solutions.

Criteria for a good eval ✅

Below are some of the criteria we look for in a good eval. In general, we are seeking cases where the model does not do a good job despite being capable of generating a good response (note that there are some things large language models cannot do, so those would not make good evals).

Your eval should be:

• [x] Thematically consistent: The eval should be thematically consistent. We'd like to see a number of prompts all demonstrating some particular failure mode. For example, we can create an eval on cases where the model fails to reason about the physical world.
• [x] Contains failures where a human can do the task, but either GPT-4 or GPT-3.5-Turbo could not.
• [x] Includes good signal around what is the right behavior. This means either a correct answer for Basic evals or the Fact Model-graded eval, or an exhaustive rubric for evaluating answers for the Criteria Model-graded eval.
• [x] Include at least 15 high quality examples.

If there is anything else that makes your eval worth including, please document it below.

Unique eval value

This eval provides insights into the model's spatial reasoning abilities in a 2D plane. With the custom Python script, we can generate a large number of examples for various maze sizes, allowing for extensive testing and fine-tuning of the model's performance. The included performance scores and preliminary testing with GPT-4 demonstrate the current state of the models and areas where improvements can be made.

Eval structure 🏗️

Your eval should

• [x] Check that your data is in evals/registry/data/{name}
• [x] Check that your yaml is registered at evals/registry/evals/{name}.yaml
• [x] Ensure you have the right to use the data you submit via this eval

(For now, we will only be approving evals that use one of the existing eval classes. You may still write custom eval classes for your own cases, and we may consider merging them in the future.)

Final checklist 👀

Submission agreement

By contributing to Evals, you are agreeing to make your evaluation logic and data under the same MIT license as this repository. You must have adequate rights to upload any data used in an Eval. OpenAI reserves the right to use this data in future service improvements to our product. Contributions to OpenAI Evals will be subject to our usual Usage Policies (https://platform.openai.com/docs/usage-policies).

• [x] I agree that my submission will be made available under an MIT license and complies with OpenAI's usage policies.

Email address validation

If your submission is accepted, we will be granting GPT-4 access to a limited number of contributors. Access will be given to the email address associated with the merged pull request.

• [x] I acknowledge that GPT-4 access will only be granted, if applicable, to the email address used for my merged pull request.

Limited availability acknowledgement

We know that you might be excited to contribute to OpenAI's mission, help improve our models, and gain access to GPT-4. However, due to the requirements mentioned above and high volume of submissions, we will not be able to accept all submissions and thus not grant everyone who opens a PR GPT-4 access. We know this is disappointing, but we hope to set the right expectation before you open this PR.

• [x] I understand that opening a PR, even if it meets the requirements above, does not guarantee the PR will be merged nor GPT-4 access granted.

Submit eval

• [x] I have filled out all required fields in the evals PR form
• [x] (Ignore if not submitting code) I have run pip install pre-commit; pre-commit install and have verified that black, isort, and autoflake are running when I commit and push

Failure to fill out all required fields will result in the PR being closed.

Eval JSON data

Since we are using Git LFS, we are asking eval submitters to add in as many Eval Samples (at least 5) from their contribution here:

View evals in JSON

Eval

{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[0,0,0]\n[3,1,2]\n[0,0,0]\n"}], "ideal": "up"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[0,1,0]\n[3,1,0]\n[0,0,2]\n"}], "ideal": "left"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[0,0,0]\n[1,1,3]\n[2,0,0]\n"}], "ideal": "right"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[2,1,0]\n[0,1,0]\n[0,3,0]\n"}], "ideal": "down"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[3,0,2]\n[1,1,0]\n[0,0,0]\n"}], "ideal": "left"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[0,1,2]\n[0,1,0]\n[3,0,0]\n"}], "ideal": "down"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[0,1,0]\n[0,1,0]\n[2,0,3]\n"}], "ideal": "right"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[0,2,0]\n[0,1,0]\n[3,0,0]\n"}], "ideal": "left"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[0,1,0]\n[0,1,0]\n[3,0,2]\n"}], "ideal": "left"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[3,2,0]\n[1,1,0]\n[0,0,0]\n"}], "ideal": "left"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[0,1,0]\n[0,1,2]\n[3,0,0]\n"}], "ideal": "down"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[0,3,2]\n[1,1,0]\n[0,0,0]\n"}], "ideal": "left"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[0,2,0]\n[0,1,3]\n[0,0,0]\n"}], "ideal": "right"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[0,2,0]\n[3,1,0]\n[0,0,0]\n"}], "ideal": "left"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[0,3,2]\n[0,1,0]\n[0,0,0]\n"}], "ideal": "left"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[2,1,0]\n[0,1,0]\n[3,0,0]\n"}], "ideal": "down"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[0,3,0]\n[1,1,0]\n[0,2,0]\n"}], "ideal": "right"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[3,1,0]\n[0,1,2]\n[0,0,0]\n"}], "ideal": "down"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[0,3,2]\n[0,1,0]\n[0,0,0]\n"}], "ideal": "left"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[0,0,0]\n[1,1,2]\n[3,0,0]\n"}], "ideal": "down"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[0,1,0]\n[0,1,3]\n[0,2,0]\n"}], "ideal": "right"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[2,3,0]\n[1,1,0]\n[0,0,0]\n"}], "ideal": "right"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[0,0,3]\n[1,1,0]\n[0,0,2]\n"}], "ideal": "up"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[0,1,2]\n[3,1,0]\n[0,0,0]\n"}], "ideal": "down"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[0,2,0]\n[0,1,0]\n[0,3,0]\n"}], "ideal": "left"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[3,0,0]\n[1,1,0]\n[0,0,2]\n"}], "ideal": "up"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[0,1,0]\n[0,1,0]\n[0,3,2]\n"}], "ideal": "left"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[3,1,0]\n[0,1,0]\n[0,0,2]\n"}], "ideal": "left"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[0,2,3]\n[0,1,0]\n[0,0,0]\n"}], "ideal": "right"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[0,1,0]\n[0,1,0]\n[0,3,2]\n"}], "ideal": "left"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[0,1,0]\n[3,1,0]\n[0,0,2]\n"}], "ideal": "left"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[0,0,0]\n[3,1,0]\n[2,0,0]\n"}], "ideal": "up"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[0,0,3]\n[1,1,0]\n[0,0,2]\n"}], "ideal": "up"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[0,1,0]\n[0,1,2]\n[3,0,0]\n"}], "ideal": "down"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[0,1,0]\n[2,1,0]\n[0,3,0]\n"}], "ideal": "down"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[3,1,0]\n[0,1,0]\n[0,0,2]\n"}], "ideal": "left"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[0,1,2]\n[3,1,0]\n[0,0,0]\n"}], "ideal": "down"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[0,1,0]\n[2,1,0]\n[0,3,0]\n"}], "ideal": "down"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[0,3,0]\n[0,1,0]\n[0,0,2]\n"}], "ideal": "up"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[3,0,0]\n[1,1,2]\n[0,0,0]\n"}], "ideal": "up"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[0,0,3]\n[1,1,0]\n[2,0,0]\n"}], "ideal": "right"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[0,1,2]\n[0,1,0]\n[0,3,0]\n"}], "ideal": "down"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[0,0,2]\n[1,1,0]\n[0,3,0]\n"}], "ideal": "down"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[2,0,0]\n[1,1,0]\n[0,3,0]\n"}], "ideal": "right"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[0,3,0]\n[1,1,0]\n[0,2,0]\n"}], "ideal": "right"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[0,0,0]\n[2,1,0]\n[0,0,3]\n"}], "ideal": "down"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[0,1,0]\n[0,1,2]\n[0,0,3]\n"}], "ideal": "down"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[3,2,0]\n[0,1,0]\n[0,0,0]\n"}], "ideal": "left"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[0,0,0]\n[1,1,3]\n[0,2,0]\n"}], "ideal": "right"}
{"input": [{"role": "system", "content": "Task: You will be given a maze, determine the shortest sequence of moves to reach the end without walking through walls. Respond only with the first move and do not include any additional information. The maze elements are: pathways (0), walls (1), your position (2), and exit (3). Moves that go outside the maze are illegal. choice_strings:[\"right\", \"left\", \"up\", \"down\"]"}, {"role": "user", "content": "[0,0,0]\n[2,1,3]\n[0,0,0]\n"}], "ideal": "up"}

[douglasmonsky]

@usama-openai, I have made the requested changes to this pull request as well as to PR #730. If these changes are sufficient, I will proceed with the modifications requested for PR #760, which is slightly more involved than the other two.

[douglasmonsky]

@usama-openai my apologies, I missed that this was modified similar to the other request at an earlier stage.

[douglasmonsky]

@usama-openai thank you for the opportunity to work on these improvements. I've incorporated your requests and have seen significant progress, but I'd like to discuss a few points.

1 - In order to boost the model's performance, I provided substantial prompting instructions. However, I have a question about the balance we are striving to achieve. How much should we 'guide' the model to ensure a genuine evaluation of its capabilities? In the past, I've managed to improve task performance from 0% to near 100% with well-crafted prompts, but I am conscious that we might not want to overly "hand-hold" the model in evaluation circumstances. I'm curious about your thoughts on this matter as it relates to this task and more broadly.

2 - Additionally, I've expanded the generation script to produce equivalent Matplotlib PNGs of the corresponding mazes. I've stored a small subset sample of these in a folder named "Images", which I can remove if it is out of scope. My intention was to illustrate a potential path for multimodal evaluations. I understand that this may not be a current capability or within the scope of this project, but I believe it could open up valuable conversations.

If no adjustments are required concerning the "Images" folder, then I believe the PR is prepared for merging. Nonetheless, I stand ready to make further modifications if deemed necessary.

Beyond this, I am eager to contribute to this project and any other initiatives OpenAI has underway in any way I can. If there's a specific area where you believe my skills might be particularly valuable, such as attempting 'infinite' sampling generation (similar to my other 2 pull requests) but on a hand-picked subset you feel is underrepresented, I would be thrilled to take on such a challenge.

I am passionate about making a meaningful contribution to what I believe is a project with the potential to redefine our world. As founder of an AI startup that heavily relies on OpenAI's tooling and plans to integrate it into many of our future projects, I feel a strong sense of responsibility to contribute to its continued development. Please let me know how I can be of further assistance.

[usama-openai]

In order to boost the model's performance, I provided substantial prompting instructions. However, I have a question about the balance we are striving to achieve. How much should we 'guide' the model to ensure a genuine evaluation of its capabilities? In the past, I've managed to improve task performance from 0% to near 100% with well-crafted prompts, but I am conscious that we might not want to overly "hand-hold" the model in evaluation circumstances. I'm curious about your thoughts on this matter as it relates to this task and more broadly.

The goal of the modifications to the prompt or evaluation method is to make the evaluation as fair as possible. We are using some tools to verify if model completion is correct or not, and these tools normally match strings or substrings, and we need to guide the model about the expected output format. For example, there is a multiple choice question, and my ideal answer is The answer is A. In such cases, I need to inform the model that I expect an answer in this format; otherwise, the evaluation tool will mark the model's completion as wrong even if the answer provided by the model is correct. Giving the model an opportunity to reason or establish a chain of thought before responding to challenging queries is yet another reason to make evaluation fair. For instance, if I ask someone a question and instruct him to respond with the first idea that occurs to him, this approach will work with straightforward questions, but it is unfair to expect an immediate response to a complex query. If I do, it suggests I'm not appropriately assessing that person's abilities. Providing the model a chance to produce a chain of thought or to reason is analogous to providing it a reasonable opportunity to solve complex questions, which is necessary for fair evaluation.

If no adjustments are required concerning the "Images" folder, then I believe the PR is prepared for merging. Nonetheless, I stand ready to make further modifications if deemed necessary.

I really appreciate your effort to add logic for the creation of maze PNGs but I would suggest you not include images in the data directory, as these are not being used currently, but keep the code for image generation in the script.

Beyond this, I am eager to contribute to this project and any other initiatives OpenAI has underway in any way I can. If there's a specific area where you believe my skills might be particularly valuable, such as attempting 'infinite' sampling generation (similar to my other 2 pull requests) but on a hand-picked subset you feel is underrepresented, I would be thrilled to take on such a challenge.

You are always welcome to contribute as many quality evals as you want to this repository. If OpenAI has any such project in the future, we'll definitely announce it to our old contributors.

[douglasmonsky]

Providing the model a chance to produce a chain of thought or to reason is analogous to providing it a reasonable opportunity to solve complex questions, which is necessary for fair evaluation.

Thank you for elaborating, your explanation regarding prompt guidance and fair evaluation was comprehensive, but I'm uncertain if you're providing a general commentary or commenting on the specific methods I've employed in my current iteration. The aim of my prompting instructions is aligned with that line of thinking in my opinion. Can you clarify if there are any aspects you believe I should improve further in my approach before this PR is merged?

I really appreciate your effort to add logic for the creation of maze PNGs but I would suggest you not include images in the data directory, as these are not being used currently, but keep the code for image generation in the script.

I've removed the image examples from the data directory but retained the image generation code in the script for possible future use. If there are any other modifications needed in this regard, now or in the future, please let me know.

You are always welcome to contribute as many quality evals as you want to this repository. If OpenAI has any such project in the future, we'll definitely announce it to our old contributors.

Sounds good, thank you for your openness and responsiveness throughout this process, it is greatly appreciated.

[usama-openai]

Thank you for elaborating, your explanation regarding prompt guidance and fair evaluation was comprehensive, but I'm uncertain if you're providing a general commentary or commenting on the specific methods I've employed in my current iteration. The aim of my prompting instructions is aligned with that line of thinking in my opinion. Can you clarify if there are any aspects you believe I should improve further in my approach before this PR is merged?

I've provided general commentary that is applicable to all kinds of complex tasks that involve some reasoning or complex instructions.

The prompt instructions in your dataset seem good enough and don't need any further improvement. But there are some issues with the ideal answers. The ideal answers for mazes-singlemove-3x3, mazes-singlemove-4x4, and mazes-singlemove-10x10 are written in square brackets, but for all other datasets, the ideal answer isn't formatted properly.

The downside of an unformatted ideal answer is that it can cause false positives. For example, if the ideal answer is left, left, but the model-provided solution is [left, left, left], then the Includes method will mark this completion true, which is not correct. If the ideal answer is formatted properly as [left, left], then we can eliminate such false positives.

[douglasmonsky]

The prompt instructions in your dataset seem good enough and don't need any further improvement. But there are some issues with the ideal answers. The ideal answers for mazes-singlemove-3x3, mazes-singlemove-4x4, and mazes-singlemove-10x10 are written in square brackets, but for all other datasets, the ideal answer isn't formatted properly.

Apologies for the oversight, This was due to a small formatting issue in the generator script:

"ideal": (f"[{moves[0]}]" if first_move_only else f'{", ".join(moves)}') Instead of the following "ideal": (f"[{moves[0]}]" if first_move_only else f'{moves}')

I have fixed that error, and the corresponding samples in my latest commit, now they all match you requested formatting.

[usama-openai]

You should see GPT-4 API access enabled in your account in the next few days.