“Ask for feedback” step.

[AntonOsika]

Create a step that asks “did it run/work/perfect”?, and store to memory folder.

And let the benchmark.py script check that result, and convert it to a markdown table like benchmark/RESULTS.md , and append it with some metadata to that file.

[inspire99]

I would like to work on this, could you assign to me ?

[patillacode]

Go for it @inspire99

No need to assign it, you can just pick it up and come back with a PR!

[andrewleenyk]

When i try to run the benchmark.py `Projects/gpt-engineer/scripts/benchmark.py", line 78, in run(main)

RuntimeError: Type not yet supported: int | None` my py version is: python 3.11

Edit: temporary work around: n_benchmarks: Optional[int] = None,

[SumitKumarDev10]

I think, we can use MongoDB as the database to save the results. Since MongoDB is flexible and scalable, it would make as a great DBMS system. Storing the database on the local file system is ok but it may get deleted if the library is uninstalled or the local file system gets formatted. So I think MongoDB Atlas can be used to store all the data in a centrally accessible repo on Github. However, we face yet another issue. If we save this in one place it would become confusing to know on whose system did the library run well and what was their review and rating. So it would be better if everybody using the library must have an Account made through the web interface on gpt-engineer. This would help track who had problems with the library and who were able to run their code successfully. They can generate their own API Key to run the library with the OpenAI API Key.

[mwzhu]

Is this still an open issue? I saw that the feedback is already stored in the memory folder under "review," would converting that raw data into a markdown table be helpful? If so I can work on that!

[AntonOsika]

Good job on this!