Development not completed after all the work is done.

[d9vsky]

Hi, I had no issues during the run for this statement: metagpt@0833f6fb03f7:/app/metagpt$ python startup.py "create a snake game that will be in a separate window and then snake will be in orange colour. When snake hits the wall the game is over."

MetaGPT created everything that is necessary and that was in its "thinking flow":

However, when I try to run the game by "python main.py" it opens a command line window for a split second and closes it. Nothing else is going on.

Can you please try it on your end with the same statement as I put?

[geekan]

https://github.com/geekan/MetaGPT/releases/tag/v0.1.0 released a snake game. which prompt is "snake game".

complex prompt may led the program think less, I wonder. so complex prompt may need more desc.

[d9vsky]

Thanks for your answer geekan. I used three different prompts to see what kind of outcomes it will produce, below I am placing the outcomes of all three of them for your reference. The more specific the prompt is, the more problems the MetaGPT occurs, I think. Writing a snake game is a simple task, but what about more complex programs? How to pass properly the specification of the program for MetaGPT to correctly pick it up and implement?

1. python startup.py "Write a snake game that will appear in a window. There should be a START button which starts the game and QUIT button which exits the game completely. The snake should be in orange color and when it hit the wall the game is over. The rest of snake mechanics should be as in a normal snake game. Additionally, never close the game window until the QUIT button is clicked. The actual score of the player should be shown in upper left corner" snake_game.zip

2. python startup.py "write a python snake game" py_snake_game.zip

3. python startup.py "write cli snake game" cli_snake_game.zip

[geekan]

@d9vsky The game generated by MG is so entertaining that it had me laughing for several minutes. It's truly amusing how the AI followed our prompts, but often it strictly adhered to our instructions — seeming to meet expectations, but lacking fundamental functionality. I plan to address this issue in two ways: one is to allow users to save and load, then incrementally modify after completing the requirements (much like a boss); the second is to introduce a proper review phase, enabling an agent to conduct the review and align as much as possible with human expectations. The essence of this problem is the lack of alignment with human (latent) intentions, something that OpenAI has certainly never encountered in the work we're doing.

[geekan]

@d9vsky Of course, the problem could also lie in our SOP not being quite right. My current SOP utilizes Elon Musk's five-step process method (https://www.youtube.com/watch?v=hhuaVsOAMFc&ab_channel=ColinHickey). The first step is to make the requirements less dump, the second step is to trim away useless branches, and the third step is to simplify and optimize. This should all result in the LLM designing as minimally as possible, with a bit of modification, it should be able to improve. Indeed, human SOPs may not necessarily be suitable for LLMs.

[d9vsky]

@geekan, thank you very much for the replies. Now it's very much clearer. Yeah, I was amused too, when it implemented all the requirements, but the snake was literally flying on the screen. MEANING, that a small portion of development in the existing project could really lead to a successful closure of the development! That got me thinking - if you could implement a possibility to give feedback after human testing and MetaGPT could implement changes in previously developed project, then you could have a much more precise development, BUT with some human interactions. On the otherhand, MetaGPT could leverage from writing tests for itself, because if you can write a test for a functionality - then you should clearly be able to develop it ;-)

That Elon's 5-step method is very interesting and I see a huge possibility of success when combined with self-testing and/or human-testing.

Also, what do you think about giving the requirements to the MetaGPT from a file?

I look forward to the next versions!