Dynamically operate a browser - for testing a webpage

[bjshumway]

Hey, so I have a game built from HTML/Javascript. I would like to see how well Baby AGI could try to understand and then modify the existing game. Make changes... re-run the game... play it for finding bugs, etc.

For this it will need to learn how to press buttons, type text into fields, and

1. View the existing page with vision And/or
2. Parse the page, over-and-over again, as it takes its actions, to understand what is on the page.

How might this be achieved? What plugin(s) could we pull in or other API's to link up to?

[Macx2]

Alright sure if could be fine of it And how many game's built?

On Wed, Apr 5, 2023, 5:46 PM Benjamin Shumway < @.***> wrote:

Hey, so I have a game built from HTML/Javascript. I would like to see how well JARVIS could try to understand and then modify the existing game. Make changes... re-run the game... play it for finding DuckDuckGo removed 1 tracker. More →

https://duckduckgo.com/-omzpLmBld7WG6Tbea1OXCCuA4yqZ_9v8IiBcgnmv1bqbPzepO5_b153Qvu8uV3yJhY64p0FyuF9W8AvUYfIsHCcLjnkl2IQuvCjzRAnwScmgIPzFQ085nGGR9UiEtd9nBT06OcZAmPMeB2FBS0eXfVl7_V2Do96-HwJjTvtad6FP0xqdGeoKvETcqgHEyfyCEFKSivdZyHTDjIdT_bMP_E48KkFTOsnIoE138ffm8j4psyMsDg

Hey, so I have a game built from HTML/Javascript. I would like to see how well JARVIS could try to understand and then modify the existing game. Make changes... re-run the game... play it for finding bugs, etc.

For this it will need to learn how to press buttons, type text into fields, and

1. View the existing page with vision And/or
2. Parse the page, over-and-over again, as it takes its actions, to understand what is on the page.

How might this be achieved? What plugin(s) could we pull in or other API's to link up to?

— Reply to this email directly, view it on GitHub https://github.com/yoheinakajima/babyagi/issues/44, or unsubscribe https://github.com/notifications/unsubscribe-auth/A67U5U4F7X4ZQ4MKWSK7UX3W7WOWBANCNFSM6AAAAAAWUKFSR4 . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[bjshumway]

I'm just limiting it to my game. I have a backlog of features that need to be added. I'd like to see how well something like this could perform if we allow it to iterate.

Steps:

1. Read code, try to understand it
2. Run game, try to understand it
3. If it doesn't understand the game, its code, and how the two line up... loop back to step 1
4. Look at list of features and pick whatever not built yet
5. Make changes to code
6. run game, test changes... repeat step 5 until all changes are confirmed working properly
7. If everything is built and properly tested, got to step 8, otherwise loop back to step 4
8. Shut down

• At various points, if there's something it can't figure out... pause the code and prompt the end user (me) for clarification.

[Macx2]

Sure ! how long does it take to add the backlog of features Ready..

On Wed, Apr 5, 2023, 5:54 PM Benjamin Shumway < @.***> wrote:

I'm just limiting it to my game. I have a backlog of features that need to be added. I'd like to see how well something like this could perform if we allow it to iterate. Iterations: Run game, underst DuckDuckGo removed 1 tracker. More →

https://duckduckgo.com/-3KTufG5fd0L7vrtc8cUXBuKZBsnhflnBL1CHxbNwnSwQ80qwxRUuzolMRSUtr38ztmzOfMkjz7qlV8b-kZV7S1Fs3xFVazLi2X2R2tj1vbbedn62U5sBPpt2uMsPvVazG3N5uNiVUz-RHTEgEbdqAPG7D5EsWJKqRy4UCjbjDVX_7IMEFwc0yp_SSUasNt3F35vL-yTSzrA4

I'm just limiting it to my game. I have a backlog of features that need to be added. I'd like to see how well something like this could perform if we allow it to iterate.

Iterations: Run game, understand it... make changes... run game, test changes... etc. etc.

— Reply to this email directly, view it on GitHub https://github.com/yoheinakajima/babyagi/issues/44#issuecomment-1497824254, or unsubscribe https://github.com/notifications/unsubscribe-auth/A67U5U5ONCJU5YB6HLOSHNDW7WPUTANCNFSM6AAAAAAWUKFSR4 . You are receiving this because you commented.Message ID: @.***>

[bjshumway]

Well we could test just a few small features first to see how well it performs.

A small thing would be: Add a "Gender" field to the user's character. When the user creates a character this field will say "no preference", but a dropdown list will allow you to choose "Male", "Female", or "Other". When the user clicks a button called "Describe my character" (which currently runs a GPT prompt to create the character's description)... include in the GPT prompt the character's gender so that GPT knows to describe the character using the proper pronuon.

A big thing would be: Implement a leaderboard that pulls the information from the game's cloud database. Create a button on the main (home) screen to view the leaderboard. When user's ratings change, update the leaderboard as in the cloud.

[bjshumway]

Oh wow,

I found a chrome extension called "TaxyAI" that allows GPT4 to interact with the browser.

https://github.com/TaxyAI/browser-extension#how-it-works---the-action-cycle

Could TaxyAI be integrated into Auto-GPT?