Define an architecture & support multiple project

[ph-ausseil]

EDIT : UPDATE : 26/04 15:10 CET

• New PR #3316
• Definition of 3 levels architecture

EDIT : UPDATE : 25/04 18:40 CET

• Added clarification as #1937 & #3316 are actualy very different

EDIT : UPDATE : 25/04 13:00 CET

• New yaml format supported (shown below)
• Code AIConfig & new yaml format to support recursion following @ntindle idea
• Change not published , I will commit this evening
• Went from 2/11 Task to 4/9 task

Idea

In a 3 layers architecture where you have :

• Level 1 : Management of project / instances The capacity to create different project with different targets (example build a website, plan a travel, plan a wedding & do my shopping)
• Level 2 : Management of team of Agents : The capacity to operate different agent willing to work on the same project(example build a website)
• Level 3 : Agent The capacity to operate with a AI Model to perform a certain set of task/logical operations.

Discussion : Identify which of these changes must be "core" & which of these change can be supported via plug-ins.

The PR associated # 3316 cover set first level of architecture & enable the second one.

Dev Branches

The code should respect the coding conventions I have observed :

• This branch has memory issues : https://github.com/ph-ausseil/Auto-GPT/tree/multiple-config
• This branch makes the choice to break the memory framework: you will have an exception running it_ https://github.com/ph-ausseil/Auto-GPT/tree/multiple-config-memory I will focus on LocalCache & maybe Pinecone.

Proof of Concept/illustrations:

This is an example of output :

Plan

• [x] Get a proof of concept
• [x] Present different ai_settings.yaml
• [ ] Get green lights
• [ ] Define how much memory should be supported
• [ ] Implement memory compatibility update : 25/04 workaround found
• [ ] Implement tests
• [ ] Get support for E2E tests (as GPT-3.5 limit scenarios)

Optional :

• [x] Split AIConfig in two
• Implement migration of ai_settings.yaml
• Implement migration of memories
• [x] Implement support for one of the multiple instances of suggestions

ai_settings.yaml

Current file structure

ai_goals:
- 'You will make a detailed summary of each file located in "./python_doc/" as if
  you were an engineer willing to teach another engineer on a new concept. The summary
  must be detailed and contains all subtilities required. Then each summary must be
  saved in a file located in the folder "./python_sumary". '
- Analyze the code in the ./python_doc/ folder and provide suggestions for improvements and optimizations
- Create a list of frequently asked questions (FAQs) related to the content in the ./python_doc/ folder
ai_name: newPythonGPT
ai_role: Produce high-quality and well-detailed documentation and tutorial for Python  Developers

New file structure :

version: X.Y.X
projects : 
 - project : 
   project_name : MyProjectName
   lead_agent :
     agent_goals: 
        - Increase net worth
        - Grow Twitter Account
        - Develop and manage multiple businesses autonomously
      agent_role: an AI designed to autonomously develop and run businesses with the sole goal of increasing your net worth.
      agent_name: Entrepreneur-GPT
      agent_model: A model
   delegated_agents :
    - agent_goals: 
      - There are files located in the folder ./python_doc/ , I want you to go through them and produce a detailed summary aimed at engineers and save these summaries in files located in the folder ./python_summary
      - Analyze the code in the ./python_doc/ folder and provide suggestions for improvements and optimizations
      - Create a list of frequently asked questions (FAQs) related to the content in the ./python_doc/ folder
      agent_name: PythonGPT
      agent_role: Produce high-quality and well-detailed documentation and tutorial for Python Developers 
      agent_model: A model
      agent_model_type: A model type
 - project : 
   project_name : MyOtherProjectName
   lead_agent :
      agent_goals: 
        - Everything non-scrum is bad
        - Let others feel bad
        - Never admit you are wrong
      agent_name: ScrumGPT
      agent_role: Advocate for scrum on Twitter like a 13y/o keyboard warrior 

Originally posted by @ph-ausseil in https://github.com/Significant-Gravitas/Auto-GPT/discussions/922#discussioncomment-5699513

Also, in order to do my developments, GPT-4 API access would really help if any of you have a contact that may help :). I've been on the waiting list for a very long time :(

[ph-ausseil]

Status :

Encountering issues related to LLM, but tests are ok, except for the issues below.

Major Issue :

• As it is, memory seems broken
• Need team feedback/green lights

Minor issue :

Migration of ai_settings.yaml

The existing yaml will not be backward compatible as is; it can be fixed :

if the versions don't exist
execute transform_config_v0_to_VX()
if the version is between VY  and  VX 
execute transform_config_VY_to_VX()

Migration of Cache

The existing memory will not be backward compatible as is; it can be fixed :

if the versions don't exist
execute transform_cache_v0_to_VX()
if the version is between VY  and  VX 
execute transform_cache_VY_to_VX()

GPT-4 E2E test

I many of my test scenarios can’t go End-to-End because I do not have GPT-4 API.

Discussions :

• Is it wanted ? if wanted, can this be re-used for multiple instances? (I can work on it…) do we need retro compatibility on yaml & cache, or is it acceptable at this stage? if yes, do we need it on all cache systems? Or does the suggested functionality outweigh the requirement of having X+ cache solutions? (define X 😊)
• Can I split AIConfig in two? Not required for functionality but for modularity and clarity of the code. AIConfigList / AIConfigLoader (that contains a list of AIConfig) AIConfig
• Multiple-instance suggestions 1 & 2 show a need for planning The model needs to be defined today; it is though as two options slow/fast because it is though for LLM How to implement models that are not LLM (car plate recognition, image generation, plant strain recognition, Coding expertise, Mathematic expertise, etc...)

Lesson Learned :

Supporting too many memories as core functions (and not plug-ins) at this stage of the development of AutoGPT (before multiple instances, multiple configs, multiple LLM support...) of the software is not a good idea 😢

[ido777]

Can I split AIConfig in two, not required for functionality but for modularity and clarty of the code. ** AIConfigList / AIConfigLoader (that contains a list of AIConfig) ** AIConfig

IMO, it is required for functionality. From my perspective, this is one of the most important changes from the original design. Currently, the autoGPT has 2 "internal models" FAST_LLM, and SMART_LLM, and the PROMPT comes partly from code and partly from the user (in different forms)

In order to design multiple different bots from different models (and probably working on different goals) talking with each other it will help if we will

1. differentiate between different possible configurations (e.g. different system_promt (what you called prompt_generator) for different models, or even for different testing. what you called AIConfigList / AIConfigLoader
2. differentiate between different actual configurations, for multiple copies of the same configuration (e.g. to avoid duplicating of the same goal such as PythonGPT & PythonGPT2.
3. move the current connection between autoGPT internal logic and prompt generation so all the adaptions to different models won't hide in the code. moreover, it will ease testing and development

[ido777]

Question: How do you envision controlling PythonGPT & PythonGPT2, from duplicating efforts? (It seems their definition should be to do what the third agent will ask them)

[ph-ausseil]

Apr 24, 2023

It is not required, as show the screenshot the POC is working and the only issue at the moment is memory loss. But It would be none the less very easy to set & beneficial for code maintenability. At this stage I am waiting for contributor feedbacks.

: How do you envision controlling PythonGPT & PythonGPT2, from duplicating efforts? (It seems their definition should be to do what the third agent will ask them)

This is out of scope. . The implementation of one of the two suggestions Multiple-instance suggestion 1 & Multiple-instance suggestion 2 may set the foundation for the functionality you are refering to. If you want to have multiple instance you can implement it many way , but one of them is to upgrade ai_settings.yaml management.

Also, please example are just example 😄

[iller75]

This is kind of exactly what i suggested a day or two ago, awesome work :D

[ph-ausseil]

Hey, Is it possible to have any thoughs from the team @k-boikov @ntindle @Pwuts @Torantulino @BillSchumacher , I'm getting a real workload managing conflicts 😅

[ntindle]

Dont migrate the files, backward compatibility hasn't been considered until now, and I don't plan to start

[ntindle]

Multiple-instance suggestion 1: add: projects add: projects > project add: projects > project > lead_ai_name add: projects > project > ai add: projects > project > ai > model

Multiple-instance suggestion 2: add: projects add: projects > project add : projects > project > lead_ai add : projects > project > delegated_ai add: projects > project > delegated_ai > model

Have we considered a goal > objective > leader architecture?

We could so something like:

goal: Make more paperclips
Leader: 
 - model: GPT4
   name: THE PAPERCLIPPER
   objectives:
     - goal: Gather Materials
        leader: 
        - model: GPT3.5
          name: the miner
          objectives:
          - mine
      - goal: Build Factories
        leader:
        - model: GPT4
           name: Factory Construction Supervisor
           objectives: 
           - goal: Design Factory
             leader: GPT4
             objectives:
               - goal: Generate an image of a factory
                  leader: HF
           - goal: Get factory materials
             leader: GPT3.5
             . . .
           - goal: combine materials
            . . .
           - goal: finish factory
          . . . 

[ntindle]

I also don't love defining agents in this level of detail in yaml. feels like it wasn't designed for this

[ph-ausseil]

• Core model : Should we keep both model ? Should we push one one forward ? I think your Core Model may pose problem when the Agent is not interacting with a LLM. So I will definitively push my idea until proven otherwise.

The separation of projects and agents in my approach was decided to simplify the configs so that they could be transferred to other projects more easily, and to allow for the growth of the agent models.

It could be possible to combine the different models, i.e. have a project folder structure, then a single yaml to handle the projects and agents.

Hey , There is a bit more to that the difference is that :

• Via #1937 you can build interaction with each agent of the Project , also agent must be LLM but it is scalable
• Via #3039 you interact with a leader that does all the job of managing the other agents

It is why we provide response to similar needs but with two very different solution which are not mergeable, it can be two modes offered but it is definitively two very different way of seeing the product

[ph-ausseil]

We could so something like:

goal: Make more paperclips
Leader: 
 - model: GPT4
   name: THE PAPERCLIPPER
   objectives:
     - goal: Gather Materials
        leader: 
        - model: GPT3.5
          name: the miner
          objectives:
          - mine
      - goal: Build Factories
        leader:
        - model: GPT4
           name: Factory Construction Supervisor
           objectives: 
           - goal: Design Factory
             leader: GPT4
             objectives:
               - goal: Generate an image of a factory
                  leader: HF
           - goal: Get factory materials
             leader: GPT3.5
             . . .
           - goal: combine materials
            . . .
           - goal: finish factory
          . . . 

@ntindle I have restructured my code to do what you want, make it as modular as possible & allow sub project & team as you suggested here and on discord.

I will publish tonigh.

[joeflack4]

Bah, I'm a fool. I started working on this and have to abandon what I did. I should have looked to see what was cooking first. I had a much more incremental approach in mind. @ph-ausseil Are you getting good buy-in &/or headway with this? I'm inclined to agree with @ntindle on this one in the near term. I actually do like the ideas, but it may be too much for a first PR. If there's still flexibility to change the implementation, perhaps breaking down into multiple issues and wrapping a grouping issue around it all would be good.

[ntindle]

If you're in the discord, shoot a message and I can help you find a place to contribute that will help this effort

[Boostrix]

Seems, several folks are working on this independently at the moment ?

https://github.com/Significant-Gravitas/Auto-GPT/pull/1937#issuecomment-1529404002

[ph-ausseil]

Hi @joeflack4 & @Boostrix ,

So, they are differents attemps on several of these activities, including a past implementation from me & #1937 , however they involve many core changes that diminishes/anihilate their chances to be integrated.

The idea is to get a solution that don't changes core, manage migration to a new back-end 1 project/folder or 1 project per agent. There is a lot of suggestions at the moment :

• Status
• State awareness
• 1 folder per agent

In an incremental approach, where the :

• First thing would be to get a project abstraction layer which requires, specifications (and this discussion is here to help)
• Second thing is to get energies on it as energies are in memories, , gui, ect !
• Third thing is to define what's an MVP with safe and essential features and what should added afterward.

[Boostrix]

I agree that collaboration would be best, obviously it's not easy to look up/read on everything that's been said and done in this context. Maybe, it would help to state the goals/non-goals of different efforts/people and then take it from there, like say, incrementally ?

Personally, my interests/goals would be best summed up as:

• provide support for multiple projects/tasks, with a workspace abstraction and sub-folders
• I would like to use this for nested hierarchies of agents
• but also to do regression testing (benchmarks), the idea being to keep a git log of agents (ai_settings.yaml) and their workspaces
• this would then also make it possible to run these agents as part of a release process (think CI)
• for starters, there would be simple agents/tasks such as "create folder/file", "edit file", "copy file"
• with more complex tasks built on top of these (think create XML file, create HTML file, create JavaScript file" etc
• the majority of these could be created using agents, i.e. in a semi-automated fashion
• use the sub-agent facility to "abstract away" tasks and sub-tasks
• I would be interested in tracking/monitoring progress of sub-agents/tasks to be able to suspend/resume or terminate/restart agents
• a killer feature would be figuring out dependencies between agents/tasks, and figure out a way to use concurrent agents/tasks, possibly by using makefiles/ninja

In this sort of scheme, security could be implemented by a higher set of agents that observe/watch critical points, such as execution of shell commands or compliance with constraints (think token limit etc)

[joeflack4]

@ph-ausseil I didn't get very far with mine, but what I made headway on was the multiple projects.

What I was working on

(i) A single folder for all projects. A CLI arg for its path. I re-used the existing workspace folder for that. (ii) each project gets a folder Folder includes ai_settings.yaml and workspace for I/O. Defaulting to being nested under 'i', but could be overridden by CLI args. I've only used one agent for per project, but I agree with you, @Boostrix, and others that agent hierarchy per project is needed. Not sure if that should be in the first incremental PR or not. Maybe @ntindle or another lead dev has an opinion on this? (iii) Accessing from project list When you start an AutoGPT session, the startup text is exactly the same, showing most recent agent, etc, except there would be an option to see a list of other agents.

Thoughts

I feel like this is the easiest thing to do incrementally and get merged, so I would make an MVP for that first. Not that the other things (e.g. status) are not awesome.

You already have a bunch of good checkboxes in the OP, but adding some more for these various other things or breaking into other issues would be nice.

@ph-ausseil What do you mean by energies?

Second thing is to get energies on it as energies are in memories, , gui, ect !

Maybe speccing is the best thing for this project because so many people are involved, but I hate speccing and the slowness it can bring. I feel like if we stuck with a small MVP for multiple projects, the PR for that would be sufficient. State awareness might be something to spec more, I think.

@Boostrix I read through your ideas and they are very cool too. For me again, I'm more concerned with the "multiple project" MVP than sophisticated architecture. I do have a lot of thoughts here but don't want to spam you guys with that here/now. Maybe discord some time (Shamaya#6594). Only one of your ideas I disagree with is VCS for agents within AutoGPT itself. My thought is that the best way to do that would be to create ones own separate repoand pip install autogpt, then set up a run config that points to your projects/agents folder, and commit those there.

[Boostrix]

each project gets a folder, which includes ai_settings.yaml and workspace for I/O. I've only used one agent for per project but I see that there can be more.

locally, I've seen multiple sub-agents to work better - even if these are operating sequentially. Far fewer repetition of redundant tasking - it seems, using just a single agent is a sure way to confuse the agent with its "plan" - thus, once some contigency kicks in, it will restart its loop and repeat steps it previously finished successfully.

I've only used one agent for per project, but I agree with you, @Boostrix, and others that agent hierarchy per project is needed.

That sounds far more sophisticated than my takeaway actually: Using a sub-agent for steps would mean that these don't get to see much of the "bigger picture" involved, which makes their plans/actions much more narrowed down apparently.

So basically, I would consider the sub-agent approach analogous to how function calls get to see only relevant state on their stack frame, despite the stack frame itself being potentially huge (think recursion). I believe, being able to fire off a sub-agent is also a good way to "isolate" their tasks/plans.

At some point, we may want to differentiate between sub-agents that actually stick around or that work in an async fashion - but what I am envisioning right now is far less sophisticated: a means to split up an agents tasks to use sub-agents for each, probably in a purely sequential and deterministic fashion (aka let the parent agent just wait to get things working first).

It is at that point that we can think about using this just to manage, and encapusulate, workloads - via something like anonymous agents (fire & forget) or actual use named agents that may have more sophisticated behavior.

[github-actions[bot]]

This issue has automatically been marked as stale because it has not had any activity in the last 50 days. You can unstale it by commenting or removing the label. Otherwise, this issue will be closed in 10 days.

[github-actions[bot]]

This issue was closed automatically because it has been stale for 10 days with no activity.