[WIP] Project manager: Create & manage multiple projects with specific AI workers

[arrenv]

NOTE: This PR is still a WIP, submitting it now to make sure I'm not clashing with existing plans.

Background:

The vision is to be able to fully manage projects with a team of specialized AI agents/workers that can work together autonomously.

To start with, currently, the AI settings are saved in a fixed location with a static file name, they are also individual. This makes it difficult to manage projects as a whole, many separate projects and agents. While I have read that there are plans to move api_sessions.yaml into it's own folder, I still think this could be extended further with a project manager. The project manager is managed within it's own directory, where all agent specific configs are stored in the 'agents' folder, project related files are also stored within each individual project folder.

[Demo creating a new project]

Changes:

• New 'ProjectManager' class added
• Updated the AIConfig class to creating and saving project settings
• Added commands for managing projects
• Refactored setup to handle projects

Documentation:

• AI settings will be saved in a folder structure: ./projects/{project_name}/agents
• The AI settings file will be named ai_{ai_name}.yaml
• Project specific folders and files will be added to the `./projects/{project_name}' folder

Test Plan:

• Basic unit test has been added
• Development is still underway, further testing will be added

PR Quality Checklist

• [ ] My pull request is atomic and focuses on a single change.
• [ ] I have thoroughly tested my changes with multiple different prompts.
• [ ] I have considered potential risks and mitigations for my changes.
• [ ] I have documented my changes clearly and comprehensively.
• [ ] I have not snuck in any "extra" small tweaks changes

To Do

• [ ] List and switch projects
• [x] List project agents
• [x] Account for stand-alone agents

[github-actions[bot]]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[github-actions[bot]]

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.

[github-actions[bot]]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[github-actions[bot]]

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.

[github-actions[bot]]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[github-actions[bot]]

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.

[github-actions[bot]]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[github-actions[bot]]

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.

[github-actions[bot]]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[github-actions[bot]]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[github-actions[bot]]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[ntindle]

Watching this with so much interest

[github-actions[bot]]

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.

[github-actions[bot]]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[ntindle]

@collijk made some changes that will directly affect this, but hopefully make your life easier!

[ph-ausseil]

TO ALL :

Main differences between PR #1937 and issue #3039 :

• Core model : # 1937 build interaction with each Agent of the Project vs # 3039 aim to interact with only a Lead Agent that coordinate other agents.
• File structure : # 1937 have 1 folder per project & 1 yaml per Agent vs # 3039 has 1 yaml for ALL.
• Process of thought & methodology : # 1937 not shared vs # 3039 reported all known issues (memory loss, backward compatibility), offert solution & consulted 3rd person opinion I also identified upcoming chalenges .

PS : As it is not mentioned memory loss & backward compatibility are common issues to PR #1937 and issue #3039

TO @arrenv :

Hi @arrenv , I have implemented similar functions a different way would be happy to be in touch with you to better understand why you made some choices with regards to implementation as we have dealt with the issue two very different ways.

I believe that Core Model & File Structure are personal opinions, there is no good are bad. Some will prefer my implementation, somme will prefer yours.

• Core model : Should we keep both model ? Should we push one one forward ? I think your Core Model may pose problem when the Agent is not interacting with a LLM. So I will definitively push my idea until proven otherwise.
• File structure : I like it your way IF you add a "version" in each .yaml file to manage backward compatibility. Would the team decide to support backward modification of config file and memory. Your inconvenient : Many file to manage when changing the structure (more work to maintain). Your pros : Extra clarity with project folder. My inconvenient : For extra security, concurrent access have to be managed in AIConfig. My pros : Extra safe as long concurrent access is properly managed, easier to maintain & manage backward compatibility. Overall, I really like your folder set-up, I also think that my pros in term of maintainability is a better argument but I believe it is a matter of taste.
• Process of thought & methodology : I would really like you share yours and we have exchanges to find the best solution.

[github-actions[bot]]

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

[arrenv]

@ph-ausseil Happy to discuss more, please feel free to reach out to me on Discord. I recently posted in the Auto-GPT Discord, within the 'Auto-GPT orgs' thread. In the meantime, I will look over your comments and repo and provide my thoughts.

[ph-ausseil]

@ph-ausseil Happy to discuss more, please feel free to reach out to me on Discord. I recently posted in the Auto-GPT Discord, within the 'Auto-GPT orgs' thread. In the meantime, I will look over your comments and repo and provide my thoughts.

Please read my issue ahead of the code as I have made the choice not to implement my thought and wait for the team feedback especially AICofing.save() . If I have green light intend to split AICofing and create a AIConfigBroker() anyway :-).

[arrenv]

• Core model : Should we keep both model ? Should we push one one forward ? I think your Core Model may pose problem when the Agent is not interacting with a LLM. So I will definitively push my idea until proven otherwise.

The separation of projects and agents in my approach was decided to simplify the configs so that they could be transferred to other projects more easily, and to allow for the growth of the agent models.

It could be possible to combine the different models, i.e. have a project folder structure, then a single yaml to handle the projects and agents.

• File structure : I like it your way IF you add a "version" in each .yaml file to manage backward compatibility. Would the team decide to support backward modification of config file and memory. Your inconvenient : Many file to manage when changing the structure (more work to maintain). Your pros : Extra clarity with project folder. My inconvenient : For extra security, concurrent access have to be managed in AIConfig. My pros : Extra safe as long concurrent access is properly managed, easier to maintain & manage backward compatibility. Overall, I really like your folder set-up, I also think that my pros in term of maintainability is a better argument but I believe it is a matter of taste.

I agree with each of your points, and would again add that a combined approach could be a good compromise.

• Process of thought & methodology : I would really like you share yours and we have exchanges to find the best solution.

For me, the idea was to create independent agents that were specialised in their purpose and goals which seems to work the best for me when prompting LLMs.

Then, I wanted organisation of these agents and the ability for agents to work together, and also agents with the purpose of directing specialised agents. With the added ability of shared memory and context.

I also wanted the organisation of the outputs from each of the agents, so that outputs become accessible to each agent with a project. For example, a developer agent in one project working on all files within that project directory and a reviewer agent able to review all the dev output contained within that project.

I also wanted to experiment with different agents within a project without removing the config of an agent, for example, dev agent 1 verse dev agent 2. I can see their outputs, compare them, then iterate off of the winner. I wanted to do this without losing the history or cluttering a single config file.

[ph-ausseil]

• Core model : Should we keep both model ? Should we push one one forward ? I think your Core Model may pose problem when the Agent is not interacting with a LLM. So I will definitively push my idea until proven otherwise.

The separation of projects and agents in my approach was decided to simplify the configs so that they could be transferred to other projects more easily, and to allow for the growth of the agent models.

It could be possible to combine the different models, i.e. have a project folder structure, then a single yaml to handle the projects and agents.

Hey , There is a bit more to that the difference is that :

• Via #1937 you can build interaction with each agent of the Project , also agent must be LLM but it is scalable
• Via #3039 you interact with a leader that does all the job of managing the other agents

It is why we provide response to similar needs but with two very different solution which are not mergeable, it can be two modes offered but it is definitively two very different way of seeing the product

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
docs	⬜️ Ignored (Inspect)			Apr 26, 2023 9:04pm

[github-actions[bot]]

This PR exceeds the recommended size of 200 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size

[arrenv]

Closing as there is more discussion and planning required.

[Boostrix]

Maybe, could you please briefly sum up where you got stuck and what's missing in your opinion to pull this off ? I am kinda running into this degree of complexity in an effort to use Auto-GPT to write its own commands/plugins and got a first example to fetch open Auto-GPT pulls/issues from github to re-prioritize those using labels/comments: https://github.com/Significant-Gravitas/Auto-GPT/issues/15#issuecomment-1529383061

The point being, multiple steps in a plan are currently too fragile, the agent keeps tackling steps that it had already solved. So, I was hoping to use the sub-agent mechanism in order to come up with more specific plans for each sub-angent, so that parent agent would be able to observe progress and kill/restart agents (or ask for human feedback) in case they're getting stuck https://github.com/Significant-Gravitas/Auto-GPT/issues/3444#issuecomment-1529005391

The idea to use separate (sub)agents would be in line with the approach outlined here: https://github.com/Significant-Gravitas/Auto-GPT/issues/3593

However, the current way of having a top-level ai_settings.yaml file and a single workspace simply doesn't scale for this sort of setup. Thus, I was thinking of using empty directories where the ai_settings.yaml file would be used to create new sub-workspaces as needed.

├── ai_settings.yaml ├── Agent-1.running │ ├── ai_settings.yaml │ └── workspace ├── Agent-2.finished │ ├── ai_settings.yaml │ └── workspace ├── Agent-3.running │ ├── ai_settings.yaml │ └── workspace ├── Agent-4.running │ ├── ai_settings.yaml │ └── workspace ├── Agent-5.finished │ ├── ai_settings.yaml │ └── workspace └── workspace

By convention each sub-agent would have permission to create its own agents (think of it like a stack of agents), but would only be able to manage sub-agents, and would also only be able to access workspaces of sub-agents - that way, each agent creating a sub-agent would be acting as the "manager" of each of its agents, but would not be able to interfere with agents higher up in the hierarchy

the state (running, finished) would be one way to communicate state without using the agent messaging API, but would also make it easier to observe progress - it might also be possible to use a percentage instead to indicate how many tasks it has accomplished so far

This sort of directory layout/setup would also make it possible to use the same mechanism to easily share/maintain agents for other purposes like regression testing and benchmarking #1359

[arrenv]

@Boostrix You have summed up very well why I started down this path. There were no challenges that I was running into, my progress was halted due to conflicting architecture proposals and with the need to focus on other more fundamental improvements, such as memory and agent encapsulation.

You can see more on the architecture discussion here, a lot of great work from @ph-ausseil put into it - https://github.com/Significant-Gravitas/Auto-GPT/discussions/3392

I closed the issue to minimize PR clutter and as there was minimal chance of a merge, I am continuing locally for the time being.