[AutoBuild] Introducing CaptainAgent

[LeoLjl]

trafficstars

Introducing CaptainAgent, an agent that can break down tasks and dynamically call AutoBuild as a tool to solve the subtasks. In one tool call, the CaptainAgent will provide the building task, execution task and group name. The building task is used to assemble group of experts, the execution task is what the experts need to solve.

We propose a RAG-like pipeline for expert group generation. Given a building task, we perform similarity search in agent library and find topk candidates, then let LLM to select the best fit. If no expert is suitable for the task, new experts will be generated. Tools will be retrieved and binded to the experts.

This is the updated version of PR #2414 #1749.

This PR is still under active construction. Below is the roadmap for merging it.

• [x] Add full content of tool library.
• [ ] Add test for captain agent.
• [ ] Add agent library. The full content of agent library is too large and requires deduplication.
• [x] Revise the notebook that introduces MetaAgent. The current one only involves tool library.
• [ ] Finish blog post.

Additional feature(s) that are on the todo list.

• [ ] Make it more customizable. The current design is a static library. Introduce API for users to add their own tools and agents into metaagent library.

Why are these changes needed?

Related issue number

PR #2414 #1749

Checks

• [ ] I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
• [ ] I've added tests (if relevant) corresponding to the changes introduced in this PR.
• [ ] I've made sure all auto checks have passed.

[gitguardian[bot]]

⚠️ GitGuardian has uncovered 96 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

Since your pull request originates from a forked repository, GitGuardian is not able to associate the secrets uncovered with secret incidents on your GitGuardian dashboard. Skipping this check run and merging your pull request will create secret incidents on your GitGuardian dashboard.

🔎 Detected hardcoded secrets in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
12853598	Triggered	Generic High Entropy Secret	79dbb7bc2561713bc11225849e408dc74db1228f	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	e43a86c78f3f947b6e142b3aaf36e7a9852f7078	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	bdb40d77d7be9ea42e5fa28f6e851edf549ef0af	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	954ca451f949a4924578bb7e3e93c97ad4ba1dd5	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	79dbb7bc2561713bc11225849e408dc74db1228f	test/oai/test_utils.py	View secret
10404662	Triggered	Generic CLI Secret	eff19acf1365e34fe17d9ac0939666f32b3ceda5	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	06a0a5ddb39fb3000e40cb4872741dead9fcb7e0	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	0524c774dda696855e9a8f87fe0b9ed96cce13f6	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	d7ea410501cb96bc97a203a0a95431541515f9cc	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	e43a86c78f3f947b6e142b3aaf36e7a9852f7078	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	841ed315e2f19d79a6b86ed587eb6e0fc4a0c0da	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	802f099588bedf1d022b2bba5fb534635df8e6f1	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	9a484d8589761d58940e3c0b79215204ce6a23c1	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	e973ac38ea4f7b36687ea03aa44f770d7e2ddcac	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	89650e74f572bbc60e9c24e04b4c601616e439c7	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	e07b06bc939353dee2afc6dd52c159d6cb3428d5	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	abe4c419c45a244e2399508f9ffef606ce6a4685	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	7362fb9cf4f0a6b3e7ae4e97229afad655d5d87b	.github/workflows/dotnet-release.yml	View secret
12853599	Triggered	Generic High Entropy Secret	79dbb7bc2561713bc11225849e408dc74db1228f	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	e43a86c78f3f947b6e142b3aaf36e7a9852f7078	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	954ca451f949a4924578bb7e3e93c97ad4ba1dd5	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	bdb40d77d7be9ea42e5fa28f6e851edf549ef0af	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	abad9ff4487444324d1916a6b94c8049b4cab9e7	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	954ca451f949a4924578bb7e3e93c97ad4ba1dd5	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	c7bb588a684a038dbee7f9dc9afd6d44ce35ac3a	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	b97b99d4b2cfe735a7aa46258508d4a7cda9cad5	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	e43a86c78f3f947b6e142b3aaf36e7a9852f7078	test/oai/test_utils.py	View secret
12853600	Triggered	Generic High Entropy Secret	79dbb7bc2561713bc11225849e408dc74db1228f	test/oai/test_utils.py	View secret
12853601	Triggered	Generic High Entropy Secret	79dbb7bc2561713bc11225849e408dc74db1228f	test/oai/test_utils.py	View secret
10493810	Triggered	Generic Password	49e8053dd1e5456d3758b4a85f5721e9c9b12e16	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	501610b4fc0c649fa2b1cbf0fd72fa6f14f026d0	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	49e8053dd1e5456d3758b4a85f5721e9c9b12e16	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	501610b4fc0c649fa2b1cbf0fd72fa6f14f026d0	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	d422c63596fce7b84e0ef8a7bd6518d5f4336eaf	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	97fa339ac749b3c7d51f1b0fae156d41c02b214e	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	49e8053dd1e5456d3758b4a85f5721e9c9b12e16	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	d422c63596fce7b84e0ef8a7bd6518d5f4336eaf	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	97fa339ac749b3c7d51f1b0fae156d41c02b214e	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	d422c63596fce7b84e0ef8a7bd6518d5f4336eaf	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	97fa339ac749b3c7d51f1b0fae156d41c02b214e	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	501610b4fc0c649fa2b1cbf0fd72fa6f14f026d0	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10404696	Triggered	Generic High Entropy Secret	954ca451f949a4924578bb7e3e93c97ad4ba1dd5	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	bdb40d77d7be9ea42e5fa28f6e851edf549ef0af	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	79dbb7bc2561713bc11225849e408dc74db1228f	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	e43a86c78f3f947b6e142b3aaf36e7a9852f7078	test/oai/test_utils.py	View secret
10422482	Triggered	Generic High Entropy Secret	79dbb7bc2561713bc11225849e408dc74db1228f	test/oai/test_utils.py	View secret
10422482	Triggered	Generic High Entropy Secret	bdb40d77d7be9ea42e5fa28f6e851edf549ef0af	test/oai/test_utils.py	View secret
12853602	Triggered	Generic High Entropy Secret	79dbb7bc2561713bc11225849e408dc74db1228f	test/oai/test_utils.py	View secret
11616921	Triggered	Generic High Entropy Secret	a86d0fde2e667f9177eea55da17312b770c9d76b	notebook/agentchat_agentops.ipynb	View secret
11616921	Triggered	Generic High Entropy Secret	394561b4629222c19d8bb3bc58222fc8813a5833	notebook/agentchat_agentops.ipynb	View secret
11616921	Triggered	Generic High Entropy Secret	3eac646b8974e1d1be3fde557b055563f56f2f5f	notebook/agentchat_agentops.ipynb	View secret
11616921	Triggered	Generic High Entropy Secret	f45b55337aa7f9d6151f21f2c55ba5dc83f95b79	notebook/agentchat_agentops.ipynb	View secret
11616921	Triggered	Generic High Entropy Secret	65632487224aae2c7faca67de40fd2bda3ad3905	notebook/agentchat_agentops.ipynb	View secret
12853598	Triggered	Generic High Entropy Secret	2b3a9ae05569d38cfc89d9daa4b1612b8406d178	test/oai/test_utils.py	View secret
12853598	Triggered	Generic High Entropy Secret	c03558f5b2708c31df36666486f54b2714435162	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	c03558f5b2708c31df36666486f54b2714435162	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	2b3a9ae05569d38cfc89d9daa4b1612b8406d178	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	0a3c6c49834b4f68c688c1d2e76a5ebb8f7d91e2	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	76f5f5a66532a3b95d03fe0ae8a56a59e43012e1	test/oai/test_utils.py	View secret
10404662	Triggered	Generic CLI Secret	954ca451f949a4924578bb7e3e93c97ad4ba1dd5	.github/workflows/dotnet-build.yml	View secret
12853599	Triggered	Generic High Entropy Secret	2b3a9ae05569d38cfc89d9daa4b1612b8406d178	test/oai/test_utils.py	View secret
12853599	Triggered	Generic High Entropy Secret	c03558f5b2708c31df36666486f54b2714435162	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	76f5f5a66532a3b95d03fe0ae8a56a59e43012e1	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	0a3c6c49834b4f68c688c1d2e76a5ebb8f7d91e2	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	3b79cc6fdfa485026262ec44aa5f1f0238bf7c3d	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	11baa52155c3846299d39ed0b5a814b33b0eb671	test/oai/test_utils.py	View secret
12853600	Triggered	Generic High Entropy Secret	c03558f5b2708c31df36666486f54b2714435162	test/oai/test_utils.py	View secret
12853600	Triggered	Generic High Entropy Secret	2b3a9ae05569d38cfc89d9daa4b1612b8406d178	test/oai/test_utils.py	View secret
12853601	Triggered	Generic High Entropy Secret	c03558f5b2708c31df36666486f54b2714435162	test/oai/test_utils.py	View secret
12853601	Triggered	Generic High Entropy Secret	2b3a9ae05569d38cfc89d9daa4b1612b8406d178	test/oai/test_utils.py	View secret
10493810	Triggered	Generic Password	3b79cc6fdfa485026262ec44aa5f1f0238bf7c3d	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	11baa52155c3846299d39ed0b5a814b33b0eb671	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	11baa52155c3846299d39ed0b5a814b33b0eb671	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	3b79cc6fdfa485026262ec44aa5f1f0238bf7c3d	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10404696	Triggered	Generic High Entropy Secret	0a3c6c49834b4f68c688c1d2e76a5ebb8f7d91e2	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	76f5f5a66532a3b95d03fe0ae8a56a59e43012e1	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	c03558f5b2708c31df36666486f54b2714435162	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	2b3a9ae05569d38cfc89d9daa4b1612b8406d178	test/oai/test_utils.py	View secret
10422482	Triggered	Generic High Entropy Secret	2b3a9ae05569d38cfc89d9daa4b1612b8406d178	test/oai/test_utils.py	View secret
10422482	Triggered	Generic High Entropy Secret	c03558f5b2708c31df36666486f54b2714435162	test/oai/test_utils.py	View secret

and 16 others.

🛠 Guidelines to remediate hardcoded secrets

1. Understand the implications of revoking this secret by investigating where it is used in your code.
2. Replace and store your secrets safely. Learn here the best practices.
3. Revoke and rotate these secrets.
4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

• following these best practices for managing and storing secrets including API keys and other credentials
• install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

---

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 0% with 190 lines in your changes missing coverage. Please review.

Project coverage is 14.68%. Comparing base (6279247) to head (df6a7b2). Report is 11 commits behind head on main.

Files	Patch %	Lines
autogen/agentchat/contrib/meta_user_proxy_agent.py	0.00%	104 Missing :warning:
autogen/tool_utils.py	0.00%	35 Missing :warning:
autogen/agentchat/contrib/tool_retriever.py	0.00%	31 Missing :warning:
autogen/agentchat/contrib/meta_agent.py	0.00%	20 Missing :warning:

:exclamation: There is a different number of reports uploaded between BASE (6279247) and HEAD (df6a7b2). Click for more details.

HEAD has 1 upload less than BASE

Flag	BASE (6279247)	HEAD (df6a7b2)
unittests	2	1

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #3042       +/-   ##
===========================================
- Coverage   32.90%   14.68%   -18.23%     
===========================================
  Files          94       99        +5     
  Lines       10235    10617      +382     
  Branches     2193     2420      +227     
===========================================
- Hits         3368     1559     -1809     
- Misses       6580     9008     +2428     
+ Partials      287       50      -237     

Flag	Coverage Δ
unittests	14.68% <0.00%> (-18.23%)	:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.