AutoGPT
AutoGPT copied to clipboard
Significant-Gravitas
Extend abilities via gradio tools
Background
AutoGPT can be more powerful it had more commands at its disposal. Some developers may want to give their agent special abilities in order for it to accomplish its goals.
Changes
Introduces a tools module that stores all the tools the developer wants to give to their agent.
A tool is anything that conforms to this interface:
@dataclass
class Tool:
name: str
description: str
run: Callable[[str], str]
args: Dict[str, str]
To add a tool, developers implement this interface and modify the list of TOOLS list defined in tools.py.
For a given run, the developer can specify which tool to use by specifying AUTOGPT_TOOLS variable in the .env file as a comma separated list.
Additionally, I've added a couple of tools from gradio-tools, a library which implements some tools from the thousands of Gradio spaces on huggingface.
As more tools are added to gradio-tools, more AutoGPT users will benefit.
Documentation
Waiting on feedback on API before writing documentation.
Test Plan
Sample Output
PR Quality Checklist
- [x] My pull request is atomic and focuses on a single change.
- [ ] I have thoroughly tested my changes with multiple different prompts.
- [ ] I have considered potential risks and mitigations for my changes.
- [x] I have documented my changes clearly and comprehensively.
- [x] I have not snuck in any "extra" small tweaks changes
Took me a while to get what was happening here. This is really cool. I think we already have stable diffusion somewhere. So I think this might only be enabling image captioning. Could you describe what the value of using gradio is vs say hitting a huggingface transformers lib directly
I definitely could have done a better job of explaining this @dschonholtz 😂
I think the benefit of this PR is two fold:
-
A framework for extending AutoGPT with new abilities, which I call Tools. A tool can really be anything as long as you implement the minimal interface so the potential is huge. It can even be a transformers model running locally. And developers can add tools without submitting a PR! They just have to edit tools.py and their local AutoGPT will pick it up.
-
The addition of some tools that leverage Huggingface Spaces. HuggingFace spaces hosts thousands of ML apps that are not accessible via API thanks to the gradio client library. My gradio-tools library is the start of a curated list of ML apps that are accessible via API that can be used by LLM agents. Spaces has state of the art apps for any kind of task, e.g. text to video, audio transcription, controlnet, text to music, so the benefit of including them in AutoGPT is that developers can enhance their agent’s ability with minimal set up.
Gotcha. A couple things there, one we have a plugins system coming that will build a system similar to that interface for all of our commands so they can be configured by the user before app start. I think? This is in the pipeline. Therefore we should probably not attempt to build a directly system with that so we have one standardized plugins system.
Second the hugging face module usage through spaces is very cool! Would this be subject to the same querying for a model that you see when using the spaces web UI?
Generally, I think the spaces usage is great and enabling that is fantastic, but we should probably keep that separate from a generic tool interface as that is a separate issue.
@BillSchumacher or @Torantulino if you want to weigh in that’d be great.
@dschonholtz I totally understand what your point about not wanting to add a framework that potentially interferes with the future plugins system. I think part of the value of this PR is that you can add/detract to the set of spaces your agent can call without manually having to add commands to command.py. I wonder if we can merge this in as is and I'd be happy to refactor to conform to the future plugins system when that's ready?
This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.
This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.
This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.
@freddyaboulton I actually do not have merge powers here. @nponeccop What are your thoughts on this? What would you need to see in order for this to go in assuming merge conflicts to go in? I do think there are a couple valuable ideas here.
@dschonholtz I guess the best course would be to wait until plugins are ready and release this as a plugin. That would require some reworking.
Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.
This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.
Heyo, lets look at making this a plugin. Put it in a folder that has a descriptive name in https://github.com/Significant-Gravitas/Auto-GPT-Plugins/tree/master/plugins
Hi @ntindle @dschonholtz @nponeccop , the plugin PR is here: https://github.com/Significant-Gravitas/Auto-GPT-Plugins/pull/14 ! Thank you