AutoGPT

AutoGPT copied to clipboard

Background

This change allows the use of different vector-based memories for Auto-GPT as long as the Memory interface is implemented. As an example, this change integrates Weaviate as an alternative to Pinecone that can be instantiated locally.

Changes

• We created a providers directory which is meant to hold the various memory implementations.
• The original scripts/memory.py file has been moved into the providers module and it declares Memory as an interface to be implemented.
• A new scripts/factory.py file has been added which declares a factory method for creating the appropriate memory support class. A new environment variable, MEMORY_PROVIDER allows to configure which provider to use.
• The Pinecone implementation has been moved into the providers module.
• An Weaviate implementation of the Memory interface has been added.

Test Plan

Added factory tests in tests/memory_tests.py

Change Safety

• [x] I have added tests to cover my changes
• [ ] I have considered potential risks and mitigations for my changes

I have noticed that the latest commands list in commands.py no longer supports the memory-related command (memory_del, memory_ovr). In this pull request I have removed the associated handlers (delete_memory and overwrite_memory) as well as the original commit_memory in part because they no longer follow the memory interface. Let me know whether these memory commands should be back on the menu and I can implement them.

Thank you so much I was actually working on a similar branch last week! 😃

I have now merged the memory strategy from the master branch and added weaviate to the supported providers. I have also added tests for the weaviate integration.

@nponeccop - applied all requests.

@cs0lar what are your thoughts on adding support for embedded weaviate too as part of this PR?

@hsm207 Thanks for spotting that. I have added weaviate embedded support now. Please check it out.

@cs0lar question about the schema:

{
    "class": "Autogpt",

    },
    "properties": [
        {
            "dataType": [
                "text"
            ],
            "description": "original text for the embedding",
            "name": "raw_text",
            "tokenization": "word"
        },
        {
            "dataType": [
                "text"
            ],
            "description": "This property was generated by Weaviate\'s auto-schema feature on Wed Apr 12 16:53:47 2023",
            "name": "class",
            "tokenization": "word"
        }
    ],
   ...
}'

is the property class intended? Asking because it was autogenerated.

@cs0lar There are conflicts again due to the massive merge

@cs0lar LGTM, but there are conflicts now due to merging of other PRs.

@cs0lar I don't have anymore feedback other than my last comment about weaviate_auth. Great job!

@cs0lar There are conflicts now though.

@cs0lar There are conflicts now

Hi, @nponeccop and @cs0lar - is there a way how we can coordinate the fix and merge? Otherwise, we might keep going in circles :)

@bobvanluijt it would be great to know what's holding merging this branch (apart from the latest conflicts). If this functionality is not part of the plan it would be great to know sometime soon so we can move on from it. BTW honour to @nponeccop for relentlessly checking the PRs and conflict fixes!

Run flake8 autogpt/ tests/ --select E303,W293,W291,W292,E305,E231,E302 autogpt/memory/init.py:30:1: E302 expected 2 blank lines, found 1 autogpt/memory/init.py:56:1: W293 blank line contains whitespace autogpt/memory/base.py:10:1: E302 expected 2 blank lines, found 1 autogpt/memory/weaviate.py:9:1: E302 expected 2 blank lines, found 1 autogpt/memory/weaviate.py:21:1: E302 expected 2 blank lines, found 1 autogpt/memory/weaviate.py:76:5: E303 too many blank lines (2) autogpt/memory/weaviate.py:80:80: W291 trailing whitespace tests/integration/weaviate_memory_tests.py:14:1: E302 expected 2 blank lines, found 1 tests/integration/weaviate_memory_tests.py:41:1: W293 blank line contains whitespace tests/integration/weaviate_memory_tests.py:47:1: W293 blank line contains whitespace tests/integration/weaviate_memory_tests.py:56:1: W293 blank line contains whitespace tests/integration/weaviate_memory_tests.py:70:1: W293 blank line contains whitespace tests/integration/weaviate_memory_tests.py:87:5: E303 too many blank lines (2) tests/integration/weaviate_memory_tests.py:102:5: E303 too many blank lines (2)

@cs0lar There are conflicts again

@bobvanluijt it would be great to know what's holding merging this branch (apart from the latest conflicts).

Thanks, @nponeccop, I think it's the merging of other PRs. Seems we are almost there. Your help is appreciated in keeping this PR up-to-date 🙏

Almost there, it seems: https://github.com/Significant-Gravitas/Auto-GPT/pull/424#issuecomment-1509865221

BTW honour to @nponeccop for relentlessly checking the PRs and conflict fixes!

Yup, thanks @nponeccop

Awesome! I think the PR can be merged 🥳

@cs0lar, I think we need your help again. People on the Auto-GPT Discord are asking for this merge 😂

Hi @nponeccop and @richbeales. I think the merge conflict is resolved. Would be awesome if you could review and merge. Thanks 🙏

Thanks for reaching out @bobvanluijt - we'll get to this shortly. I'll ping you when ready.

I think Milvus and Weaviate should also be appended here, right? https://github.com/Significant-Gravitas/Auto-GPT/blob/master/.env.template#L53

Weaviate integration is mentioned in its own section, but Milvus doesn't have that mention and rather than being scathered all around the config, they should all be neatly gathered at the top (imho).

🎉 thanks all for the help and merge 🎉

Added a minor change here: https://github.com/Significant-Gravitas/Auto-GPT/pull/1813

Basically, so that people don't have to install the Weaviate client manually.