support zilliz cloud as memory backend

[chyezh]

Background

I found that someone encounter some problem when using milvus as memory backend with zilliz cloud. This PR will make easier to interact with zilliz cloud.

#1812

Changes

• add zilliz cloud to milvus memory backend implement.
• add document to readme.

Documentation

• new feature and code is full documented.

Test Plan

• no error occur when using the open-source milvus as memory backend.
• no error occur when using the zilliz cloud as memory backend.

PR Quality Checklist

• [x] My pull request is atomic and focuses on a single change.
• [x] I have thoroughly tested my changes with multiple different prompts.
• [x] I have considered potential risks and mitigations for my changes.
• [x] I have documented my changes clearly and comprehensively.
• [x] I have not snuck in any "extra" small tweaks changes

[chyezh]

I create a bad force-update after pull-request-merge from master. Now, it's fixed and ready for review.

@ngo275 @richbeales

[Kenneth-zh]

Do I need to initialize collection myself?

[ngo275]

@Kenneth-zh No, a new collection called autogpt will be generated by default.

[Kenneth-zh]

OK,it work perfectly

[github-actions[bot]]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[github-actions[bot]]

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.

[chyezh]

fixed: black format check failed.

[xiaofan-luan]

Blocked by this issue, @Torantulino. Is there any way we can merge this as soon as possible? Thanks!

[ghost]

We found that many users use zilliz cloud to support Auto-GPT, but they have difficulties in using it. This PR can solve this problem. This is very important to us, hope this PR can be merged as soon as possible. @BillSchumacher @richbeales thank you very much.

[chyezh]

it improves the use experience of Auto-GPT with zilliz cloud, can we merge this and close the related issue in shortly? Thanks. @richbeales

[github-actions[bot]]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[github-actions[bot]]

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.

[Torantulino]

We're taking a look at this, thanks!

[chyezh]

@Pwuts I have merged zilliz cloud support with milvus support as much as possible. For showing correct memory backend type in command line, derived type ZillizMemory is still preserved.

[chyezh]

@Pwuts thank you for your patient review. I agree with the most of the review suggestions and have implemented them in the new change. Some explanations about the review are as follows:

• About index_parameters: because that Zilliz Cloud support AutoIndex to create index but Milvus not and Zilliz Cloud only support AutoIndex, the code handling index_parameter is still preserved. The unified behavior in pymilvus may be supported in the future, but now we need to add such a piece of code to keep compatibility for older version pymilvus in AutoGPT.
• Type ZillizMemory is also merged with MivusMemory, and the display of real memory backend support will be commit with separate PR in the future.

[chyezh]

@Pwuts @ashutoshpw

• Rebase to latest master and pass the tests.
• milvus_username and milvus_password is set to empty string by default to keep consistent with pymilvus‘s default.

[codecov[bot]]

Codecov Report

Patch coverage: 15.15% and project coverage change: +3.22 :tada:

Comparison is base (ad5d8b2) 36.22% compared to head (4463712) 39.44%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2127      +/-   ##
==========================================
+ Coverage   36.22%   39.44%   +3.22%     
==========================================
  Files          60       60              
  Lines        2849     2918      +69     
  Branches      471      480       +9     
==========================================
+ Hits         1032     1151     +119     
+ Misses       1755     1692      -63     
- Partials       62       75      +13     

Impacted Files	Coverage Δ
autogpt/memory/__init__.py	43.39% <ø> (ø)
autogpt/memory/milvus.py	3.44% <6.66%> (+0.41%)	:arrow_up:
autogpt/config/config.py	74.49% <100.00%> (-0.16%)	:arrow_down:

... and 10 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

[chyezh]

@Pwuts All review has been reply. Is there anything else need to be modify in these commit? If not, can we merge the commit and close the related issue? Looking forward to your reply.

[github-actions[bot]]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[github-actions[bot]]

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.

[chyezh]

Resolve conflict, rebase to latest master, pass manually test. But codecov/patch task not pass, how to fix it?

[ntindle]

You'll need to add some more tests that cover your changes. We have CodeCov configured not to let test coverage go down too far and your falls outside the threshold. It shouldn't be too bad just to cover the lines it says with tests if you click details on the failed run

[chyezh]

@Pwuts Thank you for your feedback. Based on your review, I made the following fixes.

• Add MILVUS_SECURE for open-source milvus users to enable secure option.
• connect to Milvus is executed according to the following rules now.
  • If your MILVUS_ADDR start with https:// or http:// or tcp://, it will use uri argument to connect, else use address argument.
  • If your MILVUS_ADDR start with https, it will enable secure option automatically.
  • If your MILVUS_ADDR meets the regex of zilliz cloud (provided by official), it will use AutoIndex automatically.

New commits covering more usage scenarios of milvus.

codecov/patch is not pass because of insufficient test coverage, could we merge the PR first and provide testing in future new PR. Many users use zilliz cloud to support Auto-GPT, but have trouble in running it. This is very important to us, hope this PR can be merged as soon as possible.

[ntindle]

As the person doing test and coverage checks, I accept Codecov/patch failing for this PR. The team members who have been helping you are asleep but I’ve ping them to have a look when they wake up. That will probably be more than 7 hours from now

[Pwuts]

Doing final review now

[Pwuts]

PyMilvus indeed has empty strings as default user, password etc. However, this is of no concern to the user or main application config, so I amended parse_configure (now configure) to handle this (and to raise an error if one of them is set but not both). Also edited __init__ to use one connect() statement as they were almost complete duplicates.