Add concept of multiple LLM providers and add anthropic as example

[anthony-chaudhary]

Add general ability to have more than one LLM provider. Tried to keep to current code pattern.

Background

There is support for multiple memory providers (pinecone, etc.), but there is currently only support for OpenAI. There are lots of open alternatives and discussion around that, but those all take more effort to add. As a V1 of next step in supporting more LLM providers, adding a similar company. This should align with overall direction of not being tied to one provider (OpenAI)

Changes

-> Adds config for llm_provider -> Adds Anthropic

Documentation

-> Follows same Env template documentation, and similar code patterns, does not create any new functions only adding within existing pattern. -> Left openai as default, so no changes by default -> JSON formatting limitations, the JSON clean up / response expectations may be different from GPT / may fail in different ways

Test Plan

Manually tested. There are some issues with the JSON formatting sometimes, but this appears to be a more general issue and this change should still be aligned with next steps to expand that.

PR Quality Checklist

• [X] My pull request is atomic and focuses on a single change.
• [X] I have thoroughly tested my changes with multiple different prompts.
• [X] I have considered potential risks and mitigations for my changes.
• [X] I have documented my changes clearly and comprehensively.
• [X] I have not snuck in any "extra" small tweaks changes

[anthony-chaudhary]

This doesn't address the embeddings side yet e.g.: https://github.com/Significant-Gravitas/Auto-GPT/blob/5752a466a243548d84f67a1679dd6b0cbc2a7165/autogpt/memory/base.py#L14

[github-actions[bot]]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[anthony-chaudhary]

@BillSchumacher Are LLM providers going to be considered plugins too? e.g. OpenAI would also become a plugin?

[p-i-]

This is a mass message from the AutoGPT core team. Our apologies for the ongoing delay in processing PRs. This is because we are re-architecting the AutoGPT core!

For more details (and for infor on joining our Discord), please refer to: https://github.com/Significant-Gravitas/Auto-GPT/wiki/Architecting

[sntxerror]

Isn't it better to allow to have multiple LLM providers simultaneously, while configuring provider and model for specific types of task?

For example, Anthropic just released 100K context model, which is great for summarization or Vertex AI, or today we have very cheap PaLM2 textembedding-gecko for text embeddings and so on.

I see this like a separate config with keys of currently available tasks and values of available providers/models.

[ntindle]

@anthony-sarkis i have no idea how this wasn’t picked up sooner but we are doing a rearch to support this feature. If you’re in our discord a quick chat would be awesome. I have the same name there

[Pwuts]

With the recent release of OpenAI's function calling feature, which we are planning to make use of, LLMs just became less hot-swappable. @ntindle should this PR remain open?

[anthony-chaudhary]

@anthony-sarkis i have no idea how this wasn’t picked up sooner but we are doing a rearch to support this feature. If you’re in our discord a quick chat would be awesome. I have the same name there

@ntindle Thanks for the note, sorry I haven't been able to get to this. @mikelambert Would you or someone on your team be able to pick this up maybe?

[ntindle]

Yes. This should remain as we shouldn't be dependent on one LLMs feature set

[sntxerror]

@anthony-sarkis just add an abstract composition layer between LLM entities and app core https://refactoring.guru/design-patterns/composite

[Pwuts]

Yes. This should remain as we shouldn't be dependent on one LLMs feature set

Sure, agreed. I look at it from a practical POV though:

• Does it make sense to merge this PR?
  • Particularly in context of ongoing re-arch
  • Does it make more sense to merge this PR than to merge another PR that adds support for other LLMs?

From this angle, I'm not so sure. But that's mainly because re-arch is a bit of a blur at the moment.

[sntxerror]

I think that none of this should be merged.

Since re-arch is long-term development project, but more and more LLMs getting released today, I see that it would be much better for the project if someone could implement common solution for integrating multiple LLMs.

Those two PRs could merged after common solution implementation, when there will be a standard way to add new LLMs

On Thu, Jun 15, 2023, 6:38 PM Reinier van der Leer @.***> wrote:

Yes. This should remain as we shouldn't be dependent on one LLMs feature set

Sure, agreed. I look at it from a practical POV though:

• Does it make sense to merge this PR?
  • Particularly in context of ongoing re-arch
  • Does it make more sense to merge this PR than to merge another PR that adds support for other LLMs?

From this angle, I'm not so sure. But that's mainly because re-arch is a bit of a blur at the moment.

— Reply to this email directly, view it on GitHub https://github.com/Significant-Gravitas/Auto-GPT/pull/2434#issuecomment-1593308723, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADL3SOMWRHEAU4HJ6LZGLHLXLMT7LANCNFSM6AAAAAAXDEJCEA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[Pwuts]

@sntxerror I couldn't have said it better.

I have labeled it with fridge, so we'll look at it again once more of the long-term infrastructure is in place. For now, I'm turning this back into a draft PR, as we can't merge it in its current form.