LibreChat
LibreChat copied to clipboard
danny-avila
Enhancement: Amazon Bedrock Support
What features would you like to see added?
I'd like to see support for Amazon Bedrock. I have several customers that are looking to leverage models in Bedrock to give their users a ChatGPT like experience. LibreChat would be an awesome option for this.
More details
Support for Amazon Bedrock and KB would be an awesome feature.
Which components are impacted by your request?
No response
Pictures
No response
Code of Conduct
- [X] I agree to follow this project's Code of Conduct
You may could use LiteLLM as a temporary workaround.
https://docs.litellm.ai/docs/providers/bedrock
anthropic keeps banning account for no reason, hopefully bedrock support is coming soon
Hello, we are planning on using LibreChat at a wide scale at Cornell University. We have a requirement to use Bedrock and have a workaround at the moment, however we want to understand the current state of this native integration. We may be able to contribute if it's just a matter of priorities being elsewhere. However, if there are technical roadblocks preventing this enhancement from being implemented at the moment, it would be good to know that so that we can focus on the workaround for now.
Thanks, Marty Sullivan
Hi @marty-sullivan that's exciting!
We may be able to contribute if it's just a matter of priorities being elsewhere.
It is just a matter of priorities being elsewhere. I'm hoping to get to it early July. My idea is that it should be a separate endpoint, similar to Azure OpenAI vs. OpenAI
Also, correct me if I'm wrong, I think Anthropic models have a different API than the other bedrock models (and there may be more differences if Mistral or other big providers are in the same boat).
For this reason, I was thinking of making "Anthropic - Bedrock", "Mistral - Bedrock" their own endpoints, resembling Azure OpenAI vs. OpenAI as mentioned before.
@danny-avila Thanks for the response! I believe that the API call itself in Bedrock is the same for all the models, however, there is obviously a significant difference in the configuration that needs to be passed to the BedrockRuntime.invoke_model / BedrockRuntime.invoke_model_with_response_stream via the body parameter.
Given that, your idea probably makes sense to maybe have an endpoint for each model, rather than trying to abstract that, but not positive.
Our priorities are in other places right now as well, but your timeline may align well with ours. Perhaps we could sync up on where things are on the week of 7/8 and if there is anywhere we can help we can assess then.
@danny-avila Just as an update, I think we are going to move forward with using LiteLLM as suggested in other areas.
I do think this would be a great enhancement though, to natively support Bedrock. We will revisit later, unless you get to it first.
Hi all, this is a great tool, and would be awesome to use it with AWS bedrock.
I have tried to setup with LiteLLM as well following the tutorials though unsuccessfully. I did try a fresh install following the GitHub thread. Anyone who had luck getting litellm work, please tell how!
I might be missing a simple step somewhere but have tried several times.
@danny-avila Just as an update, I think we are going to move forward with using LiteLLM as suggested in other areas.
I do think this would be a great enhancement though, to natively support Bedrock. We will revisit later, unless you get to it first.
Native AWS Setup is coming, and I've already tested extensively using Bedrock models.
Hi all, this is a great tool, and would be awesome to use it with AWS bedrock.
I have tried to setup with LiteLLM as well following the tutorials though unsuccessfully. I did try a fresh install following the GitHub thread. Anyone who had luck getting litellm work, please tell how!
I might be missing a simple step somewhere but have tried several times.
It's not just you; we also had issues using LiteLLM (which works fine on its own) as a custom endpoint in LibreChat. Having native AWS support that integrates directly with Bedrock, similar to this project would be fantastic. Bedrock is expanding rapidly, supporting more inference endpoints for some of the best models in the world. Now that AWS Activate credits can be used, more companies are migrating to it. I applaud @danny-avila for making this a priority in the near future, as he seems to have suggested.
Thanks @NN1985 for confirming the issue with LiteLLM. And more so for mentioning the AWS Activate. As big companies battle it out, the SMEs and entrepreneurs like us have only benefits to reap. Ever since we found LibreChat, we saved money in ChatGPT subscriptions. And now, we are holding up any further recharge to our OpenAI account to replace it with bedrock.
Have you already tried the aws-samples repo for using bedrock? It could serve as a temporary alternative for us if there's some wait for LibreChat bedrock support..
Thanks @NN1985 for confirming the issue with LiteLLM. And more so for mentioning the AWS Activate. As big companies battle it out, the SMEs and entrepreneurs like us have only benefits to reap. Ever since we found LibreChat, we saved money in ChatGPT subscriptions. And now, we are holding up any further recharge to our OpenAI account to replace it with bedrock.
Have you already tried the aws-samples repo for using bedrock? It could serve as a temporary alternative for us if there's some wait for LibreChat bedrock support..
No problem @aryansaurav . I am indeed using the linked repo project to access Bedrock directly for the Anthropic API, which would be prohibitively expensive to use at scale otherwise. I haven't found anything else quite like it at the moment—it's a true one-command AWS console zero-config install, and we were up and running in about 30 minutes. We can add all our users with a login similar to LibreChat, but be warned that it's quite basic and buggy in comparison.
I want to be clear that I strongly support LibreChat and vastly prefer it. Once native Bedrock endpoint support is added, or even better, true AWS native support, we would migrate immediately. As you mentioned, I only suggest using this as a temporary measure in the meantime.
Regarding your comment about recharging your OpenAI account, I'd suggest looking into Azure startup credits to cover OpenAI models (if you're not already aware of this option). Since Bedrock doesn't support the OpenAI API and isn't a replacement for us, using Azure for OpenAI models and AWS for Bedrock would give us full coverage with all the models and APIs we need under a single application—LibreChat. This approach could significantly reduce your costs.
@NN1985 Can you expand on what is generally covered by AWS Activate credits? Like are the Claude models covered by those credits, for example? I ask because Claude works through the marketplace, and AWS credits generally can't cover marketplace products.
@NN1985 Can you expand on what is generally covered by AWS Activate credits? Like are the Claude models covered by those credits, for example? I ask because Claude works through the marketplace, and AWS credits generally can't cover marketplace products.
I'm happy to clarify, though I'll keep it brief out of respect for Danny and the main topic of LibreChat. As of April, AWS Activate credits can indeed be used for Claude models, despite their availability through the marketplace. This change is part of a special exception, which you can find detailed in the updated terms and conditions of the credits. I've also verified this with AWS support, and our billing reports confirm that the credits are being correctly applied.
For more information, please refer to this article:
AWS Activate credits now accepted for third-party models on Amazon Bedrock
@danny-avila Are you also looking into Bedrock support on the rag_api side? It would be nice to have the ability to use the Titan / Cohere embedding models natively and have everything running in AWS.
Thanks @NN1985 for confirming the issue with LiteLLM. And more so for mentioning the AWS Activate. As big companies battle it out, the SMEs and entrepreneurs like us have only benefits to reap. Ever since we found LibreChat, we saved money in ChatGPT subscriptions. And now, we are holding up any further recharge to our OpenAI account to replace it with bedrock. Have you already tried the aws-samples repo for using bedrock? It could serve as a temporary alternative for us if there's some wait for LibreChat bedrock support..
No problem @aryansaurav . I am indeed using the linked repo project to access Bedrock directly for the Anthropic API, which would be prohibitively expensive to use at scale otherwise. I haven't found anything else quite like it at the moment—it's a true one-command AWS console zero-config install, and we were up and running in about 30 minutes. We can add all our users with a login similar to LibreChat, but be warned that it's quite basic and buggy in comparison.
I want to be clear that I strongly support LibreChat and vastly prefer it. Once native Bedrock endpoint support is added, or even better, true AWS native support, we would migrate immediately. As you mentioned, I only suggest using this as a temporary measure in the meantime.
Regarding your comment about recharging your OpenAI account, I'd suggest looking into Azure startup credits to cover OpenAI models (if you're not already aware of this option). Since Bedrock doesn't support the OpenAI API and isn't a replacement for us, using Azure for OpenAI models and AWS for Bedrock would give us full coverage with all the models and APIs we need under a single application—LibreChat. This approach could significantly reduce your costs.
Agree with you @NN1985. Our support for LibreChat is iron-clad . Not just because it's open source and independent of other companies, but also the containerized setup keeps things much simpler and better organized.
I just took a look at the AWS bedrock-claude-chat. Despite the one command simple setup it offers, it would create more than five different cloud resources.. which is not bad but can be a bit hectic to keep track of. Sometimes, if things go wrong (bot attack etc), your cloud resources can generate a huge spike in your billing in a day! (Happened to me once). Perhaps, a strategic implementation by AWS.
LibreChat requires one t2-micro instance (minimal). And chances of such unpleasant surprises are minimal. The only time I faced an issue was when I had mistakenly left the signup option on frontend enabled (some bots created accounts!!) I'll wait for a week or two for the LibreChat update before trying an alternative. Yes, Azure credits are on my radar too but will use them later (if the offer remains available).
@danny-avila Just as an update, I think we are going to move forward with using LiteLLM as suggested in other areas. I do think this would be a great enhancement though, to natively support Bedrock. We will revisit later, unless you get to it first.
Native AWS Setup is coming, and I've already tested extensively using Bedrock models.
@danny-avila Are there any updates on when is this going to be available. Thanks!
I also would love to see this @danny-avila and look forward to testing. Bedrock is a fantastic service and we have about 6ms latency to the bedrock runtime. Putting litellm in between makes things slower for us. Bedrock is also one of the lowest possible cybersecurity risks when running cloud based LLM as many organizations already have an existing relationship with AWS and approve of their cybersecurity approach and don't want to deal with new and unknown vendors.
There's an open PR, and I'm hoping to merge it today https://github.com/danny-avila/LibreChat/pull/3935
Updating docs soon.
Basic Setup:
Required environment variables:
BEDROCK_AWS_DEFAULT_REGION=us-east-1 # A default region must be provided
BEDROCK_AWS_ACCESS_KEY_ID=someAccessKey
BEDROCK_AWS_SECRET_ACCESS_KEY=someSecretAccessKey
# Note: This example list is not meant to be exhaustive. If omitted, all known, supported model IDs will be included for you.
BEDROCK_AWS_MODELS=anthropic.claude-3-5-sonnet-20240620-v1:0,meta.llama3-1-8b-instruct-v1:0
# See all Bedrock model IDs here: https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids.html#model-ids-arns
# Notes on specific models:
# 'ai21.j2-mid-v1', # Not supported, as it doesn't support streaming
# 'ai21.j2-ultra-v1', # Not supported, as it doesn't support conversation history
There are also librechat.yaml options you can configue:
endpoints:
bedrock:
availableRegions:
- "us-east-1"
- "us-west-2"
streamRate: 35
titleModel: 'anthropic.claude-3-haiku-20240307-v1:0'
streamRate - (optional) in milliseconds, like the existing option for other endpoints, determines the rate of processing each new token. This can be helpful to free up the event loop to stabilize processing of many concurrent requests. Can also help provide a smoother UX for frontend stream rendering
titleModel - (optional) sets the model to use for titles. Recommended: anthropic.claude-3-haiku-20240307-v1:0. Omit or set as current_model to use the model used for chat.
availableRegions - (optional) If provided, the parameters will include the regions you set as an optional dropdown. if a region is not selected, the default region is used instead. This is useful to also allow use of many regions, as region availability can vary per model
More info: https://github.com/danny-avila/LibreChat/pull/3935
Thanks a lot @danny-avila .. I have been trying to test it, though not successful yet. I still can't see the endpoint for bedrock at all.
I set the bedrock access key, secret key in the .env file as suggested. I also tried pasting the optional lines in librechat.yaml although I couldn't understand where they need to be inserted.
I have tried this on a fresh clone of librechat main branch and replaced the values of the bedrock credentials only.. Recreated docker containers. Still couldn't see the bedrock endpoint.
I know this may not be enough info to help me debug this. Please let me know if you would like to see any logs.. and if there is any other file that need to be edited except the .env and librechat.yaml files...
@aryansaurav
Check your ENDPOINTS environment variable, that would be the only thing limiting access to the endpoint, if you have everything else configured.
That and, making sure you are using the latest image: https://github.com/danny-avila/LibreChat/pkgs/container/librechat-dev
librechat-dev:latest not librechat:latest
I've successfully configured the new endpoint on every instance I manage.
Docker update instructions: https://www.librechat.ai/docs/local/docker#update-librechat
You mean the ENDPOINTS variable in .env file? @danny-avila It's disable in the .env.example
# ENDPOINTS=openAI,assistants,azureOpenAI,bingAI,google,gptPlugins,anthropic
do I enable it and add bedrock to it?
ENDPOINTS=openAI,assistants,azureOpenAI,bingAI,google,gptPlugins,anthropic,bedrock
@djaym7 which lines did you uncomment in the .env file?
@danny-avila for me it does not work either. I wipe everything (including removing all the docker images) and just clone the main LibreChat branch. The only thing i am configuring in .env is the 3 required BEDROCK entries, then I edit docker-compose.override.yml with the content below and run the whole thing with docker compose. I can login and the bedrock entry shows up but when I click on it nothing happens, same happens when I explicitly configure ENDPOINTS .
# ---------------------------------------------------
services:
# # USE LIBRECHAT CONFIG FILE
# api:
# volumes:
# - type: bind
# source: ./librechat.yaml
# target: /app/librechat.yaml
# # LOCAL BUILD
# api:
# image: librechat
# build:
# context: .
# target: node
# # BUILD FROM LATEST IMAGE
api:
image: ghcr.io/danny-avila/librechat-dev:latest
# # BUILD FROM LATEST IMAGE (NUMBERED RELEASE)
# api:
# image: ghcr.io/danny-avila/librechat:latest
# # BUILD FROM LATEST API IMAGE
# api:
# image: ghcr.io/danny-avila/librechat-dev-api:latest
ah, i see that LibreChat from github already has librechat-dev:latest so no override required, not clear on the difference between these 2 though
api:
image: ghcr.io/danny-avila/librechat-dev:latest
api:
image: ghcr.io/danny-avila/librechat-dev-api:latest