Support for AWS Bedrock

[abtawfik]

It would be nice to have support for Bedrock so that application that use AWS can integrate more seamlessly.

[samuelcolvin]

PR welcome to add AWS Bedrock.

I think from a brief search that Bedrock is like VertexAI - you use a custom API, but the format of the response matches the foundational model used, so actually using Bedrock with (say) Anthropic is more about customising the URL and auth, then using the (as yet unimplemented, see #63) Anthropic logic.

That's how The VertexAIModel works - it does it's own auth and sets the url, then just returns an instance of GeminiAgentModel.

[stephenhibbert]

@samuelcolvin - a couple of points to add...

The Anthropic SDK currently supports Amazon Bedrock though the InvokeModel API with a HTTPX client and boto3 for auth.

Amazon Bedrock also has a newer Converse API that provides a consistent interface for all the models it supports. This includes the new Amazon Nova models as well as models from Meta, Mistral and Cohere models, the full list is here.

Maybe it makes sense to add AnthropicBedrock support in the Anthropic PR (since I see you're using the anthropic python SDK) and then handle the broader Bedrock support separately with the Converse API?

[PabloSimonSainz]

Hi, I can help with this issue

[YanSte]

Any help needed ? I can work on it. @PabloSimonSainz or @samuelcolvin

[dmenini]

Following up on what @stephenhibbert said, currently the AnthropicModel class accepts the anthropic_client param with type AsyncAnthropic. However, the current logic also works with the AsyncAnthropicBedrock client from anthropic.

I agree that it would be nice to have built-in support for any model on Bedrock, but it's still a valid workaround for those who are using Anthropic models :) Although it would be nice to have the correct type hint to take both clients into account.

[stephenhibbert]

Spot on @dmenini. As a workaround we can install boto3 and pass an instance of AsyncAnthropicBedrock to the AnthropicModel like this

from anthropic import AsyncAnthropicBedrock
from pydantic_ai.models.anthropic import AnthropicModel
from pydantic_ai import Agent
from pydantic import BaseModel

anthropic_bedrock_client = AsyncAnthropicBedrock(aws_region='us-east-1')

model = AnthropicModel(
    model_name='anthropic.claude-3-haiku-20240307-v1:0',
    anthropic_client=anthropic_bedrock_client
)

class MyModel(BaseModel):
    city: str
    country: str

print(f'Using model: {model}')
agent = Agent(model, result_type=MyModel)

if __name__ == '__main__':
    result = agent.run_sync('The windy city in the US of A.')
    print(result.data)
    print(result.usage())


Using model: AnthropicModel(model_name='anthropic.claude-3-haiku-20240307-v1:0')    
city='Chicago' country='United States'
Usage(requests=1, request_tokens=459, response_tokens=56, total_tokens=515, details=None)

Process finished with exit code 0   

But it would be nice to have 1) documentation for this in the Anthropic section 2) correct type hints for Anthropic Bedrock models 3) full Bedrock support for any model using the ConverseAPI. Happy to help work on/review/test this too.

This https://github.com/pydantic/logfire/pull/701 should also mean the logfire integration will work out the box when merged.

[PierreColombo]

Hey, I'm still interrested in https://github.com/pydantic/logfire/pull/701 to be merged :) this will remove some friction to adapt my current code!

[Shekswess]

Hey guys !!! @samuelcolvin sorry for being late with this comment, do you plan to add Bedrock support ? I'm extremely excited to try PydanticAI with Bedrock models (mostly Claude models and the new AWS Nova models).... Can I somehow help and contribute ?

[samuelcolvin]

We will support Claude via bedrock, as it should just be setting the url and configuring auth.

Not sure about other models.

PR welcome to try to implement it, otherwise we'll try to get it it soon.

[wolfgangcolsman]

This repo maps the OpenAI API to Bedrock API. Maybe useful as another starting point.

https://github.com/aws-samples/bedrock-access-gateway/tree/main

[wolfgangcolsman]

Using Anthropic Bedrock does not work with llama 3.3 on AWS Bedrock.

anthropic.BadRequestError: Error code: 400 - {'message': 'Malformed input request: #: required key [prompt] not found#: extraneous key [system] is not permitted#: extraneous key [max_tokens] is not permitted#: extraneous key [messages] is not permitted#: extraneous key [tool_choice] is not permitted#: extraneous key [anthropic_version] is not permitted#: extraneous key [tools] is not permitted, please reformat your input and try again.'}

[PabloSimonSainz]

Using Anthropic Bedrock does not work with llama 3.3 on AWS Bedrock.

anthropic.BadRequestError: Error code: 400 - {'message': 'Malformed input request: #: required key [prompt] not found#: extraneous key [system] is not permitted#: extraneous key [max_tokens] is not permitted#: extraneous key [messages] is not permitted#: extraneous key [tool_choice] is not permitted#: extraneous key [anthropic_version] is not permitted#: extraneous key [tools] is not permitted, please reformat your input and try again.'}

To call models through aws bedrock, some models can be invoked through invoke_model and others with converse (for example anthropic/sonnet can only be called with converse).

As a note, another small consideration is the availability of the model in the region. models-regions

[PabloSimonSainz]

Regarding this issue, would there be any problem if it were addressed directly with the python aws SDK (boto3) with just the right dependencies?

[stephenhibbert]

To call models through aws bedrock, some models can be invoked through invoke_model and others with converse (for example anthropic/sonnet can only be called with converse).

That is incorrect. In fact, at the time of writing, the anthropic python SDK currently exclusively uses the invoke API and this works for the latest models: https://github.com/anthropics/anthropic-sdk-python/blob/93cbbbde964e244f02bf1bd2b579c5fabce4e267/src/anthropic/lib/bedrock/_client.py#L56. As well as the invoke API, there is also the newer converse API which provides a consistent interface and is supported for some of the newer models: https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference-supported-models-features.html

At the moment there is a valid workaround for using Anthropic models on Bedrock in my comment above. The bigger unlock is implementing wider support for Bedrock models (Llama, Nova, Anthropic, Mistral etc) which I would suggest be implemented by using the converse API for simplicity and ease of long term support in PydanticAI. The popular new models will very likely all be supported with the converse API moving forwards.

[abtawfik]

The bigger unlock is implementing wider support for Bedrock models (Llama, Nova, Anthropic, Mistral etc) which I would suggest be implemented by using the converse API for simplicity and ease of long term support in PydanticAI. The popular new models will very likely all be supported with the converse API moving forwards.

Totally agree with this statement. The converse api is going to be the standard going forward for boto3.

[PabloSimonSainz]

To call models through aws bedrock, some models can be invoked through invoke_model and others with converse (for example anthropic/sonnet can only be called with converse).

That is incorrect. In fact, at the time of writing, the anthropic python SDK currently exclusively uses the invoke API and this works for the latest models: https://github.com/anthropics/anthropic-sdk-python/blob/93cbbbde964e244f02bf1bd2b579c5fabce4e267/src/anthropic/lib/bedrock/_client.py#L56. As well as the invoke API, there is also the newer converse API which provides a consistent interface and is supported for some of the newer models: https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference-supported-models-features.html

At the moment there is a valid workaround for using Anthropic models on Bedrock in my comment above. The bigger unlock is implementing wider support for Bedrock models (Llama, Nova, Anthropic, Mistral etc) which I would suggest be implemented by using the converse API for simplicity and ease of long term support in PydanticAI. The popular new models will very likely all be supported with the converse API moving forwards.

I had that problem recently on a project with AWS Bedrock and had to use the converse API instead of the invoke API for anthropic models.

[stephenhibbert]

I had that problem recently on a project with AWS Bedrock and had to use the converse API instead of the invoke API for anthropic models.

@PabloSimonSainz It's hard to tell exactly what's wrong with your code without seeing the source but that error message suggests you're using the old style Anthropic Text Completions instead of the new Messages format. If you use the Messages API correctly with the invoke API then you can use the Haiku model and it will also work with the converse API. Hope that helps!

https://docs.anthropic.com/en/api/migrating-from-text-completions-to-messages https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-anthropic-claude-messages.html#claude-messages-supported-models

[Wh1isper]

Found this project https://github.com/aws-samples/bedrock-access-gateway which using AWS converse API to proxy bedrock api as openai api.

from openai import OpenAI

client = OpenAI(
    api_key="...",
    base_url="...",
)

from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIModel

model = OpenAIModel('us.amazon.nova-lite-v1:0', openai_client=client)
agent = Agent(model)

A few minor issues still, and I have a fork of my own https://github.com/Wh1isper/bedway

[Wh1isper]

UPDATE: https://github.com/pydantic/pydantic-ai/issues/118#issuecomment-2594989126

Hi everyone, I just draft a pr(https://github.com/pydantic/pydantic-ai/pull/680) that partially implements bedrock's functionality and may need help with it!

• Lint issue ~~- Implement converse_stream API~~

I may not have enough time to implement everything so feel free to cherry-pick!

You can try it with this repo and package: https://github.com/Wh1isper/pydantic-ai-bedrock

from pydantic_ai import Agent
from pydantic_ai_bedrock.bedrock import (
    BedrockModel,
)  # Replace from pydantic_ai.bedrock import BedrockModel when pydantic_ai support bedrock

model = BedrockModel(
    model_name="anthropic.claude-3-5-sonnet-20241022-v2:0",
)
agent = Agent(model, system_prompt="You are a helpful assistant.")


if __name__ == "__main__":
    result = agent.run_sync("Hello world!")
    print(result.data)
    print(result.usage())

Also, feel free to let me know if you want to contribute or own this package. Cheers

[lambda-science]

Very interested in this feature as Bedrock is a major gen AI provider in corporate environnement. We can't use PydanticAI without support for this for our company use-cases.

[Wh1isper]

Hello folks,

I’d like to share some updates about pydantic-ai-bedrock(https://github.com/Wh1isper/pydantic-ai-bedrock). I’ve implemented two APIs, converse and converse_stream, completed unit tests and done some real-world testings for the Nova and Claude 3.5 models.

However, due to type issues with boto3, my implementation may have difficulty passing the lint checks for this project, and the test cases are not elegant too...

I kindly request the maintainers(@samuelcolvin @sydney-runkle ) to provide some assistance on this, as the Bedrock model is a major generative AI provider that we all want to support. I will continue maintaining the pydantic-ai-bedrock(https://github.com/Wh1isper/pydantic-ai-bedrock) project until either pydantic-ai supports Bedrock or I no longer use Bedrock.

Here is the draft PR and feel free to cherry-pick: https://github.com/pydantic/pydantic-ai/pull/698

[wolfgangcolsman]

You may want to consider the new caching functions in the cost calculation to get more accurate numbers. Example: https://github.com/langchain-ai/langchain-aws/pull/324/files

[derekwsgray]

Hello folks,

I’d like to share some updates about pydantic-ai-bedrock(https://github.com/Wh1isper/pydantic-ai-bedrock). I’ve implemented two APIs, converse and converse_stream, completed unit tests and done some real-world testings for the Nova and Claude 3.5 models.

However, due to type issues with boto3, my implementation may have difficulty passing the lint checks for this project, and the test cases are not elegant too...

I kindly request the maintainers(@samuelcolvin @sydney-runkle ) to provide some assistance on this, as the Bedrock model is a major generative AI provider that we all want to support. I will continue maintaining the pydantic-ai-bedrock(https://github.com/Wh1isper/pydantic-ai-bedrock) project until either pydantic-ai supports Bedrock or I no longer use Bedrock.

Here is the draft PR and feel free to cherry-pick: #698

Oh lovely, wish I'd seen this sooner. I wrote my own ConverseModel too, but hadn't finished the converse_stream half.

[vladionescu]

Any update here? Looks like we've had a good PR two months ago but it was closed without any clear path forward.

[Kludex]

Yep. I have a PR open. I'll get it merged till end of the week. Maybe I have time tomorrow.

Sorry for the wait.

[gerardpc]

Just wanted to say that we're waiting for this to happen to add Pydantic AI support in my company 🎉

[drobbins-ancile]

Yep. I have a PR open. I'll get it merged till end of the week. Maybe I have time tomorrow.

Sorry for the wait.

Thanks for taking this one!

One request I would add is that it should support using the credentials from an assigned IAM role for the use-case where the application is hosted in AWS via lambda, ECS, etc.

It should also support using an AWS cli profile instead of specifying an access key/secret key.

[Wh1isper]

@drobbins-ancile We currently use boto3 SDK, which should fulfill this need: https://github.com/pydantic/pydantic-ai/pull/994/files#diff-7da0d2fa009c3fbd0c070716d4801f92a5267cc7b99c6961082c7d8e06efeaa2R57

[Kludex]

Latest Pydantic AI supports Bedrock.