Aiden Dai

Thanks for raising this request. This repo was originally built using InvokeModel api, and then changed to Converse API later. Currently, there is no plan to reverse this change back...

Sorry, have been busy recently. Llama 3.1 should be supported now, you can follow this https://github.com/aws-samples/bedrock-access-gateway#how-to-upgrade to start testing it. Note that Tool call for Llama 3.1 is not yet...

Llama 3.1 / Llama 3.2 are now supportted, please try again and raise a new one if any further issues. Closing this old issue.

Concurrency and asynchronous call is natively supportted by [FastAPI](https://fastapi.tiangolo.com/async/#concurrency-and-async-await), I did a quick test with 2 concurrency requests (with long response) and I can see both are streaming in parallel,...

Sorry, it's been a long time to address this issue. Now the performance is improved based on my test. Now this project make async call to converse api. We don't...

Thanks for the PR, I will need to validate it first. Will try to update as soon as I can.

For urgent usage, please do a customization to the code and deploy and host your own ECR images.

This may be a limit of Amazon Bedrock. Can you please: 1. Try to use the SDK directly and see if the issue exists: https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-anthropic-claude-messages.html 2. Provide the detailed response...

Close for no updates.

Thanks for the issue. The tiktoken is only used in the Embedding API to ensure compatibility. The recommendation is that you customize the code and deploy your own ECR image...