llama-stack icon indicating copy to clipboard operation
llama-stack copied to clipboard

fix broken bedrock inference provider

Open moritalous opened this issue 5 months ago • 2 comments

Support for Bedrock inference providers has been applied with the following merges:

https://github.com/meta-llama/llama-stack/commit/95abbf576b4b078e72b779f534cbaf696e30ecab

However, it was overwritten in the next merge.

https://github.com/meta-llama/llama-stack/commit/56aed59eb4c9915676c6fc7aac009dad97e7ead2

As a result, Bedrock is not displayed as an inference provider.

llama stack list-providers inference
+------------------------+----------------------------------------------------------------------------------+
| Provider Type          | PIP Package Dependencies                                                         |
+------------------------+----------------------------------------------------------------------------------+
| remote                 |                                                                                  |
+------------------------+----------------------------------------------------------------------------------+
| meta-reference         | accelerate,blobfile,fairscale,fbgemm-                                            |
|                        | gpu==0.8.0,torch,torchvision,transformers,zmq                                    |
+------------------------+----------------------------------------------------------------------------------+
| remote::sample         |                                                                                  |
+------------------------+----------------------------------------------------------------------------------+
| remote::ollama         | ollama                                                                           |
+------------------------+----------------------------------------------------------------------------------+
| remote::tgi            | huggingface_hub,aiohttp                                                          |
+------------------------+----------------------------------------------------------------------------------+
| remote::hf::serverless | huggingface_hub,aiohttp                                                          |
+------------------------+----------------------------------------------------------------------------------+
| remote::hf::endpoint   | huggingface_hub,aiohttp                                                          |
+------------------------+----------------------------------------------------------------------------------+
| remote::fireworks      | fireworks-ai                                                                     |
+------------------------+----------------------------------------------------------------------------------+
| remote::together       | together                                                                         |
+------------------------+----------------------------------------------------------------------------------+

I fix it.

+------------------------+----------------------------------------------------------------------------------+
| remote::bedrock        | boto3                                                                            |
+------------------------+----------------------------------------------------------------------------------+

moritalous avatar Sep 29 '24 15:09 moritalous