llama-stack icon indicating copy to clipboard operation
llama-stack copied to clipboard

fix bedrock impl

Open dineshyv opened this issue 1 year ago • 0 comments

This is needed after the async refactor. testplan:

  • llama stack build --template bedrock --image-type conda
  • llama stack run bedrock --port 5000
  • python examples/inference/client.py localhost 5000
User>hello world, write me a 2 sentence poem about the moon
Assistant>

Here is a 2-sentence poem about the moon:

The moon glows bright in the midnight sky,
A beacon of wonder, as it passes by.
[ModelDefWithProvider(identifier='Llama3.1-8B-Instruct', llama_model='Llama3.1-8B-Instruct', metadata={}, provider_id='remote::bedrock'), ModelDefWithProvider(identifier='Llama3.1-70B-Instruct', llama_model='Llama3.1-70B-Instruct', metadata={}, provider_id='remote::bedrock'), ModelDefWithProvider(identifier='Llama3.1-405B-Instruct', llama_model='Llama3.1-405B-Instruct', metadata={}, provider_id='remote::bedrock')]

dineshyv avatar Nov 02 '24 18:11 dineshyv