llama-stack
llama-stack copied to clipboard
fix bedrock impl
This is needed after the async refactor. testplan:
- llama stack build --template bedrock --image-type conda
- llama stack run bedrock --port 5000
- python examples/inference/client.py localhost 5000
User>hello world, write me a 2 sentence poem about the moon
Assistant>
Here is a 2-sentence poem about the moon:
The moon glows bright in the midnight sky,
A beacon of wonder, as it passes by.
[ModelDefWithProvider(identifier='Llama3.1-8B-Instruct', llama_model='Llama3.1-8B-Instruct', metadata={}, provider_id='remote::bedrock'), ModelDefWithProvider(identifier='Llama3.1-70B-Instruct', llama_model='Llama3.1-70B-Instruct', metadata={}, provider_id='remote::bedrock'), ModelDefWithProvider(identifier='Llama3.1-405B-Instruct', llama_model='Llama3.1-405B-Instruct', metadata={}, provider_id='remote::bedrock')]