Missing number of Input token and output token in output response.

[kambalaashok4]

when an API request is sent to the bedrock-mistral model below details are missing. "prompt_tokens":0,"completion_tokens":0,"total_tokens":0

Please complete the following information: API used: api/v1/chat/completions model used: mistral.mistral-7b-instruct-v0:2

To Reproduce curl http://albenpoint/api/v1/chat/completions -H "Content-Type: application/json" -H "Authorization: Bearer bedrock" -d '{ "model": "mistral.mistral-7b-instruct-v0:2", "messages": [{"role":"user","content":"what is hyperloop"}] }'

Expected behavior Output response must contain a number of input tokens and output tokens, prompt tokens

Screenshots

[daixba]

unfortunately, the bedrock SDK does not provide any info about the token usage for mistral/mixtral model when using invoke_model so far. Other models are fine.

You can switch on the streaming mode (set stream to true) to get the token usage at this stage.

[kambalaashok4]

Thanks, daixba. can you please help me to where I can set stream to true(which .py file)

[daixba]

It's a parameter in openAI chat completion API. See https://cookbook.openai.com/examples/how_to_stream_completions for more details.

[daixba]

This is now supported with the new bedrock api. Please try it again.