langchain icon indicating copy to clipboard operation
langchain copied to clipboard

support deepseek's token usage format

Open manukall opened this issue 7 months ago • 2 comments

Deepseek includes a choice with empty content in their token usage chunk when streaming. otherwise, their API is compatible with OpenAI's. This commit adds support for this chunk format.

Example of a token usage chunk from Deepseek:

data: {
   "id":"339eade8-b499-472e-bc0d-b267b6e4bceb",
   "object":"chat.completion.chunk",
   "created":1743968719,
   "model":"deepseek-chat",
   "system_fingerprint":"fp_3d5141a69a_prod0225",
   "choices":[
      {
         "index":0,
         "delta":{
            "content":""
         },
         "logprobs":null,
         "finish_reason":"stop"
      }
   ],
   "usage":{
      "prompt_tokens":11,
      "completion_tokens":11,
      "total_tokens":22,
      "prompt_tokens_details":{
         "cached_tokens":0
      },
      "prompt_cache_hit_tokens":0,
      "prompt_cache_miss_tokens":11
   }
}

manukall avatar Apr 06 '25 20:04 manukall

@brainlid I think I added the test you've asked for. Makes total sense.

However, looking at the test, I'm not sure we should emit a MessageDelta when the content = "". Would it make more sense to return :skip in those cases?

manukall avatar Apr 22 '25 15:04 manukall

Hi @manukall! Please check out the v0.4.0-rc.0 release. There are other breaking changes that I expect are needed for DeepSeek to be supported, but your TokenUsage update would be best applied to that branch as TokenUsage is being stored in the received Message's metadata.usage.

brainlid avatar Apr 23 '25 04:04 brainlid