leapfrogai icon indicating copy to clipboard operation
leapfrogai copied to clipboard

Return finish reason field & VLLM Refactor

Open CollectiveUnicorn opened this issue 1 year ago • 2 comments

When performing a completion or chat completion return the finish_reason according to the api spec.

  • Adds finish reason in completion/chat completion that mirrors the openai spec
  • Adds token usage in completion/chat completion responses that mirrors the openai spec

Closes https://github.com/defenseunicorns/leapfrogai/issues/463 Closes https://github.com/defenseunicorns/leapfrogai/issues/431

CollectiveUnicorn avatar May 09 '24 23:05 CollectiveUnicorn

Deploy Preview for leapfrogai-docs canceled.

Name Link
Latest commit 99661575447dd4871a298155cf60c73dde288cc0
Latest deploy log https://app.netlify.com/sites/leapfrogai-docs/deploys/664cd436bf4db50008143d93

netlify[bot] avatar May 09 '24 23:05 netlify[bot]

Conforming to the specs for token length will require updating all of the backends. It assumes that there's the token counting function. TBD if doing that in this commit.

CollectiveUnicorn avatar May 09 '24 23:05 CollectiveUnicorn