text-generation-inference

text-generation-inference copied to clipboard

System Info

Trying to access the serverless inference endpoints using the OpenAI compatible route leads to status 400.

Invalid URL: missing field `name`

Information

• [ ] Docker
• [ ] The CLI directly

Tasks

• [x] An officially supported command
• [ ] My own modifications

Reproduction

Here is a curl command to run inference, requires to setup your HF_TOKEN.

curl https://api-inference.huggingface.co/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $HF_TOKEN" \
-d '{
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "messages": [
    {
      "role": "user",
      "content": "Write a short poem."
    }
  ]
}'

Expected behavior

This endpoint should be "openai" compatible.