vllm 400 status code (no body) error

[rickychen-infinirc]

Hello everyone, I use the vllm openapi service, but I encountered a 400 status code (no body) error. How can I change it? Thanks

vllm:

python -m vllm.entrypoints.openai.api_server --model /home/rickychen/桌面/llm/models/Infinirc-Llama3-8B-5G-v1.0 --dtype auto --worker-use-ray --tensor-parallel-size 2 --port 8001 --served-model-name Infinirc-Llama3-8B-5G-v1.0

huggingface chatui:

MODELS=`[
  {
    "name": "Infinirc-Llama3-8B-5G-v1.0",
    "id": "Infinirc-Llama3-8B-5G-v1.0",
    "endpoints": [{
      "type": "openai",
      "baseURL": "http://10.0.9.5:8001/v1",
      "apiKey": "a"
    }],
    "chatPromptTemplate": "{{#each messages}}{{#ifUser}}Human: {{content}}\n{{/ifUser}}{{#ifAssistant}}Assistant: {{content}}\n{{/ifAssistant}}{{/each}}Human: ",
    "promptExamples": [
      {
        "title": "Write an email from bullet list",
        "prompt": "As a restaurant owner, write a professional email to the supplier to get these products every week: \n\n- Wine (x10)\n- Eggs (x24)\n- Bread (x12)"
      },
      {
        "title": "Code a snake game",
        "prompt": "Code a basic snake game in python, give explanations for each step."
      },
      {
        "title": "Assist in a task",
        "prompt": "How do I make a delicious lemon cheesecake?"
      }
    ],
    "parameters": {
      "temperature": 0.1,
      "top_p": 0.95,
      "max_new_tokens": 1024
    }
  }
]`

error:

BadRequestError: 400 status code (no body)
    at APIError.generate (file:///Volumes/MacPro/LLM/ChatUI/chat-ui-main/node_modules/openai/error.mjs:41:20)
    at OpenAI.makeStatusError (file:///Volumes/MacPro/LLM/ChatUI/chat-ui-main/node_modules/openai/core.mjs:256:25)
    at OpenAI.makeRequest (file:///Volumes/MacPro/LLM/ChatUI/chat-ui-main/node_modules/openai/core.mjs:299:30)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async eval (/Volumes/MacPro/LLM/ChatUI/chat-ui-main/src/lib/server/endpoints/openai/endpointOai.ts:111:36)
    at async Module.generate (/Volumes/MacPro/LLM/ChatUI/chat-ui-main/src/lib/server/textGeneration/generate.ts:8:30)
    at async textGenerationWithoutTitle (/Volumes/MacPro/LLM/ChatUI/chat-ui-main/src/lib/server/textGeneration/index.ts:56:3)
    at async Module.mergeAsyncGenerators (/Volumes/MacPro/LLM/ChatUI/chat-ui-main/src/lib/utils/mergeAsyncGenerators.ts:13:34)
    at async Module.textGeneration (/Volumes/MacPro/LLM/ChatUI/chat-ui-main/src/lib/server/textGeneration/index.ts:24:3)
    at async Object.start (/Volumes/MacPro/LLM/ChatUI/chat-ui-main/src/routes/conversation/[id]/+server.ts:325:26) {
  status: 400,
  headers: {
    'content-length': '297',
    'content-type': 'application/json',
    date: 'Sun, 14 Jul 2024 12:47:33 GMT',
    server: 'uvicorn'
  },
  request_id: undefined,
  error: undefined,
  code: undefined,
  param: undefined,
  type: undefined
}
[20:47:33.972] ERROR (31253): 400 status code (no body)
    err: {
      "type": "BadRequestError",
      "message": "400 status code (no body)",
      "stack":
          Error: 400 status code (no body)
              at APIError.generate (file:///Volumes/MacPro/LLM/ChatUI/chat-ui-main/node_modules/openai/error.mjs:41:20)
              at OpenAI.makeStatusError (file:///Volumes/MacPro/LLM/ChatUI/chat-ui-main/node_modules/openai/core.mjs:256:25)
              at OpenAI.makeRequest (file:///Volumes/MacPro/LLM/ChatUI/chat-ui-main/node_modules/openai/core.mjs:299:30)
              at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
              at async eval (/Volumes/MacPro/LLM/ChatUI/chat-ui-main/src/lib/server/endpoints/openai/endpointOai.ts:111:36)
              at async Module.generateFromDefaultEndpoint (/Volumes/MacPro/LLM/ChatUI/chat-ui-main/src/lib/server/generateFromDefaultEndpoint.ts:11:23)
              at async generateTitle (/Volumes/MacPro/LLM/ChatUI/chat-ui-main/src/lib/server/textGeneration/title.ts:54:10)
              at async Module.generateTitleForConversation (/Volumes/MacPro/LLM/ChatUI/chat-ui-main/src/lib/server/textGeneration/title.ts:17:19)
      "status": 400,
      "headers": {
        "content-length": "1748",
        "content-type": "application/json",
        "date": "Sun, 14 Jul 2024 12:47:33 GMT",
        "server": "uvicorn"
      }
    }

[AnisAchkar]

I am having the same error. Any answer on this one.

[rickychen-infinirc]

Here’s my attempt to solve the problem with an example. Please clone the latest version of the project and save the following content as .env.local. Make sure to replace the API with your server URL and model name.

MODELS=`[
  {
    "name": "Infinirc-Llama3-8B-5G-v1.0",
    "id": "Infinirc-Llama3-8B-5G-v1.0",
    "parameters": {
      "temperature": 0.9,
      "top_p": 0.95,
      "repetition_penalty": 1.2,
      "top_k": 50,
      "truncate": 1000,
      "max_new_tokens": 1024,
      "stop": []
    },
    "endpoints": [{
      "type" : "openai",
      "baseURL": "http://192.169.1.11:8001/v1"
    }]
  }
]`

[rickychen-infinirc]

我有同樣的錯誤。關於這個的任何答案。

Here’s my attempt to solve the problem with an example. Please clone the latest version of the project and save the following content as .env.local. Make sure to replace the API with your server URL and model name.

MODELS=`[
  {
    "name": "Infinirc-Llama3-8B-5G-v1.0",
    "id": "Infinirc-Llama3-8B-5G-v1.0",
    "parameters": {
      "temperature": 0.9,
      "top_p": 0.95,
      "repetition_penalty": 1.2,
      "top_k": 50,
      "truncate": 1000,
      "max_new_tokens": 1024,
      "stop": []
    },
    "endpoints": [{
      "type" : "openai",
      "baseURL": "http://192.169.1.11:8001/v1"
    }]
  }
]`