text-generation-inference

text-generation-inference copied to clipboard

System Info

Yes, the output did not say whut error, it just said Server error: and then blank.

I am using a windows 11 environment with python 11 huggingface hub and Llama 3.2 11B Vision serverless interference

Information

• [ ] Docker
• [x] The CLI directly

Tasks

• [X] An officially supported command
• [ ] My own modifications

Reproduction

1. Run this code

from huggingface_hub import InferenceClient

client = InferenceClient(api_key="hf_xxxxxxxxxxxxxxxxx")

image_url = "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"

for message in client.chat_completion(
	model="meta-llama/Llama-3.2-11B-Vision-Instruct",
	messages=[
		{
			"role": "user",
			"content": [
				{"type": "image_url", "image_url": {"url": image_url}},
				{"type": "text", "text": "Describe this image in one sentence."},
			],
		}
	],
	max_tokens=500,
	stream=True,
):
	print(message.choices[0].delta.content, end="")

2. the error (along with first output: "The"):

TheTraceback (most recent call last):
  File "D:\projmom\instructttt.py", line 7, in <module>
    for message in client.chat_completion(
  File "D:\projmom\venv\Lib\site-packages\huggingface_hub\inference\_common.py", line 321, in _stream_chat_completion_response
    output = _format_chat_completion_stream_output(item)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\projmom\venv\Lib\site-packages\huggingface_hub\inference\_common.py", line 356, in _format_chat_completion_stream_output
    raise _parse_text_generation_error(json_payload["error"], json_payload.get("error_type"))
huggingface_hub.errors.GenerationError: Request failed during generation: Server error: 

Expected behavior

I expected it to just ouput the answer to my prompt and image.