microsoft
Transformers issue when invoking the model via curl
curl "http://localhost:44413/v1/chat/completions"
-H "Content-Type: application/json"
-d '{
"model": "meta-llama/Llama-3.2-1B",
"messages": [
{"role": "user", "content": "what was my last input to you"}
]
}'
Got output in terminal {"object":"error","message":"As of transformers v4.44, default chat template is no longer allowed, so you must provide a chat template if the tokenizer does not define one. None","type":"BadRequestError","param":null,"code":400}(base)
From the server we get these logs (PatchedvLLMServer pid=1782948) ERROR 08-14 16:17:00 [serving_chat.py:222] Error in preprocessing prompt inputs (PatchedvLLMServer pid=1782948) ERROR 08-14 16:17:00 [serving_chat.py:222] Traceback (most recent call last): (PatchedvLLMServer pid=1782948) ERROR 08-14 16:17:00 [serving_chat.py:222] File Desktop/agent_lightning/agent-lightning/venv/lib/python3.10/site-packages/vllm/entrypoints/openai/serving_chat.py", line 205, in create_chat_completion (PatchedvLLMServer pid=1782948) ERROR 08-14 16:17:00 [serving_chat.py:222] ) = await self._preprocess_chat( (PatchedvLLMServer pid=1782948) ERROR 08-14 16:17:00 [serving_chat.py:222] File/Desktop/agent_lightning/agent-lightning/venv/lib/python3.10/site-packages/vllm/entrypoints/openai/serving_engine.py", line 813, in _preprocess_chat (PatchedvLLMServer pid=1782948) ERROR 08-14 16:17:00 [serving_chat.py:222] request_prompt = apply_hf_chat_template( (PatchedvLLMServer pid=1782948) ERROR 08-14 16:17:00 [serving_chat.py:222] File /Desktop/agent_lightning/agent-lightning/venv/lib/python3.10/site-packages/vllm/utils/init.py", line 1292, in inner (PatchedvLLMServer pid=1782948) ERROR 08-14 16:17:00 [serving_chat.py:222] return fn(*args, **kwargs) (PatchedvLLMServer pid=1782948) ERROR 08-14 16:17:00 [serving_chat.py:222] File /Desktop/agent_lightning/agent-lightning/venv/lib/python3.10/site-packages/vllm/entrypoints/chat_utils.py", line 1212, in apply_hf_chat_template (PatchedvLLMServer pid=1782948) ERROR 08-14 16:17:00 [serving_chat.py:222] raise ValueError( (PatchedvLLMServer pid=1782948) ERROR 08-14 16:17:00 [serving_chat.py:222] ValueError: As of transformers v4.44, default chat template is no longer allowed, so you must provide a chat template if the tokenizer does not define one.
Tried downgrading the transformers version , but didnt work, Please suggest the workaround, Do we need to apply jinja template where vllm server is being started?
