sglang
sglang copied to clipboard
Published
1 month ago
•
sgl-project
sgl-project
[Bug] pydantic validation errors for ChatCompletion
Checklist
- [x] 1. I have searched related issues but cannot get the expected help.
- [x] 2. The bug has not been fixed in the latest version.
- [ ] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
- [ ] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
- [ ] 5. Please use English, otherwise it will be closed.
Describe the bug
when use autogen with qwen2.5
messages=[SystemMessage(content='You are a helpful AI assistant. Solve tasks using your tools. Reply with TERMINATE when the task has been completed.', type='SystemMessage'), UserMessage(content='shanghai weather', source='user', type='UserMessage')], client.chat.completions.create( messages=messages, stream=False, tools=converted_tools, **create_args, ) when using the SGLang inference framework, it can normally infer and call the function named "get_weather." However, if the conversation continues and carries the history of previous chats, there will be an issue.
messages=[SystemMessage(content='You are a helpful AI assistant. Solve tasks using your tools. Reply with TERMINATE when the task has been completed.', type='SystemMessage'), UserMessage(content='shanghai weather', source='user', type='UserMessage'), AssistantMessage(content=[FunctionCall(id='0', arguments='{"city": "shanghai"}', name='get_weather')], source='weather_agent', type='AssistantMessage'), FunctionExecutionResultMessage(content=[FunctionExecutionResult(content='The weather in shanghai is 73 degrees and Sunny.', call_id='0', is_error=False)], type='FunctionExecutionResultMessage'), UserMessage(content='beijing weather', source='user', type='UserMessage')]，
pydantic_core._pydantic_core.ValidationError: 5 validation errors for ChatCompletionRequest messages.2.ChatCompletionMessageGenericParam.content.str Input should be a valid string [type=string_type, input_value=None, input_type=NoneType] For further information visit https://errors.pydantic.dev/2.10/v/string_type messages.2.ChatCompletionMessageGenericParam.content.list[ChatCompletionMessageContentTextPart] Input should be a valid list [type=list_type, input_value=None, input_type=NoneType] For further information visit https://errors.pydantic.dev/2.10/v/list_type messages.2.ChatCompletionMessageUserParam.role Input should be 'user' [type=literal_error, input_value='assistant', input_type=str] For further information visit https://errors.pydantic.dev/2.10/v/literal_error messages.2.ChatCompletionMessageUserParam.content.str Input should be a valid string [type=string_type, input_value=None, input_type=NoneType] For further information visit https://errors.pydantic.dev/2.10/v/string_type messages.2.ChatCompletionMessageUserParam.content.list[union[ChatCompletionMessageContentTextPart,ChatCompletionMessageContentImagePart]] Input should be a valid list [type=list_type, input_value=None, input_type=NoneType] For further information visit https://errors.pydantic.dev/2.10/v/list_type
Reproduction
python3 -m sglang.launch_server --model-path /mnt/vdb1/model/qwen/Qwen2___5-7B-Instruct-GPTQ-Int4/ --served-model-name qwen2-72b --host 0.0.0.0 --port 8001 --trust-remote-code --mem-fraction-static 0.2 --chunked-prefill-size 8192 --schedule-conservativeness 0.3 --tool-call-parser qwen25
Environment
ubuntu autogen 0.4.6
I'm also seeing this in my PR https://github.com/sgl-project/sglang/pull/2798, when the input is None?
@shuaills Seeing the exact same issue, any tool call will fail on qwen25 model (I'm using Qwen2.5-72B-Instruct, apparently deepseek model have the same problem). Can someone help look into this? This is blocking and I had to fallback to vLLM beacuse of this :( I appreciate any insights to this problem.
I can confirm my LangGraph script is correct because the same script when using GPT4o API works, but fails when using SGLang.
The sglang server errors when a toolcall result (ToolMessage) is added to the context: (flow works like following snippet output from LangGraph). It is the same flow as the issue author @xxll88 , SGLang can call tools properly, but it cannot receive the tool results.
Original Query -> Model Choose Tool -> Tool Response -> Model (Fails here)
[-1:checkpoint] State at the end of step -1:
{'messages': []}
[0:tasks] Starting 1 task for step 0:
- __start__ -> {'messages': [HumanMessage(content='CALL A Search TOOL NOW, on how to cook a lobster', additional_kwargs={}, response_metadata={}, id='ca68f151-298b-4165-a973-c8e25a358fd9')]}
[0:writes] Finished step 0 with writes to 1 channel:
- messages -> [HumanMessage(content='CALL A Search TOOL NOW, on how to cook a lobster', additional_kwargs={}, response_metadata={}, id='ca68f151-298b-4165-a973-c8e25a358fd9')]
[0:checkpoint] State at the end of step 0:
{'messages': [HumanMessage(content='CALL A Search TOOL NOW, on how to cook a lobster', additional_kwargs={}, response_metadata={}, id='ca68f151-298b-4165-a973-c8e25a358fd9')]}
[1:tasks] Starting 1 task for step 1:
- agent -> {'is_last_step': False,
'messages': [HumanMessage(content='CALL A Search TOOL NOW, on how to cook a lobster', additional_kwargs={}, response_metadata={}, id='ca68f151-298b-4165-a973-c8e25a358fd9')],
'remaining_steps': 24}
[1:writes] Finished step 1 with writes to 1 channel:
- messages -> [AIMessage(content='', additional_kwargs={'tool_calls': [{'id': '0', 'function': {'arguments': '{"query": "how to cook a lobster"}', 'name': 'duckduckgo_results_json'}, 'type': 'function'}], 'refusal': None}, response_metadata={'token_usage': {'completion_tokens': 27, 'prompt_tokens': 197, 'total_tokens': 224, 'completion_tokens_details': None, 'prompt_tokens_details': None}, 'model_name': 'default', 'system_fingerprint': None, 'finish_reason': 'stop', 'logprobs': None}, id='run-4a86d515-a34a-41cb-82c7-a3e25ef7b9f1-0', tool_calls=[{'name': 'duckduckgo_results_json', 'args': {'query': 'how to cook a lobster'}, 'id': '0', 'type': 'tool_call'}], usage_metadata={'input_tokens': 197, 'output_tokens': 27, 'total_tokens': 224, 'input_token_details': {}, 'output_token_details': {}})]
[1:checkpoint] State at the end of step 1:
{'messages': [HumanMessage(content='CALL A Search TOOL NOW, on how to cook a lobster', additional_kwargs={}, response_metadata={}, id='ca68f151-298b-4165-a973-c8e25a358fd9'),
AIMessage(content='', additional_kwargs={'tool_calls': [{'id': '0', 'function': {'arguments': '{"query": "how to cook a lobster"}', 'name': 'duckduckgo_results_json'}, 'type': 'function'}], 'refusal': None}, response_metadata={'token_usage': {'completion_tokens': 27, 'prompt_tokens': 197, 'total_tokens': 224, 'completion_tokens_details': None, 'prompt_tokens_details': None}, 'model_name': 'default', 'system_fingerprint': None, 'finish_reason': 'stop', 'logprobs': None}, id='run-4a86d515-a34a-41cb-82c7-a3e25ef7b9f1-0', tool_calls=[{'name': 'duckduckgo_results_json', 'args': {'query': 'how to cook a lobster'}, 'id': '0', 'type': 'tool_call'}], usage_metadata={'input_tokens': 197, 'output_tokens': 27, 'total_tokens': 224, 'input_token_details': {}, 'output_token_details': {}})]}
[2:tasks] Starting 1 task for step 2:
- tools -> {'is_last_step': False,
'messages': [HumanMessage(content='CALL A Search TOOL NOW, on how to cook a lobster', additional_kwargs={}, response_metadata={}, id='ca68f151-298b-4165-a973-c8e25a358fd9'),
AIMessage(content='', additional_kwargs={'tool_calls': [{'id': '0', 'function': {'arguments': '{"query": "how to cook a lobster"}', 'name': 'duckduckgo_results_json'}, 'type': 'function'}], 'refusal': None}, response_metadata={'token_usage': {'completion_tokens': 27, 'prompt_tokens': 197, 'total_tokens': 224, 'completion_tokens_details': None, 'prompt_tokens_details': None}, 'model_name': 'default', 'system_fingerprint': None, 'finish_reason': 'stop', 'logprobs': None}, id='run-4a86d515-a34a-41cb-82c7-a3e25ef7b9f1-0', tool_calls=[{'name': 'duckduckgo_results_json', 'args': {'query': 'how to cook a lobster'}, 'id': '0', 'type': 'tool_call'}], usage_metadata={'input_tokens': 197, 'output_tokens': 27, 'total_tokens': 224, 'input_token_details': {}, 'output_token_details': {}})],
'remaining_steps': 23}
[2:writes] Finished step 2 with writes to 1 channel:
- messages -> [ToolMessage(content='[{"snippet": "From that point, boil the lobsters for 7 to 14 minutes or longer, depending on the size of the lobster. 7 to 10 minutes for a 1-pound lobster, 8 to 12 minutes for a 1 1/4-pound lobster, and 10 to 14 minutes for a 1 1/2-pound lobster.", "title": "How to Boil and Eat Lobster - Simply Recipes", "link": "https://www.simplyrecipes.com/recipes/how_to_boil_and_eat_lobster/"}, {"snippet": "Learn how to choose, prepare, and cook lobster in different methods, such as boiling, steaming, and baking. Find out how to break down, serve, and explore the versatility of this culinary treasure.", "title": "How to Cook Lobster: Foolproof Recipe for Perfect Results Every Time", "link": "https://oliviaoven.com/how-to-cook-lobster/"}, {"snippet": "For humane cooking, place the lobster in the freezer for 15-20 minutes to gently put it to sleep. 3. Master Boiling. Boiling is one of the simplest ways to cook a lobster. Bring a large pot of salted water to a rolling boil, then add your lobster headfirst. Cook for about 7-8 minutes per pound. The shell should turn bright red when it\'s done. 4.", "title": "Ways to Cook a Lobster: 10 Beginner Tips for Fresh Results", "link": "https://www.wholesalemainelobster.com/blog/top-10-fresh-lobster-cooking-tips-for-beginners/"}, {"snippet": "Learn how to cook lobster at home with various methods, such as boiling, steaming, and grilling. Find out how to choose the best type of lobster, what tools to use, and how to serve it for a restaurant-quality experience.", "title": "How to Cook Lobster: Step-by-Step Guide for Perfect Lobster at Home", "link": "https://globalseafoods.com/blogs/recipes/how-to-cook-a-lobster-a-step-by-step-guide"}]', name='duckduckgo_results_json', tool_call_id='0', artifact=[{'snippet': 'From that point, boil the lobsters for 7 to 14 minutes or longer, depending on the size of the lobster. 7 to 10 minutes for a 1-pound lobster, 8 to 12 minutes for a 1 1/4-pound lobster, and 10 to 14 minutes for a 1 1/2-pound lobster.', 'title': 'How to Boil and Eat Lobster - Simply Recipes', 'link': 'https://www.simplyrecipes.com/recipes/how_to_boil_and_eat_lobster/'}, {'snippet': 'Learn how to choose, prepare, and cook lobster in different methods, such as boiling, steaming, and baking. Find out how to break down, serve, and explore the versatility of this culinary treasure.', 'title': 'How to Cook Lobster: Foolproof Recipe for Perfect Results Every Time', 'link': 'https://oliviaoven.com/how-to-cook-lobster/'}, {'snippet': "For humane cooking, place the lobster in the freezer for 15-20 minutes to gently put it to sleep. 3. Master Boiling. Boiling is one of the simplest ways to cook a lobster. Bring a large pot of salted water to a rolling boil, then add your lobster headfirst. Cook for about 7-8 minutes per pound. The shell should turn bright red when it's done. 4.", 'title': 'Ways to Cook a Lobster: 10 Beginner Tips for Fresh Results', 'link': 'https://www.wholesalemainelobster.com/blog/top-10-fresh-lobster-cooking-tips-for-beginners/'}, {'snippet': 'Learn how to cook lobster at home with various methods, such as boiling, steaming, and grilling. Find out how to choose the best type of lobster, what tools to use, and how to serve it for a restaurant-quality experience.', 'title': 'How to Cook Lobster: Step-by-Step Guide for Perfect Lobster at Home', 'link': 'https://globalseafoods.com/blogs/recipes/how-to-cook-a-lobster-a-step-by-step-guide'}])]
[2:checkpoint] State at the end of step 2:
{'messages': [HumanMessage(content='CALL A Search TOOL NOW, on how to cook a lobster', additional_kwargs={}, response_metadata={}, id='ca68f151-298b-4165-a973-c8e25a358fd9'),
AIMessage(content='', additional_kwargs={'tool_calls': [{'id': '0', 'function': {'arguments': '{"query": "how to cook a lobster"}', 'name': 'duckduckgo_results_json'}, 'type': 'function'}], 'refusal': None}, response_metadata={'token_usage': {'completion_tokens': 27, 'prompt_tokens': 197, 'total_tokens': 224, 'completion_tokens_details': None, 'prompt_tokens_details': None}, 'model_name': 'default', 'system_fingerprint': None, 'finish_reason': 'stop', 'logprobs': None}, id='run-4a86d515-a34a-41cb-82c7-a3e25ef7b9f1-0', tool_calls=[{'name': 'duckduckgo_results_json', 'args': {'query': 'how to cook a lobster'}, 'id': '0', 'type': 'tool_call'}], usage_metadata={'input_tokens': 197, 'output_tokens': 27, 'total_tokens': 224, 'input_token_details': {}, 'output_token_details': {}}),
ToolMessage(content='[{"snippet": "From that point, boil the lobsters for 7 to 14 minutes or longer, depending on the size of the lobster. 7 to 10 minutes for a 1-pound lobster, 8 to 12 minutes for a 1 1/4-pound lobster, and 10 to 14 minutes for a 1 1/2-pound lobster.", "title": "How to Boil and Eat Lobster - Simply Recipes", "link": "https://www.simplyrecipes.com/recipes/how_to_boil_and_eat_lobster/"}, {"snippet": "Learn how to choose, prepare, and cook lobster in different methods, such as boiling, steaming, and baking. Find out how to break down, serve, and explore the versatility of this culinary treasure.", "title": "How to Cook Lobster: Foolproof Recipe for Perfect Results Every Time", "link": "https://oliviaoven.com/how-to-cook-lobster/"}, {"snippet": "For humane cooking, place the lobster in the freezer for 15-20 minutes to gently put it to sleep. 3. Master Boiling. Boiling is one of the simplest ways to cook a lobster. Bring a large pot of salted water to a rolling boil, then add your lobster headfirst. Cook for about 7-8 minutes per pound. The shell should turn bright red when it\'s done. 4.", "title": "Ways to Cook a Lobster: 10 Beginner Tips for Fresh Results", "link": "https://www.wholesalemainelobster.com/blog/top-10-fresh-lobster-cooking-tips-for-beginners/"}, {"snippet": "Learn how to cook lobster at home with various methods, such as boiling, steaming, and grilling. Find out how to choose the best type of lobster, what tools to use, and how to serve it for a restaurant-quality experience.", "title": "How to Cook Lobster: Step-by-Step Guide for Perfect Lobster at Home", "link": "https://globalseafoods.com/blogs/recipes/how-to-cook-a-lobster-a-step-by-step-guide"}]', name='duckduckgo_results_json', id='82c9c5ed-a9ba-4c19-9ce1-60344c5ceeef', tool_call_id='0', artifact=[{'snippet': 'From that point, boil the lobsters for 7 to 14 minutes or longer, depending on the size of the lobster. 7 to 10 minutes for a 1-pound lobster, 8 to 12 minutes for a 1 1/4-pound lobster, and 10 to 14 minutes for a 1 1/2-pound lobster.', 'title': 'How to Boil and Eat Lobster - Simply Recipes', 'link': 'https://www.simplyrecipes.com/recipes/how_to_boil_and_eat_lobster/'}, {'snippet': 'Learn how to choose, prepare, and cook lobster in different methods, such as boiling, steaming, and baking. Find out how to break down, serve, and explore the versatility of this culinary treasure.', 'title': 'How to Cook Lobster: Foolproof Recipe for Perfect Results Every Time', 'link': 'https://oliviaoven.com/how-to-cook-lobster/'}, {'snippet': "For humane cooking, place the lobster in the freezer for 15-20 minutes to gently put it to sleep. 3. Master Boiling. Boiling is one of the simplest ways to cook a lobster. Bring a large pot of salted water to a rolling boil, then add your lobster headfirst. Cook for about 7-8 minutes per pound. The shell should turn bright red when it's done. 4.", 'title': 'Ways to Cook a Lobster: 10 Beginner Tips for Fresh Results', 'link': 'https://www.wholesalemainelobster.com/blog/top-10-fresh-lobster-cooking-tips-for-beginners/'}, {'snippet': 'Learn how to cook lobster at home with various methods, such as boiling, steaming, and grilling. Find out how to choose the best type of lobster, what tools to use, and how to serve it for a restaurant-quality experience.', 'title': 'How to Cook Lobster: Step-by-Step Guide for Perfect Lobster at Home', 'link': 'https://globalseafoods.com/blogs/recipes/how-to-cook-a-lobster-a-step-by-step-guide'}])]}
[3:tasks] Starting 1 task for step 3:
- agent -> {'is_last_step': False,
'messages': [HumanMessage(content='CALL A Search TOOL NOW, on how to cook a lobster', additional_kwargs={}, response_metadata={}, id='ca68f151-298b-4165-a973-c8e25a358fd9'),
AIMessage(content='', additional_kwargs={'tool_calls': [{'id': '0', 'function': {'arguments': '{"query": "how to cook a lobster"}', 'name': 'duckduckgo_results_json'}, 'type': 'function'}], 'refusal': None}, response_metadata={'token_usage': {'completion_tokens': 27, 'prompt_tokens': 197, 'total_tokens': 224, 'completion_tokens_details': None, 'prompt_tokens_details': None}, 'model_name': 'default', 'system_fingerprint': None, 'finish_reason': 'stop', 'logprobs': None}, id='run-4a86d515-a34a-41cb-82c7-a3e25ef7b9f1-0', tool_calls=[{'name': 'duckduckgo_results_json', 'args': {'query': 'how to cook a lobster'}, 'id': '0', 'type': 'tool_call'}], usage_metadata={'input_tokens': 197, 'output_tokens': 27, 'total_tokens': 224, 'input_token_details': {}, 'output_token_details': {}}),
ToolMessage(content='[{"snippet": "From that point, boil the lobsters for 7 to 14 minutes or longer, depending on the size of the lobster. 7 to 10 minutes for a 1-pound lobster, 8 to 12 minutes for a 1 1/4-pound lobster, and 10 to 14 minutes for a 1 1/2-pound lobster.", "title": "How to Boil and Eat Lobster - Simply Recipes", "link": "https://www.simplyrecipes.com/recipes/how_to_boil_and_eat_lobster/"}, {"snippet": "Learn how to choose, prepare, and cook lobster in different methods, such as boiling, steaming, and baking. Find out how to break down, serve, and explore the versatility of this culinary treasure.", "title": "How to Cook Lobster: Foolproof Recipe for Perfect Results Every Time", "link": "https://oliviaoven.com/how-to-cook-lobster/"}, {"snippet": "For humane cooking, place the lobster in the freezer for 15-20 minutes to gently put it to sleep. 3. Master Boiling. Boiling is one of the simplest ways to cook a lobster. Bring a large pot of salted water to a rolling boil, then add your lobster headfirst. Cook for about 7-8 minutes per pound. The shell should turn bright red when it\'s done. 4.", "title": "Ways to Cook a Lobster: 10 Beginner Tips for Fresh Results", "link": "https://www.wholesalemainelobster.com/blog/top-10-fresh-lobster-cooking-tips-for-beginners/"}, {"snippet": "Learn how to cook lobster at home with various methods, such as boiling, steaming, and grilling. Find out how to choose the best type of lobster, what tools to use, and how to serve it for a restaurant-quality experience.", "title": "How to Cook Lobster: Step-by-Step Guide for Perfect Lobster at Home", "link": "https://globalseafoods.com/blogs/recipes/how-to-cook-a-lobster-a-step-by-step-guide"}]', name='duckduckgo_results_json', id='82c9c5ed-a9ba-4c19-9ce1-60344c5ceeef', tool_call_id='0', artifact=[{'snippet': 'From that point, boil the lobsters for 7 to 14 minutes or longer, depending on the size of the lobster. 7 to 10 minutes for a 1-pound lobster, 8 to 12 minutes for a 1 1/4-pound lobster, and 10 to 14 minutes for a 1 1/2-pound lobster.', 'title': 'How to Boil and Eat Lobster - Simply Recipes', 'link': 'https://www.simplyrecipes.com/recipes/how_to_boil_and_eat_lobster/'}, {'snippet': 'Learn how to choose, prepare, and cook lobster in different methods, such as boiling, steaming, and baking. Find out how to break down, serve, and explore the versatility of this culinary treasure.', 'title': 'How to Cook Lobster: Foolproof Recipe for Perfect Results Every Time', 'link': 'https://oliviaoven.com/how-to-cook-lobster/'}, {'snippet': "For humane cooking, place the lobster in the freezer for 15-20 minutes to gently put it to sleep. 3. Master Boiling. Boiling is one of the simplest ways to cook a lobster. Bring a large pot of salted water to a rolling boil, then add your lobster headfirst. Cook for about 7-8 minutes per pound. The shell should turn bright red when it's done. 4.", 'title': 'Ways to Cook a Lobster: 10 Beginner Tips for Fresh Results', 'link': 'https://www.wholesalemainelobster.com/blog/top-10-fresh-lobster-cooking-tips-for-beginners/'}, {'snippet': 'Learn how to cook lobster at home with various methods, such as boiling, steaming, and grilling. Find out how to choose the best type of lobster, what tools to use, and how to serve it for a restaurant-quality experience.', 'title': 'How to Cook Lobster: Step-by-Step Guide for Perfect Lobster at Home', 'link': 'https://globalseafoods.com/blogs/recipes/how-to-cook-a-lobster-a-step-by-step-guide'}])],
'remaining_steps': 22}
Actual OpenAI SDK sent request:
{'messages': [{'content': 'CALL A Search TOOL NOW, on how to cook a lobster', 'role': 'user'}, {'content': None, 'role': 'assistant', 'tool_calls': [{'type': 'function', 'id': '0', 'function': {'name': 'duckduckgo_results_json', 'arguments': '{"query": "how to cook a lobster"}'}}]}, {'content': '[{"snippet": "From that point, boil the lobsters for 7 to 14 minutes or longer, depending on the size of the lobster. 7 to 10 minutes for a 1-pound lobster, 8 to 12 minutes for a 1 1/4-pound lobster, and 10 to 14 minutes for a 1 1/2-pound lobster.", "title": "How to Boil and Eat Lobster - Simply Recipes", "link": "https://www.simplyrecipes.com/recipes/how_to_boil_and_eat_lobster/"}, {"snippet": "Learn how to choose, prepare, and cook lobster in different methods, such as boiling, steaming, and baking. Find out how to break down, serve, and explore the versatility of this culinary treasure.", "title": "How to Cook Lobster: Foolproof Recipe for Perfect Results Every Time", "link": "https://oliviaoven.com/how-to-cook-lobster/"}, {"snippet": "For humane cooking, place the lobster in the freezer for 15-20 minutes to gently put it to sleep. 3. Master Boiling. Boiling is one of the simplest ways to cook a lobster. Bring a large pot of salted water to a rolling boil, then add your lobster headfirst. Cook for about 7-8 minutes per pound. The shell should turn bright red when it\'s done. 4.", "title": "Ways to Cook a Lobster: 10 Beginner Tips for Fresh Results", "link": "https://www.wholesalemainelobster.com/blog/top-10-fresh-lobster-cooking-tips-for-beginners/"}, {"snippet": "Learn how to cook lobster at home with various methods, such as boiling, steaming, and grilling. Find out how to choose the best type of lobster, what tools to use, and how to serve it for a restaurant-quality experience.", "title": "How to Cook Lobster: Step-by-Step Guide for Perfect Lobster at Home", "link": "https://globalseafoods.com/blogs/recipes/how-to-cook-a-lobster-a-step-by-step-guide"}]', 'role': 'tool', 'tool_call_id': '0'}], 'model': 'default', 'audio': NOT_GIVEN, 'frequency_penalty': NOT_GIVEN, 'function_call': NOT_GIVEN, 'functions': NOT_GIVEN, 'logit_bias': NOT_GIVEN, 'logprobs': NOT_GIVEN, 'max_completion_tokens': 256, 'max_tokens': NOT_GIVEN, 'metadata': NOT_GIVEN, 'modalities': NOT_GIVEN, 'n': NOT_GIVEN, 'parallel_tool_calls': False, 'prediction': NOT_GIVEN, 'presence_penalty': NOT_GIVEN, 'reasoning_effort': NOT_GIVEN, 'response_format': NOT_GIVEN, 'seed': NOT_GIVEN, 'service_tier': NOT_GIVEN, 'stop': NOT_GIVEN, 'store': NOT_GIVEN, 'stream': False, 'stream_options': NOT_GIVEN, 'temperature': 0.0, 'tool_choice': NOT_GIVEN, 'tools': [{'type': 'function', 'function': {'name': 'duckduckgo_results_json', 'description': 'A wrapper around Duck Duck Go Search. Useful for when you need to answer questions about current events. Input should be a search query.', 'parameters': {'properties': {'query': {'description': 'search query to look up', 'type': 'string'}}, 'required': ['query'], 'type': 'object'}}}], 'top_logprobs': NOT_GIVEN, 'top_p': NOT_GIVEN, 'user': NOT_GIVEN}
Only messages part in json:
{
"messages": [
{
"content": "CALL A Search TOOL NOW, on how to cook a lobster",
"role": "user"
},
{
"content": null,
"role": "assistant",
"tool_calls": [
{
"type": "function",
"id": "0",
"function": {
"name": "duckduckgo_results_json",
"arguments": "{\"query\": \"how to cook a lobster\"}"
}
}
]
},
{
"content": "[{\"snippet\": \"From that point, boil the lobsters for 7 to 14 minutes or longer, depending on the size of the lobster. 7 to 10 minutes for a 1-pound lobster, 8 to 12 minutes for a 1 1/4-pound lobster, and 10 to 14 minutes for a 1 1/2-pound lobster.\", \"title\": \"How to Boil and Eat Lobster - Simply Recipes\", \"link\": \"https://www.simplyrecipes.com/recipes/how_to_boil_and_eat_lobster/\"}, {\"snippet\": \"Learn how to choose, prepare, and cook lobster in different methods, such as boiling, steaming, and baking. Find out how to break down, serve, and explore the versatility of this culinary treasure.\", \"title\": \"How to Cook Lobster: Foolproof Recipe for Perfect Results Every Time\", \"link\": \"https://oliviaoven.com/how-to-cook-lobster/\"}, {\"snippet\": \"For humane cooking, place the lobster in the freezer for 15-20 minutes to gently put it to sleep. 3. Master Boiling. Boiling is one of the simplest ways to cook a lobster. Bring a large pot of salted water to a rolling boil, then add your lobster headfirst. Cook for about 7-8 minutes per pound. The shell should turn bright red when it's done. 4.\", \"title\": \"Ways to Cook a Lobster: 10 Beginner Tips for Fresh Results\", \"link\": \"https://www.wholesalemainelobster.com/blog/top-10-fresh-lobster-cooking-tips-for-beginners/\"}, {\"snippet\": \"Learn how to cook lobster at home with various methods, such as boiling, steaming, and grilling. Find out how to choose the best type of lobster, what tools to use, and how to serve it for a restaurant-quality experience.\", \"title\": \"How to Cook Lobster: Step-by-Step Guide for Perfect Lobster at Home\", \"link\": \"https://globalseafoods.com/blogs/recipes/how-to-cook-a-lobster-a-step-by-step-guide\"}]",
"role": "tool",
"tool_call_id": "0"
}
]
}
Error message from SGLang side:
[2025-02-22 05:03:38] INFO: 127.0.0.1:60270 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
[2025-02-22 05:03:38] ERROR: Exception in ASGI application
+ Exception Group Traceback (most recent call last):
| File "/usr/local/lib/python3.10/dist-packages/starlette/_utils.py", line 76, in collapse_excgroups
| yield
| File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/base.py", line 178, in __call__
| async with anyio.create_task_group() as task_group:
| File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 767, in __aexit__
| raise BaseExceptionGroup(
| exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
| result = await app( # type: ignore[func-returns-value]
| File "/usr/local/lib/python3.10/dist-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
| return await self.app(scope, receive, send)
| File "/usr/local/lib/python3.10/dist-packages/fastapi/applications.py", line 1054, in __call__
| await super().__call__(scope, receive, send)
| File "/usr/local/lib/python3.10/dist-packages/starlette/applications.py", line 112, in __call__
| await self.middleware_stack(scope, receive, send)
| File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 187, in __call__
| raise exc
| File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 165, in __call__
| await self.app(scope, receive, _send)
| File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/base.py", line 177, in __call__
| with recv_stream, send_stream, collapse_excgroups():
| File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
| self.gen.throw(typ, value, traceback)
| File "/usr/local/lib/python3.10/dist-packages/starlette/_utils.py", line 82, in collapse_excgroups
| raise exc
| File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/base.py", line 179, in __call__
| response = await self.dispatch_func(request, call_next)
| File "/sgl-workspace/sglang/python/sglang/srt/utils.py", line 611, in authentication
| return await call_next(request)
| File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/base.py", line 154, in call_next
| raise app_exc
| File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/base.py", line 141, in coro
| await self.app(scope, receive_or_disconnect, send_no_error)
| File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/cors.py", line 85, in __call__
| await self.app(scope, receive, send)
| File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", line 62, in __call__
| await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
| File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 715, in __call__
| await self.middleware_stack(scope, receive, send)
| File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 735, in app
| await route.handle(scope, receive, send)
| File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 288, in handle
| await self.app(scope, receive, send)
| File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 76, in app
| await wrap_app_handling_exceptions(app, request)(scope, receive, send)
| File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 73, in app
| response = await f(request)
| File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 301, in app
| raw_response = await run_endpoint_function(
| File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 212, in run_endpoint_function
| return await dependant.call(**values)
| File "/sgl-workspace/sglang/python/sglang/srt/entrypoints/http_server.py", line 406, in openai_v1_chat_completions
| return await v1_chat_completions(_global_state.tokenizer_manager, raw_request)
| File "/sgl-workspace/sglang/python/sglang/srt/openai_api/adapter.py", line 1200, in v1_chat_completions
| all_requests = [ChatCompletionRequest(**request_json)]
| File "/usr/local/lib/python3.10/dist-packages/pydantic/main.py", line 214, in __init__
| validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
| pydantic_core._pydantic_core.ValidationError: 5 validation errors for ChatCompletionRequest
| messages.1.ChatCompletionMessageGenericParam.content.str
| Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]
| For further information visit https://errors.pydantic.dev/2.10/v/string_type
| messages.1.ChatCompletionMessageGenericParam.content.list[ChatCompletionMessageContentTextPart]
| Input should be a valid list [type=list_type, input_value=None, input_type=NoneType]
| For further information visit https://errors.pydantic.dev/2.10/v/list_type
| messages.1.ChatCompletionMessageUserParam.role
| Input should be 'user' [type=literal_error, input_value='assistant', input_type=str]
| For further information visit https://errors.pydantic.dev/2.10/v/literal_error
| messages.1.ChatCompletionMessageUserParam.content.str
| Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]
| For further information visit https://errors.pydantic.dev/2.10/v/string_type
| messages.1.ChatCompletionMessageUserParam.content.list[union[ChatCompletionMessageContentTextPart,ChatCompletionMessageContentImagePart]]
| Input should be a valid list [type=list_type, input_value=None, input_type=NoneType]
| For further information visit https://errors.pydantic.dev/2.10/v/list_type
+------------------------------------
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "/usr/local/lib/python3.10/dist-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
return await self.app(scope, receive, send)
File "/usr/local/lib/python3.10/dist-packages/fastapi/applications.py", line 1054, in __call__
await super().__call__(scope, receive, send)
File "/usr/local/lib/python3.10/dist-packages/starlette/applications.py", line 112, in __call__
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 187, in __call__
raise exc
File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 165, in __call__
await self.app(scope, receive, _send)
File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/base.py", line 177, in __call__
with recv_stream, send_stream, collapse_excgroups():
File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.10/dist-packages/starlette/_utils.py", line 82, in collapse_excgroups
raise exc
File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/base.py", line 179, in __call__
response = await self.dispatch_func(request, call_next)
File "/sgl-workspace/sglang/python/sglang/srt/utils.py", line 611, in authentication
return await call_next(request)
File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/base.py", line 154, in call_next
raise app_exc
File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/base.py", line 141, in coro
await self.app(scope, receive_or_disconnect, send_no_error)
File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/cors.py", line 85, in __call__
await self.app(scope, receive, send)
File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", line 62, in __call__
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 715, in __call__
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 735, in app
await route.handle(scope, receive, send)
File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 288, in handle
await self.app(scope, receive, send)
File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 76, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 73, in app
response = await f(request)
File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 301, in app
raw_response = await run_endpoint_function(
File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 212, in run_endpoint_function
return await dependant.call(**values)
File "/sgl-workspace/sglang/python/sglang/srt/entrypoints/http_server.py", line 406, in openai_v1_chat_completions
return await v1_chat_completions(_global_state.tokenizer_manager, raw_request)
File "/sgl-workspace/sglang/python/sglang/srt/openai_api/adapter.py", line 1200, in v1_chat_completions
all_requests = [ChatCompletionRequest(**request_json)]
File "/usr/local/lib/python3.10/dist-packages/pydantic/main.py", line 214, in __init__
validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
pydantic_core._pydantic_core.ValidationError: 5 validation errors for ChatCompletionRequest
messages.1.ChatCompletionMessageGenericParam.content.str
Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]
For further information visit https://errors.pydantic.dev/2.10/v/string_type
messages.1.ChatCompletionMessageGenericParam.content.list[ChatCompletionMessageContentTextPart]
Input should be a valid list [type=list_type, input_value=None, input_type=NoneType]
For further information visit https://errors.pydantic.dev/2.10/v/list_type
messages.1.ChatCompletionMessageUserParam.role
Input should be 'user' [type=literal_error, input_value='assistant', input_type=str]
For further information visit https://errors.pydantic.dev/2.10/v/literal_error
messages.1.ChatCompletionMessageUserParam.content.str
Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]
For further information visit https://errors.pydantic.dev/2.10/v/string_type
messages.1.ChatCompletionMessageUserParam.content.list[union[ChatCompletionMessageContentTextPart,ChatCompletionMessageContentImagePart]]
Input should be a valid list [type=list_type, input_value=None, input_type=NoneType]
For further information visit https://errors.pydantic.dev/2.10/v/list_type
@Superskyyy Currently, the content field of a message can't be null. If there's no message, simply set it as an empty string (like this).
{
"content": "",
"role": "assistant",
"tool_calls": [
{
"type": "function",
"id": "0",
"function": {
"name": "duckduckgo_results_json",
"arguments": "{\"query\": \"how to cook a lobster\"}"
}
}
]
}
Hello @xxll88. Could you please provide the messages in json? It seems that the content field in one of your messages happens to be None (= null in json), which leads to the fatal error:
messages.2.ChatCompletionMessageGenericParam.content.str Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]
@Superskyyy I'm not using tool calling, just running a tester https://github.com/sgl-project/sglang/blob/main/test/srt/test_vision_openai_server.py
So we should PR to make it convert None to an empty string?
I think so, it can avoid unexpected complications.
The current tool calling would have a None content, which causes this error. We are working on fix it. Thanks for reporting it.
Hi, we had a look. I think the problem is not on SGLang side. For the content, if there's a tool call, then original function calling should return content with None like
ChatCompletion(id='chatcmpl-B86VRRshntsmdse0hcF87S6xyRAEh', choices=[Choice(finish_reason='tool_calls', index=0, logprobs=None, message=ChatCompletionMessage(content=None, refusal=None, role='assistant', audio=None, function_call=None, tool_calls=[ChatCompletionMessageToolCall(id='call_fIQOen5Rwi3BdIb3ufl2aoMY', function=Function(arguments='{"city":"Beijing","unit":"celsius"}', name='get_weather'), type='function')]))], created=1741271333, model='gpt-4o-2024-08-06', object='chat.completion', service_tier='default', system_fingerprint='fp_eb9dce56a8', usage=CompletionUsage(completion_tokens=20, prompt_tokens=77, total_tokens=97, completion_tokens_details=CompletionTokensDetails(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=0, cached_tokens=0)))
Then we can get result by manually calling the function and set this result as content to the next turn of conversation. Please check)
@shuaills Hi, it's right to return content with None, but when the result is added to messages and start a new conversation, sglang will report an error.
This is an example from openai docs:
