llama.cpp
llama.cpp copied to clipboard
Published
20 hours ago •
ggml-org
ggml-org
Misc. bug: `json_schema` under `response_format` is not working on OpenAI compatible API endpoint `v1/chat/completions`
Name and Version
build: 4761 (cad53fc9) with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu
Operating systems
Linux
Which llama.cpp modules do you know to be affected?
llama-server
Command line
I'm using docker to run llama-server, the model I used is `DeepSeek-R1-Distill-Qwen-32B-Q4_K_M.gguf`
with these command options:
--jinja
--reasoning-format deepseek
--chat-template-file llama-cpp-deepseek-r1.jinja
Problem description & steps to reproduce
All the test below are for the API endpoint /v1/chat/completions
Using json_schema field directly is working
API request and response details
Request:{
"messages": [
{
"role": "user",
"content": "Tell me some short info about Bill Gates"
}
],
"json_schema": {
"title": "Person",
"type": "object",
"properties": {
"firstName": {
"type": "string",
"description": "The person's first name."
},
"lastName": {
"type": "string",
"description": "The person's last name."
},
"age": {
"description": "Age in years which must be equal to or greater than zero.",
"type": "integer",
"minimum": 0
}
}
}
}
Response:
{
"choices": [
{
"finish_reason": "stop",
"index": 0,
"message": {
"role": "assistant",
"content": "{\n \"firstName\": \"Bill\",\n \"lastName\": \"Gates\",\n \"age\": 67\n}\n \t\t\t\t\t\t\t\t \t\t\t\t\t\t "
}
}
],
"created": 1740091977,
"model": "gpt-3.5-turbo",
"system_fingerprint": "b4761-cad53fc9",
"object": "chat.completion",
"usage": {
"completion_tokens": 32,
"prompt_tokens": 14,
"total_tokens": 46
},
"id": "chatcmpl-CBeEaPjeeBUtTLmv9zVKlmMEuoP0gIdx",
"timings": {
"prompt_n": 14,
"prompt_ms": 364.622,
"prompt_per_token_ms": 26.044428571428572,
"prompt_per_second": 38.39592783759619,
"predicted_n": 32,
"predicted_ms": 1087.966,
"predicted_per_token_ms": 33.9989375,
"predicted_per_second": 29.412683852252737
}
}
Response context could be parsed as json
{
"firstName": "Bill",
"lastName": "Gates",
"age": 67
}
Using json_object field under response_format is working
API request and response details
Request:{
"messages": [
{
"role": "user",
"content": "Tell me some short info about Bill Gates"
}
],
"response_format": {
"type": "json_object",
"schema": {
"title": "Person",
"type": "object",
"properties": {
"firstName": {
"type": "string",
"description": "The person's first name."
},
"lastName": {
"type": "string",
"description": "The person's last name."
},
"age": {
"description": "Age in years which must be equal to or greater than zero.",
"type": "integer",
"minimum": 0
}
}
}
}
}
Response:
{
"choices": [
{
"finish_reason": "stop",
"index": 0,
"message": {
"role": "assistant",
"content": "{\n \"firstName\": \"Bill\",\n \"lastName\": \"Gates\",\n \"age\": 67\n}\n \t\t\t\t\t\t\t \t\t\t\t\t\t\t "
}
}
],
"created": 1740092929,
"model": "gpt-3.5-turbo",
"system_fingerprint": "b4761-cad53fc9",
"object": "chat.completion",
"usage": {
"completion_tokens": 33,
"prompt_tokens": 14,
"total_tokens": 47
},
"id": "chatcmpl-Y9PENcyllMFJQgCxMomh0vPcEy0Z2ruI",
"timings": {
"prompt_n": 13,
"prompt_ms": 338.304,
"prompt_per_token_ms": 26.023384615384614,
"prompt_per_second": 38.42697692016648,
"predicted_n": 33,
"predicted_ms": 1101.544,
"predicted_per_token_ms": 33.38012121212122,
"predicted_per_second": 29.95794993209531
}
}
Response context could be parsed as json
{
"firstName": "Bill",
"lastName": "Gates",
"age": 67
}
Using json_schema field under response_format has no effect, not working
API request and response details
Request:{
"messages": [
{
"role": "user",
"content": "Tell me some short info about Bill Gates"
}
],
"response_format": {
"type": "json_schema",
"schema": {
"title": "Person",
"type": "object",
"properties": {
"firstName": {
"type": "string",
"description": "The person's first name."
},
"lastName": {
"type": "string",
"description": "The person's last name."
},
"age": {
"description": "Age in years which must be equal to or greater than zero.",
"type": "integer",
"minimum": 0
}
}
}
}
}
Response:
{
"choices": [
{
"finish_reason": "stop",
"index": 0,
"message": {
"role": "assistant",
"reasoning_content": "Okay, so I need to figure out some short information about Bill Gates. I know he's a well-known figure, especially in the tech world, but I'm not exactly sure about all the details. Let me start by recalling what I know. \n\nFirst, Bill Gates is the co-founder of Microsoft, right? That's a big company, so he must have had a significant impact on the tech industry. I think he started Microsoft with Paul Allen. They were pretty young when they founded it, maybe in the early 80s? I'm not entirely sure about the exact year, but I remember that Microsoft became a major player in software development, especially with Windows.\n\nI also know that Bill Gates is one of the richest people in the world. He's been on the Forbes list multiple times. But I think he's stepped down from day-to-day operations at Microsoft. Maybe he's still involved in some way, but he's probably focusing more on philanthropy now. Oh yeah, the Bill & Melinda Gates Foundation! That's a big thing he's involved in, focusing on global health, education, and poverty issues. They've done a lot of work in vaccination and improving healthcare in developing countries.\n\nWait, did he write any books? I think he has, maybe \"The Road Ahead\" was one. I'm not too familiar with the content, but I know it's about technology and its impact on society. More recently, he's been talking about climate change. I think he wrote another book about that, perhaps \"How to Avoid a Climate Disaster.\" That makes sense because he's involved in various initiatives aimed at solving global problems.\n\nI should also mention his educational background. I believe he went to Harvard but didn't finish. He dropped out to start Microsoft, which is a common story among tech entrepreneurs. His early interest in computers probably started when he was a teenager, maybe even before college.\n\nIn terms of personal life, I know he's married to Melinda Gates, and they have three children. I think they have a foundation together, which is a huge part of their work. They focus on global issues, so that ties back to the philanthropy aspect.\n\nI'm not entirely sure about the timeline of his career. When exactly did he step back from Microsoft? I think he transitioned out of the CEO role a while ago, maybe in the early 2000s, and now he's more of a chairman or advisor. He's also involved in other ventures, like investment in clean energy through Breakthrough Energy Ventures. That's part of his work on climate change solutions.\n\nI should also consider his influence beyond Microsoft. He's been a significant figure in the tech industry, influencing not just software but also how technology is integrated into daily life. His vision for a computer on every desk was a big deal in the 90s and early 2000s.\n\nI'm a bit fuzzy on the exact dates and some of the specific initiatives he's involved in, but I think the main points are covered: co-founding Microsoft, philanthropy through the Gates Foundation, writing books, and his focus on climate change and education. I should structure this information clearly, making sure each point is concise and accurate.\n\nWait, did he ever run for public office? I don't think so. He's focused more on business and philanthropy. Also, his net worth is estimated to be in the tens of billions, making him one of the wealthiest individuals globally.\n\nI might have missed some details, but I think I've covered the main aspects of Bill Gates' life and career. Let me organize this into a coherent summary, ensuring each key point is addressed without getting too bogged down in minor details.",
"content": "Bill Gates is a prominent figure in the tech industry, best known as the co-founder of Microsoft, a software development giant, alongside Paul Allen. Founded in 1975, Microsoft revolutionized the tech landscape, particularly with its Windows operating system. Gates, who began his journey in computing as a teenager, attended Harvard but dropped out to focus on Microsoft, embodying the entrepreneurial spirit of many tech pioneers.\n\nTransitioning from his role as CEO in the early 2000s, Gates shifted focus to philanthropy through the Bill & Melinda Gates Foundation, addressing global health, education, and poverty. The foundation has significantly contributed to vaccination efforts and healthcare improvements in developing nations.\n\nGates has authored books, including \"The Road Ahead,\" exploring technology's societal impact, and \"How to Avoid a Climate Disaster,\" highlighting his commitment to combating climate change. He is also involved in initiatives like Breakthrough Energy Ventures, investing in clean energy solutions.\n\nBeyond his professional achievements, Gates is married to Melinda Gates, with whom he shares three children. Their philanthropic efforts remain central to their work, emphasizing global issues and technological innovation.\n\nGates' influence extends beyond Microsoft, shaping the integration of technology into daily life, with a vision that once aimed for a computer on every desk. His net worth, estimated in the tens of billions, underscores his status as one of the world's wealthiest individuals."
}
}
],
"created": 1740093066,
"model": "gpt-3.5-turbo",
"system_fingerprint": "b4761-cad53fc9",
"object": "chat.completion",
"usage": {
"completion_tokens": 1044,
"prompt_tokens": 14,
"total_tokens": 1058
},
"id": "chatcmpl-vA5An9fR4RfCkWLcrB73YU07uA5PLtCH",
"timings": {
"prompt_n": 12,
"prompt_ms": 229.987,
"prompt_per_token_ms": 19.165583333333334,
"prompt_per_second": 52.1768621704705,
"predicted_n": 1044,
"predicted_ms": 33514.323,
"predicted_per_token_ms": 32.10184195402299,
"predicted_per_second": 31.150860484336807
}
}
Response doesn't follow json_schema
