Support Open Models that allow OpenAI API-style tool use & "auto" tool choice

[K-Mistele]

DRAFT: OpenAI Tool Use Checklist

This (Draft) PR will add support for OpenAI-style tool calling in a way that is minimally opinionated about tool use formats & prompt formatting.

The following features are expected to be supported:

• Custom tool use system prompt template (if desired) -- prevent opinionation about if/how the model uses a system prompt to enable tool use
• Custom tool call return value prompt template (if desired) -- prevent opinionation about the format that tool return values should be passed to the model
• Support for tool_choice="auto" - named tool choice is already supported via guided decoding
• Streaming tool call responses from the chat completions API
• Verified Support & examples for at least the following models:
  • Mistral 7B Instruct v0.3
  • Nous Research's Hermes 2 Pro Models function-calling models including Hermes 2 Pro Llama 3 8B

I'd welcome anyone who wants to contribute on this, and would be happy to add you to the Constellate AI vllm fork that this PR is based off of - please just leave a comment!

Checklist/roadmap:

• [x] validation of tools and tool_choice
• [x] sending tools to the model with tool_choice="auto"
  • [x] CLI Argument: enable auto tool choice
  • [x] CLI argument: tool use system prompt template path - specify the template for the prompt that tells the model how to use the provided tools
  • [x] render template with specified tools
  • [x] prepend the rendered template to the existing system prompt OR use it as the only one if the client didn’t specify one.
  • [x] verify that the model will return a tool call as the chat completion response
• [x] returning tool calls to the client
  • [x] Detect if the model is returning a tool call via the first token
    • [x] CLI argument - specify the token / token ID for tool use responses
  • [x] implement custom extractor class that can be used and implemented for different models to extract their tool call formats to OpenAi's format
  • [x] non-streaming chat completion response: return the JSON as appropriate response.tool_calls
  • [x] streaming chat completion: if the tool use token is called, start streaming the tokens to the client.
• [x] providing tool call results to the model
  • [x] enable specifying a custom chat template
  • [ ] support using huggingface transformers to select the tool_use chat template
• [x] verify support:
  • [x] Nous Research’s Hermes 2 Pro LLama 3 8B
  • [x] Mistral 7B instruct v0.3
• [x] Add documentation

FIX #3237 #4656 (link existing issues this PR will resolve)

BEFORE SUBMITTING, PLEASE READ THE CHECKLIST BELOW AND FILL IN THE DESCRIPTION ABOVE

---

PR Checklist (Click to Expand)

Thank you for your contribution to vLLM! Before submitting the pull request, please ensure the PR meets the following criteria. This helps vLLM maintain the code quality and improve the efficiency of the review process.

PR Title and Classification

Only specific types of PRs will be reviewed. The PR title is prefixed appropriately to indicate the type of change. Please use one of the following:

• [Bugfix] for bug fixes.
• [CI/Build] for build or continuous integration improvements.
• [Doc] for documentation fixes and improvements.
• [Model] for adding a new model or improving an existing model. Model name should appear in the title.
• [Frontend] For changes on the vLLM frontend (e.g., OpenAI API server, LLM class, etc.)
• [Kernel] for changes affecting CUDA kernels or other compute kernels.
• [Core] for changes in the core vLLM logic (e.g., LLMEngine, AsyncLLMEngine, Scheduler, etc.)
• [Hardware][Vendor] for hardware-specific changes. Vendor name should appear in the prefix (e.g., [Hardware][AMD]).
• [Misc] for PRs that do not fit the above categories. Please use this sparingly.

Note: If the PR spans more than one category, please include all relevant prefixes.

Code Quality

The PR need to meet the following code quality standards:

• We adhere to Google Python style guide and Google C++ style guide.
• Pass all linter checks. Please use format.sh to format your code.
• The code need to be well-documented to ensure future contributors can easily understand the code.
• Include sufficient tests to ensure the project to stay correct and robust. This includes both unit tests and integration tests.
• Please add documentation to docs/source/ if the PR modifies the user-facing behaviors of vLLM. It helps vLLM user understand and utilize the new features or changes.

Notes for Large Changes

Please keep the changes as concise as possible. For major architectural changes (>500 LOC excluding kernel/data/config/test), we would expect a GitHub issue (RFC) discussing the technical design and justification. Otherwise, we will tag it with rfc-required and might not go through the PR.

What to Expect for the Reviews

The goal of the vLLM team is to be a transparent reviewing machine. We would like to make the review process transparent and efficient and make sure no contributor feel confused or frustrated. However, the vLLM team is small, so we need to prioritize some PRs over others. Here is what you can expect from the review process:

• After the PR is submitted, the PR will be assigned to a reviewer. Every reviewer will pick up the PRs based on their expertise and availability.
• After the PR is assigned, the reviewer will provide status update every 2-3 days. If the PR is not reviewed within 7 days, please feel free to ping the reviewer or the vLLM team.
• After the review, the reviewer will put an action-required label on the PR if there are changes required. The contributor should address the comments and ping the reviewer to re-review the PR.
• Please respond to all comments within a reasonable time frame. If a comment isn't clear or you disagree with a suggestion, feel free to ask for clarification or discuss the suggestion.

Thank You

Finally, thank you for taking the time to read these guidelines and for your interest in contributing to vLLM. Your contributions make vLLM a great tool for everyone!

[K-Mistele]

Progress! I as of current commits, I can now get the hermes 2 pro model to generate a tool call using the --enable-auto-tool-choice and --tool-use-prompt-template flags:

Server:

python -m vllm.entrypoints.openai.api_server --model NousResearch/Hermes-2-Pro-Llama-3-8B --tool-use-prompt-template examples/tool_template_hermes_2_pro.jinja --enable-api-tools --enable-auto-tool-choice

Client:

python examples/openai_chat_completion_client_with_tools.py

Result

Chat completion results:
ChatCompletion(id='cmpl-1354f3f373574d7aa0e1bf0b78915188', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='<tool_call>{"arguments": {"city": "Dallas", "state": "TX", "unit": "fahrenheit"}, "name": "get_current_weather"}</tool_call>', role='assistant', function_call=None, tool_calls=[]), stop_reason=None)], created=1718763539, model='NousResearch/Hermes-2-Pro-Llama-3-8B', object='chat.completion', system_fingerprint=None, usage=CompletionUsage(completion_tokens=33, prompt_tokens=367, total_tokens=400))

Now, working on getting it to work for non-streaming responses - then, streaming!

[K-Mistele]

A question I asked in the discord, with some open questions about how to handle configuration:

Setting up function calling models for an open model requires a lot of configurations if you want to be unopinionated about the model. Here is a brief list of all the parameters that would be needed:

• enable "auto" tool choice - allow models to choose the function to call (supported for some models) or ONLY tool_choice="" for named tool choice via guided decoding
• tool use prompt template - how do you render the list of tools provided in the request into the prompt / system prompt for the conversation?
• tool use prompt role - what is the role of that message? defaults to system.
• tool use response start token - different models (e.g. Hermes 2 Pro models vs. Mistral 7B Instruct v0.3) use different tokens in their tokenizer to indicate the start of a tool call response vs. a chat response. it's important that this is configured correctly so that we can know whether to send the model's response as a chat response or tool response, and how to stream the response if stream=True. Because there does not seem to be a specific convention for defining this in a uniform way across model tokenizers, it will be necessary for the user to inform the API server which token indicates a tool response is starting, in order to have fully OpenAI API-compatible tool responses.
• tool result response/return value message template the template for returning tool results to the model so it can generated based on results of the tool. - e.g. <tool_response>{"name": "function_name_here", "content": TOOL_RETURN_VALUE_HERE}</tool_response> for Hermes 2 pro models
• tool result response/return value message role - the ROLE to use for that message, e.g. tool for Hermes 2 pro models

The question is, it is better to have all of these as separate CLI flags, or would a JSON configuration file be preferable so that people can create (and track in version control!) configs that work for popular models?)

[K-Mistele]

Please see ongoing conversation with the Hugging Face team, Nous Research & transformers maintainer here - this will make it MUCH easier to implement OpenAI API-compatible tool calling into vLLM regardless of model prompt/tokenizer configs.

HF PR for Hermes 2 Pro: https://huggingface.co/NousResearch/Hermes-2-Pro-Llama-3-8B/discussions/13#66724ea9bd5875ad665f1416

HF PR for Mistral 7B instruct v0.3: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3/discussions/35

Once these are merged, there will be a STANDARD way in transformers to handle templating in tool responses just like for templating chat conversations into prompts with a chat template, and hopefully to pull out tool calls from generated text.

[interstellarninja]

'<tool_call>{"arguments": {"city": "Dallas", "state": "TX", "unit": "fahrenheit"}, "name": "get_current_weather"}</tool_call>'

hey great initiative and nice to see Hermes Pro model's tool calls working.

there's a slight issue with this tool call -- our format requires new lines after <tool_call> XML tags:

<tool_call> {"arguments": {"city": "Dallas", "state": "TX", "unit": "fahrenheit"}, "name": "get_current_weather"} </tool_call>

Also tool choice should also work since it's basically passing the chosen tool only as part of

[K-Mistele]

Thanks! Tool choice is already working via guided decoding, but I will update the PR to fix the template

[K-Mistele]

Ok the most recent commit seems to fix it: python examples/openai_chat_completion_client_with_tools.py

ChatCompletion(id='cmpl-48f019602ab64f4ab49c3563318c6d1f', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='<tool_call>\n{"arguments": {"city": "Dallas", "state": "TX", "unit": "fahrenheit"}, "name": "get_current_weather"}\n</tool_call>', role='assistant', function_call=None, tool_calls=[]), stop_reason=None)], created=1718896666, model='NousResearch/Hermes-2-Pro-Llama-3-8B', object='chat.completion', system_fingerprint=None, usage=CompletionUsage(completion_tokens=34, prompt_tokens=368, total_tokens=402))

[frankabc12341]

@K-Mistele Hi. My vllm version is 0.5.0 post1. But, when I run the command

python -m vllm.entrypoints.openai.api_server --model /home/asus/autodl-tmp/qwen/Qwen2-7B-Instruct --tool-use-prompt-template /home/asus/autodl-tmp/examples/chatml.jinja --enable-api-tools --enable-auto-tool-choice it shows api_server.py: error: unrecognized arguments: --tool-use-prompt-template --enable-api-tools --enable-auto-tool-choice

In addition, I can't find openai_chat_completion_client_with_tools.py in the example fille.

Can you give me some advice?

[K-Mistele]

@K-Mistele Hi. My vllm version is 0.5.0 post1. But, when I run the command

python -m vllm.entrypoints.openai.api_server --model /home/asus/autodl-tmp/qwen/Qwen2-7B-Instruct --tool-use-prompt-template /home/asus/autodl-tmp/examples/chatml.jinja --enable-api-tools --enable-auto-tool-choice it shows api_server.py: error: unrecognized arguments: --tool-use-prompt-template --enable-api-tools --enable-auto-tool-choice

In addition, I can't find openai_chat_completion_client_with_tools.py in the example fille.

Can you give me some advice?

Please see my reply here. This is a draft pull request, which means it has not been merged into vLLM's codebase, and it is not ready to be merged yet. It is still a work-in-progress. As such, none of its additions or features are available in vLLM yet. If you are interested in testing or contributing to this pull request, please see the origin fork at the tool-use branch. However, please be aware that the capabilities discussed in this PR are incomplete and have not been robustly tested.

[aw632]

if the tool use token is called, pull out the tool call JSON from the rest of the response (should be array)

Given that guided decoding is not enabled for "auto" tool use, what is the error handling planned in case the LLM does not output valid JSON?

[K-Mistele]

if the tool use token is called, pull out the tool call JSON from the rest of the response (should be array)

Given that guided decoding is not enabled for "auto" tool use, what is the error handling planned in case the LLM does not output valid JSON?

Great question! Unfortunately since each model that supports tool calling uses its' own format for function calls (as opposed to tool choice with guided decoding, where we're forcing the LLM to call a specific tool in a specific format at decode-time) the response format is up to the model and its' trainer.

At this point, we are still exploring ways to handle extraction of tool calls from disparate formats to OpenAI-compatible calls in a way that isn't opinionated (we want to support multiple formats including Mistral, Hermes 2 Pro, Firefunction, etc). Until we have a good answer on this, we probably won't have a good answer for how to handle errors. There are a couple possible options once we have attempted to extract the model's tool call format into OpenAI's:

• return the call to the client without validation, and allow the client to validate the tool call with libraries like Instructor, pydantic or Zod
• return an HTTP error if the model generates an invalid tool call
• automatically attempt to re-run the generation - perhaps only if the user specifies a specific CLI flag, since this could lead to performance issues or infinite retry loops depending on the model and other configurations like temperature, top_p and top_k

In the interim, we may try to solve this problem by "ignoring" it - detect if the model is generating a tool call, but avoid the destructuring/extraction issue by returning the tool call in the model's format to the client. Not exactly sure what this would look like, but it's a possibility.

[K-Mistele]

Copy/pasting from discord:

I'd really like to make progress on tool use and right now with hugging face adding support into transformers for passing tools into the chat template, that solves one of the main issues

I think the blocker now is figuring out how to handle decodiong "auto" choice tool calls from each model's specific format (mistral vs. Hermes pro vs. firefunction) and returning that to the client ESPECIALLY when streaming is requested.

Until there's a "canonical" way to do decode model-specific tool-calls into the OpenAI format e.g. through transformers or "reverse templates" or something, it might be best to approach this like chat templates which is to try & support it as good as we can where possible

Here's what I mean by this:

• mistral's format is very close to openAI's format and is probably close in terms of implementation. If a tool is being called, the first token from the response indicates this, and then a JSOn array is generated. I can use a partial JSON parser to stream tool arguments etc as they are generated to the client, basically without any translation
• Hermes 2 Pro's format is different and uses multiple XML tags <tool_call></tool_call> for each call with the JSON inside. I should also be able to handle some streaming here with XML extraction and regex.
• Firefunction v2 uses a totally different format as well.

So each one needs a very different implementation. I propose creating a ToolCallParser abstract class that can be implemented for different models like mistral and hermes. If a user is using a tool-calling model, they can use a CLI flag to toggle which parser they want, if they want one at all. If not, a tool call would be treated like a regular chat completion, and they can handle it client-side.

This way, we can ship support for tool calls incrementally in the absence of a commonly-accepted "best practice" on how to do this. People can also add support for other models that are important to them in a minimally-invasive way. Then, once a better way to implement this more broadly is available, we can deprecate this approach

I'd really appreciate feedback on this before moving forward and would especially love to hear if @mgoin and @simon-mo would consider this an appropriate approach that would be likely to be approved.

[simon-mo]

@br3no would love to get your feedback and review for this PR!

[interstellarninja]

@K-Mistele btw I just realized that there were talks to implement Hermes format with llama-cpp-python using a chatml-function-calling template, we did chime in and it was still a WIP

but today i just tried Hermes-Theta-8B using the same template and it just seems to work. there's still slight differences in the tool-call format which i think could be fixed

https://github.com/abetlen/llama-cpp-python/blob/04959f1884c8ef93bd5a4aa40ff0accb8438c0c1/llama_cpp/llama_chat_format.py#L3165

Here's the function calling template they are using:

function_calling_template = (
        "{% for message in messages %}"
        "<|im_start|>{{ message.role }}\n"
        # System message
        "{% if message.role == 'system' %}"
        "{{ message.content }}"
        "{% if tool_calls %}"
        "\n\nYou have access to the following functions:\n"
        "{% for tool in tools %}"
        "\nfunctions.{{ tool.function.name }}:\n"
        "{{ tool.function.parameters | tojson }}"
        "\n{% endfor %}"
        "\n\nYou can respond to users messages with either a single message or one or more function calls."
        "\n\nTo respond with a message begin the message with 'message:', use the following format:"
        "\n\nmessage:"
        "\n<message>"
        "\n\nTo respond with one or more function calls begin the message with 'functions.<function_name>:', use the following format:"
        "\n\nfunctions.<function_name>:"
        '\n{ "arg1": "value1", "arg2": "value2" }'
        "\nfunctions.<function_name>:"
        '\n{ "arg1": "value1", "arg2": "value2" }'
        "{% endif %}"
        "<|im_end|>\n"
        "{% endif %}"
        # User message
        "{% if message.role == 'user' %}"
        "{{ message.content }}"
        "<|im_end|>\n"
        "{% endif %}"
        # Assistant message
        "{% if message.role == 'assistant' %}"
        ## Reglar message
        "{% if message.content and message.content | length > 0 %}"
        "{% if tool_calls %}"
        "message:\n"
        "{% endif %}"
        "{{ message.content }}"
        "<|im_end|>\n"
        "{% endif %}"
        ## Function calls
        "{% if 'tool_calls' in message %}"
        "{% for tool_call in message.tool_calls %}"
        "functions.{{ tool_call.function.name }}:\n"
        "{{ tool_call.function.arguments }}"
        "{% endfor %}"
        "<|im_end|>\n"
        "{% endif %}"
        "{% endif %}"
        "{% endfor %}"
        "{% if add_generation_prompt %}<|im_start|>assistant\n{% endif %}"
    )

Please check the example implementation of Hermes function calling with llama.cpp here: https://github.com/NousResearch/Hermes-Function-Calling/blob/main/examples/lllama-cpp-multiple-fn.ipynb

[br3no]

I’ll try to have a look in the coming days!

[aw632]

I also wanted to ask a question about how tool_call_ids are going to be handled. I know HF says:

Note that we generate a random tool_call_id here. These IDs are not used by all models, but they allow models to issue multiple tool calls at once and keep track of which response corresponds to which call. You can generate them any way you like, but they should be unique within each chat.

Source, Emphasis added

Do we know how much this affects model accuracy? For instance, Mistral-7b-v0.3 does not use the uuid format we currently have in vLLM.

[interstellarninja]

@aw632 thats a great question for Hermes V2 we plan to pass back a dict for tool response as follows:

<tool_response> {"tool_call_id": , "name": , "content": } </tool_response>

IDs could be helpful specifically when multiple tool calls with the same function name are called.

[aw632]

@aw632 thats a great question for Hermes V2 we plan to pass back a dict for tool response as follows:

<tool_response> {"tool_call_id": , "name": , "content": } </tool_response>

IDs could be helpful specifically when multiple tool calls with the same function name are called.

I see, in your experience, if Hermes is trained on one type of id format (e.g., uuid4) but we use a different type in vllm (e.g., uuid1 or some other random string format), would that hurt the model performance a lot? Or is the difference negligible?

[K-Mistele]

@aw632 thats a great question for Hermes V2 we plan to pass back a dict for tool response as follows: <tool_response> {"tool_call_id": , "name": , "content": } </tool_response> IDs could be helpful specifically when multiple tool calls with the same function name are called.

I see, in your experience, if Hermes is trained on one type of id format (e.g., uuid4) but we use a different type in vllm (e.g., uuid1 or some other random string format), would that hurt the model performance a lot? Or is the difference negligible?

impossible to know without testing/benchmarking, but my gut says it wouldn't make a significant difference. The model is smart enough to understand the concept of an ID and matching it to another one somewhere else whether it's a UUID, a nanoid, or a number; so it theoretically shouldn't impact results.

[interstellarninja]

Yep I agree -- we are also just training on OpenAI generated tool call ids

[K-Mistele]

Still waiting on the Nous Research and Mistral teams to merge in the chat template updates that include tool use into their tokenizer_config.json files but I will try with custom chat templates in the meantime. @interstellarninja any updates on this from the Nous team?

[K-Mistele]

Make sure to run pip install -e . to install the updated transformers dependency!

export HF_TOKEN=...
python -m vllm.entrypoints.openai.api_server --model mistralai/Mistral-7B-Instruct-v0.3 --enable-api-tools --enable-auto-tool-choice --chat-template examples/tool_chat_template_mistral.jinja --dtype half --kv-cache-dtype fp8

Tool calls are now returned in the model-specific format for parsing in the chat completion response; they are not treated separately from chat completions.

Tested with mistralai/Mistral-7B-Instruct-v0.2 and got mixed results - sometimes tool calls are generated properly, other times not.

[K-Mistele]

Works much more consistently with Nous Research's Hermes 2 Pro Llama 3 8B:

python -m vllm.entrypoints.openai.api_server --model NousResearch/Hermes-2-Pro-Llama-3-8B --enable-api-tools --enable-auto-tool-choice --chat-template examples/tool_chat_template_hermes_2_pro.jinja --dtype half --kv-cache-dtype fp8

<tool_call>\n{\"arguments\": {\"city\": \"Dallas\", \"state\": \"TX\", \"unit\": \"fahrenheit\"}, \"name\": \"get_current_weather\"}\n</tool_call>

The only thing is that sometimes it leaves off the closing </tool_call> tag but this should be easy to fix in the parser, parse from <tool_call> to </tool_call> OR end of string.

Shoutout @interstellarninja

[K-Mistele]

Make sure to run pip install -e . to install the updated transformers dependency!

export HF_TOKEN=...
python -m vllm.entrypoints.openai.api_server --model mistralai/Mistral-7B-Instruct-v0.3 --enable-api-tools --enable-auto-tool-choice --chat-template examples/tool_chat_template_mistral.jinja --dtype half --kv-cache-dtype fp8

Tool calls are now returned in the model-specific format for parsing in the chat completion response; they are not treated separately from chat completions.

Tested with mistralai/Mistral-7B-Instruct-v0.2 and got mixed results - sometimes tool calls are generated properly, other times not.

Update: it is apparently a known issue that there is not a good jinja chat template for function calling for Mistral 7B Instruct v0.3 - they want everyone to use their templating/tokenization instead of transformers.

There are two open PRs to add a chat_template to the model but the mistral team indicated based on their evals that they are not sufficiently reliable.

So, at this point it may not be easy to support tools for Mistral 7B instruct v0.3 until someone comes up with a better template

• https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3/discussions/38 (discusses tools; conversation w/ mistral team about how jinja chat templates for tools are imperfect)
• https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3/discussions/35 n- huggingface/transformers team's PR into instruct v0.3 that has also not been accepted by mistral. (This is the template I tried using)

[K-Mistele]

using the Mixtral tool_use chat template with Mistral 7B instruct v0.3 gives MUCH more reliable tool call results:

https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1/blob/main/tokenizer_config.json

[interstellarninja]

• https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3/discussions/35

hmm are you referring to the PR by huggingface regarding the tool-use chat_template?

[K-Mistele]

• https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3/discussions/35

hmm are you referring to the PR by huggingface regarding the tool-use chat_template?

Yes :) HF's PR on the hermes repo works for Hermes, but their PR on mistral 7b instruct v0.3 doesn't work for that model. But, I got it to work reliably by borrowing the tool_use template from mixtral

[K-Mistele]

Function-calling now generates an OpenAI-compatible tool response for mistral - hermes not implemented yet

Start vLLM with mistral 7B instruct v0.3, passing in --enable-auto-tool-choice and --tool-call-parser mistral

python -m vllm.entrypoints.openai.api_server --model mistralai/Mistral-7B-Instruct-v0.3 --enable-api-tools --enable-auto-tool-choice --chat-template examples/tool_chat_template_mistral.jinja --dtype half --kv-cache-dtype fp8 --tool-call-parser mistral

Invoke the example:

python examples/openai_chat_completion_client_with_tools.py

Observe:

ChatCompletion(id='cmpl-b3dce4b440714b23a6c7afaaffd996b7', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content=None, role='assistant', function_call=None, tool_calls=[ChatCompletionMessageToolCall(id='chatcmpl-tool-0447941ef53e47759e47ddc68633d080', function=Function(arguments={'city': 'Dallas', 'state': 'TX', 'unit': 'fahrenheit'}, name='get_current_weather'), type='function')]), stop_reason=None)], created=1719895291, model='mistralai/Mistral-7B-Instruct-v0.3', object='chat.completion', system_fingerprint=None, usage=CompletionUsage(completion_tokens=40, prompt_tokens=200, total_tokens=240))

[aw632]

Btw, it looks like the stop reason/finish reason isn't accurately displaying "tool_calls" as the OpenAI API expects (this happens with named tool use too). I can open an issue for this, but I looked into this and an elegant/canonical fix would require us to add bot_token and eot_token in HF's PreTrainedTokenizerBase. A quick workaround would be to set it in serving_chat.py and override output.finish_reason.

[K-Mistele]

Btw, it looks like the stop reason/finish reason isn't accurately displaying "tool_calls" as the OpenAI API expects (this happens with named tool use too). I can open an issue for this, but I looked into this and an elegant/canonical fix would require us to add bot_token and eot_token in HF's PreTrainedTokenizerBase. A quick workaround would be to set it in serving_chat.py and override output.finish_reason.

Good catch, thanks for this! Looking into fixing this now.

[K-Mistele]

Fixed!