transformers apply_chat_template return_assistant_tokens

System Info

huggingface-hub-0.25.2 tokenizers-0.20.1 transformers-4.45.2

Who can help?

@ArthurZucker @itazap

Information

[ ] The official example scripts
[X] My own modified scripts

Tasks

[ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[X] My own task or dataset (give details below)

Reproduction

from transformers import AutoTokenizer
tk = AutoTokenizer.from_pretrained("Qwen2___5-7B-Instruct", trust_remote_code=True)
new_chat_template = '{%- if tools %}\n    {{- \'<|im_start|>system\\n\' }}\n    {%- if messages[0][\'role\'] == \'system\' %}\n        {{- messages[0][\'content\'] }}\n    {%- else %}\n        {{- \'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.\' }}\n    {%- endif %}\n    {{- "\\n\\n# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <tools></tools> XML tags:\\n<tools>" }}\n    {%- for tool in tools %}\n        {{- "\\n" }}\n        {{- tool | tojson }}\n    {%- endfor %}\n    {{- "\\n</tools>\\n\\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\\n<tool_call>\\n{\\"name\\": <function-name>, \\"arguments\\": <args-json-object>}\\n</tool_call><|im_end|>\\n" }}\n{%- else %}\n    {%- if messages[0][\'role\'] == \'system\' %}\n        {{- \'<|im_start|>system\\n\' + messages[0][\'content\'] + \'<|im_end|>\\n\' }}\n    {%- else %}\n        {{- \'<|im_start|>system\\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\\n\' }}\n    {%- endif %}\n{%- endif %}\n{%- for message in messages %}\n    {%- if (message.role == "user") or (message.role == "system" and not loop.first) or (message.role == "assistant" and not message.tool_calls) %}\n        {{- \'<|im_start|>\' + message.role + \'\\n\' + message.content + \'<|im_end|>\' + \'\\n\' }}\n    {%- elif message.role == "assistant" %}\n        {% generation %} {{- \'<|im_start|>\' + message.role }}\n        {%- if message.content %}\n            {{- \'\\n\' + message.content }}\n        {%- endif %}\n        {%- for tool_call in message.tool_calls %}\n            {%- if tool_call.function is defined %}\n                {%- set tool_call = tool_call.function %}\n            {%- endif %}\n            {{- \'\\n<tool_call>\\n{"name": "\' }}\n            {{- tool_call.name }}\n            {{- \'", "arguments": \' }}\n            {{- tool_call.arguments | tojson }}\n            {{- \'}\\n</tool_call>\' }}\n        {%- endfor %}\n        {{- \'<|im_end|>\\n\' }} {% endgeneration %}\n    {%- elif message.role == "tool" %}\n        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}\n            {{- \'<|im_start|>user\' }}\n        {%- endif %}\n        {{- \'\\n<tool_response>\\n\' }}\n        {{- message.content }}\n        {{- \'\\n</tool_response>\' }}\n        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}\n            {{- \'<|im_end|>\\n\' }}\n        {%- endif %}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- \'<|im_start|>assistant\\n\' }}\n{%- endif %}\n'
msg = [{'role': 'user', 'content': ' \n\n我想起十年前，我在大学里学习生物医学。我很熟悉书本知识，但我发现学术资格并不总是足够的。我需要实际的经验 – 现场工作，才能真正了解行业和工作流程。因此，我决定在学习期间参加志愿工作和实习项目。我参加了医院志愿者活动，这让我有机会与医疗专业人员和病人沟通，也在一定程度上了解了他们的需要。我还在当地的药房实习，这帮助我更深入地了解了医药企业的商业模式和销售策略。最终，我毕业后找到一份医药销售代表的工作，因为我既有学术背景，又有现场工作经验，能够灵活处理各种情况，更好地满足客户需求。这份工作最终让我在职业生涯中获得了巨大成功。\n\n\n基于以上这段文本内容回答： 你参加的志愿者活动和实习项目可以带给你哪些实际的经验和收获？ \n\n'}, {'role': 'assistant', 'content': '参加医院志愿者活动和实习项目让我有机会与医疗专业人员和病人沟通，了解他们的需要。在当地的药房实习让我更深入地了解了医药企业的商业模式和销售策略。这些实际的经验和收获帮助我更好地满足客户需求，并在职业生涯中取得了成功。'}]
output = tk.apply_chat_template(
    msg,
    chat_template=new_chat_template,
    tokenize=True,
    add_generation_prompt=True,
    padding=True,
    max_length=2048,
    truncation=True,
    return_dict=True,
    return_assistant_tokens_mask=True,
)
print(output['assistant_masks'])

I modified the chat template to add the {% generation %} and {% endgeneration %} the new_chat_template looks like

{%- if tools %}
    {{- '<|im_start|>system\n' }}
    {%- if messages[0]['role'] == 'system' %}
        {{- messages[0]['content'] }}
    {%- else %}
        {{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}
    {%- endif %}
    {{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
    {%- for tool in tools %}
        {{- "\n" }}
        {{- tool | tojson }}
    {%- endfor %}
    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
{%- else %}
    {%- if messages[0]['role'] == 'system' %}
        {{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}
    {%- else %}
        {{- '<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n' }}
    {%- endif %}
{%- endif %}
{%- for message in messages %}
    {%- if (message.role == "user") or (message.role == "system" and not loop.first) or (message.role == "assistant" and not message.tool_calls) %}
        {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
    {%- elif message.role == "assistant" %}
        {% generation %} {{- '<|im_start|>' + message.role }}
        {%- if message.content %}
            {{- '\n' + message.content }}
        {%- endif %}
        {%- for tool_call in message.tool_calls %}
            {%- if tool_call.function is defined %}
                {%- set tool_call = tool_call.function %}
            {%- endif %}
            {{- '\n<tool_call>\n{"name": "' }}
            {{- tool_call.name }}
            {{- '", "arguments": ' }}
            {{- tool_call.arguments | tojson }}
            {{- '}\n</tool_call>' }}
        {%- endfor %}
        {{- '<|im_end|>\n' }} {% endgeneration %}
    {%- elif message.role == "tool" %}
        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
            {{- '<|im_start|>user' }}
        {%- endif %}
        {{- '\n<tool_response>\n' }}
        {{- message.content }}
        {{- '\n</tool_response>' }}
        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
            {{- '<|im_end|>\n' }}
        {%- endif %}
    {%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
    {{- '<|im_start|>assistant\n' }}
{%- endif %}

Expected behavior

the output is

{'input_ids': [151644, 8948, 198, 2610, 525, 1207, 16948, 11, 3465, 553, 54364, 14817, 13, 1446, 525, 264, 10950, 17847, 13, 151645, 198, 151644, 872, 198, 4710, 104100, 71618, 117476, 3837, 104786, 99562, 69249, 100134, 100206, 104316, 1773, 106922, 102364, 90286, 21894, 100032, 3837, 105984, 99879, 104380, 104303, 100684, 104014, 103170, 1773, 35946, 85106, 99912, 106187, 1365, 10236, 236, 108, 82224, 99257, 3837, 101901, 100690, 99794, 99717, 33108, 99257, 102054, 1773, 101886, 3837, 35946, 103930, 18493, 100134, 101072, 101061, 101411, 99257, 33108, 102774, 73345, 1773, 35946, 106057, 100634, 104907, 99600, 3837, 43288, 104029, 106211, 57218, 100182, 99878, 99653, 33108, 104693, 104063, 3837, 104477, 106931, 17447, 99794, 34187, 104056, 85106, 1773, 35946, 104241, 109233, 99471, 99218, 102774, 3837, 43288, 100364, 35946, 33126, 100403, 29490, 99794, 34187, 101356, 104385, 108555, 33108, 100352, 104238, 1773, 103941, 3837, 35946, 109981, 101958, 104191, 101356, 100352, 99661, 104066, 3837, 106811, 107203, 104380, 102193, 3837, 105320, 100647, 111930, 3837, 100006, 105128, 54542, 100646, 99559, 3837, 105344, 101929, 116932, 1773, 106039, 99257, 103941, 104029, 18493, 111978, 15946, 105067, 102334, 19108, 1773, 1406, 104210, 70589, 107083, 108704, 43815, 102104, 5122, 220, 56568, 101061, 9370, 104907, 99600, 33108, 102774, 73345, 73670, 99278, 104314, 102224, 99912, 106187, 33108, 104619, 11319, 4710, 151645, 198, 151644, 77091, 198, 101061, 100634, 104907, 99600, 33108, 102774, 73345, 104029, 106211, 57218, 100182, 99878, 99653, 33108, 104693, 104063, 3837, 99794, 104056, 85106, 1773, 18493, 109233, 99471, 99218, 102774, 104029, 33126, 100403, 29490, 99794, 34187, 101356, 104385, 108555, 33108, 100352, 104238, 1773, 100001, 99912, 106187, 33108, 104619, 100364, 35946, 105344, 101929, 116932, 90395, 18493, 111978, 15946, 104847, 19108, 1773, 151645, 198, 151644, 77091, 198], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'assistant_masks': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}

I the assistant_masks is all zero, it is not correct

Oct 15 '24 09:10 DogeWatch

cc @Rocketknight1!

Oct 17 '24 15:10 LysandreJik

@yonigottesman do you have any idea what's going on here? I can take a look if not!

Oct 18 '24 14:10 Rocketknight1

Hey @Rocketknight1 , I tested this out with the default chat_template and think maybe the update to generation_indices was missed here https://github.com/huggingface/transformers/pull/32684 in the function below:

https://github.com/huggingface/transformers/blob/9240137897096698f5292c7dd38d0651c8a33dc8/src/transformers/utils/chat_template_utils.py#L334-L349

LMK what you think!

Oct 24 '24 07:10 itazap

Hi @DogeWatch it looks like you put the generation keyword in the wrong place. in your template there is a condition: {%- if (message.role == "user") or (message.role == "system" and not loop.first) or (message.role == "assistant" and not message.tool_calls) %} which is true as long as you dont use tools so its entering that path and not the path you put the generation token in. try using:

{%- for message in messages %}
    {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
        {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
    {%- elif (message.role == "assistant" and not message.tool_calls) %}
    {% generation %}    {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}    {% endgeneration %}
    {%- elif message.role == "assistant" %}

This works for me. What do you think?

Also, please check the spaces and \n are as expected by qwen. And finally, i think you can remove the <|im_start|>assistant\n string from within the assistant generation block as it is added by the tokenizer at inference (probably doesn't matter )

Oct 24 '24 09:10 yonigottesman

@DogeWatch @Rocketknight1 if this works can we close the issue?

Oct 31 '24 10:10 yonigottesman

Yes, I'm happy to close it, and we can reopen it if that solution doesn't work.

Oct 31 '24 12:10 Rocketknight1

Can you provide a valided chat template for return_assistant_tokens_mask?

I have used the following code, but got an jinja compiling error.

from transformers import AutoTokenizer, PreTrainedTokenizer
tokenizer: PreTrainedTokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
messages = [
    {"role": "user", "content": "你好，世界！"},
    {"role": "assistant", "content": "你好，世界"},
]

qwen25_chat_templace = """{%- if tools %}
    {{- '<|im_start|>system\n' }}
    {%- if messages[0]['role'] == 'system' %}
        {{- messages[0]['content'] }}
    {%- else %}
        {{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}
    {%- endif %}
    {{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
    {%- for tool in tools %}
        {{- "\n" }}
        {{- tool | tojson }}
    {%- endfor %}
    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
{%- else %}
    {%- if messages[0]['role'] == 'system' %}
        {{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}
    {%- else %}
        {{- '<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n' }}
    {%- endif %}
{%- endif %}
{%- for message in messages %}
    {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
        {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
    {%- elif (message.role == "assistant" and not message.tool_calls) %}
    {% generation %}    {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}    {% endgeneration %}
    {%- elif message.role == "assistant" %}
        {{- '<|im_start|>' + message.role }}
        {%- if message.content %}
            {{- '\n' + message.content }}
        {%- endif %}
        {%- for tool_call in message.tool_calls %}
            {%- if tool_call.function is defined %}
                {%- set tool_call = tool_call.function %}
            {%- endif %}
            {{- '\n<tool_call>\n{"name": "' }}
            {{- tool_call.name }}
            {{- '", "arguments": ' }}
            {{- tool_call.arguments | tojson }}
            {{- '}\n</tool_call>' }}
        {%- endfor %}
        {{- '<|im_end|>\n' }}
    {%- elif message.role == "tool" %}
        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
            {{- '<|im_start|>user' }}
        {%- endif %}
        {{- '\n<tool_response>\n' }}
        {{- message.content }}
        {{- '\n</tool_response>' }}
        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
            {{- '<|im_end|>\n' }}
        {%- endif %}
    {%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
    {{- '<|im_start|>assistant\n' }}
{%- endif %}
"""

tokenizer.chat_template = qwen25_chat_templace
tokenizer.apply_chat_template(messages, return_assistant_tokens_mask=True, return_dict=True, return_tensors="pt")

error as following:

---------------------------------------------------------------------------
TemplateSyntaxError                       Traceback (most recent call last)
Cell In[18], line 65
      6 qwen25_chat_templace = """{%- if tools %}
      7     {{- '<|im_start|>system\n' }}
      8     {%- if messages[0]['role'] == 'system' %}
   (...)     61 {%- endif %}
     62 """
     64 tokenizer.chat_template = qwen25_chat_templace
---> 65 tokenizer.apply_chat_template(messages, return_assistant_tokens_mask=True, return_dict=True, return_tensors="pt")

File /data/workspace/coderepo/llm-compose/.venv/lib/python3.11/site-packages/transformers/tokenization_utils_base.py:1637, in PreTrainedTokenizerBase.apply_chat_template(self, conversation, tools, documents, chat_template, add_generation_prompt, continue_final_message, tokenize, padding, truncation, max_length, return_tensors, return_dict, return_assistant_tokens_mask, tokenizer_kwargs, **kwargs)
   1632     logger.warning_once(
   1633         "return_assistant_tokens_mask==True but chat template does not contain `{% generation %}` keyword."
   1634     )
   1636 # Compilation function uses a cache to avoid recompiling the same template
-> 1637 compiled_template = _compile_jinja_template(chat_template)
   1639 if isinstance(conversation, (list, tuple)) and (
   1640     isinstance(conversation[0], (list, tuple)) or hasattr(conversation[0], "messages")
   1641 ):
   1642     conversations = conversation

File /data/workspace/coderepo/llm-compose/.venv/lib/python3.11/site-packages/transformers/utils/chat_template_utils.py:435, in _compile_jinja_template(chat_template)
    433 jinja_env.globals["raise_exception"] = raise_exception
    434 jinja_env.globals["strftime_now"] = strftime_now
--> 435 return jinja_env.from_string(chat_template)

File /data/workspace/coderepo/llm-compose/.venv/lib/python3.11/site-packages/jinja2/environment.py:1111, in Environment.from_string(self, source, globals, template_class)
   1109 gs = self.make_globals(globals)
   1110 cls = template_class or self.template_class
-> 1111 return cls.from_code(self, self.compile(source), gs, None)

File /data/workspace/coderepo/llm-compose/.venv/lib/python3.11/site-packages/jinja2/environment.py:771, in Environment.compile(self, source, name, filename, raw, defer_init)
    769     return self._compile(source, filename)
    770 except TemplateSyntaxError:
--> 771     self.handle_exception(source=source_hint)

File /data/workspace/coderepo/llm-compose/.venv/lib/python3.11/site-packages/jinja2/environment.py:942, in Environment.handle_exception(self, source)
    937 """Exception handling helper.  This is used internally to either raise
    938 rewritten exceptions or return a rendered traceback for the template.
    939 """
    940 from .debug import rewrite_traceback_stack
--> 942 raise rewrite_traceback_stack(source=source)

File <unknown>:27, in template()

TemplateSyntaxError: expected token 'end of print statement', got 'name'

May 06 '25 11:05 TankNee

When I deleted all the template syntax related to tool, the template could be compiled normally.

{%- for message in messages %}
    {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
        {{- '<|im_start|>' + message.role + '\\n' + message.content + '<|im_end|>' + '\\n' }}
    {%- elif (message.role == "assistant") %}
        {% generation %}    {{- '<|im_start|>' + message.role + '\\n' + message.content + '<|im_end|>' + '\\n' }}    {% endgeneration %}
    {%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
    {{- '<|im_start|>assistant\\n' }}
{%- endif %}

May 06 '25 12:05 TankNee

I used the following template and it worked.

    modified_template = """
{%- if tools %}
    {{- '<|im_start|>system\\n' }}
    {%- if messages[0]['role'] == 'system' %}
        {{- messages[0]['content'] }}
    {%- else %}
        {{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}
    {%- endif %}
    {{- "\\n\\n# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <tools></tools> XML tags:\\n<tools>" }}
    {%- for tool in tools %}
        {{- "\\n" }}
        {{- tool | tojson }}
    {%- endfor %}
    {{- "\\n</tools>\\n\\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\\n<tool_call>\\n{\\\"name\\\": <function-name>, \\\"arguments\\\": <args-json-object>}\\n</tool_call><|im_end|>\\n" }}
{%- else %}
    {%- if messages[0]['role'] == 'system' %}
        {{- '<|im_start|>system\\n' + messages[0]['content'] + '<|im_end|>\\n' }}
    {%- else %}
        {{- '<|im_start|>system\\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\\n' }}
    {%- endif %}
{%- endif %}
{%- for message in messages %}
    {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
        {{- '<|im_start|>' + message.role + '\\n' + message.content + '<|im_end|>' + '\\n' }}
    {%- elif (message.role == "assistant" and not message.tool_calls) %}
    {% generation %}    {{- '<|im_start|>' + message.role + '\\n' + message.content + '<|im_end|>' + '\\n' }}    {% endgeneration %}
    {%- elif message.role == "assistant" %}
        {% generation %}{{- '<|im_start|>' + message.role }}
        {%- if message.content %}
            {{- '\\n' + message.content }}
        {%- endif %}
        {%- for tool_call in message.tool_calls %}
            {%- if tool_call.function is defined %}
                {%- set tool_call = tool_call.function %}
            {%- endif %}
            {{- '\\n<tool_call>\\n{\\\"name\\\": \\\"' }}
            {{- tool_call.name }}
            {{- '\\\", \\\"arguments\\\": ' }}
            {{- tool_call.arguments | tojson }}
            {{- '}\\n</tool_call>' }}
        {%- endfor %}
        {{- '<|im_end|>\\n' }}{% endgeneration %}
    {%- elif message.role == "tool" %}
        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
            {{- '<|im_start|>user' }}
        {%- endif %}
        {{- '\\n<tool_response>\\n' }}
        {{- message.content }}
        {{- '\\n</tool_response>' }}
        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
            {{- '<|im_end|>\\n' }}
        {%- endif %}
    {%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
    {% generation %}{{- '<|im_start|>assistant\\n' }}{% endgeneration %}
{%- endif %}
"""
    return modified_template

May 06 '25 12:05 doruktarhan

modified_template = """ {%- if tools %} {{- '<|im_start|>system\n' }} {%- if messages[0]['role'] == 'system' %} {{- messages[0]['content'] }} {%- else %} {{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }} {%- endif %} {{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within XML tags:\n" }} {%- for tool in tools %} {{- "\n" }} {{- tool | tojson }} {%- endfor %} {{- "\n\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": , \"arguments\": }\n</tool_call><|im_end|>\n" }} {%- else %} {%- if messages[0]['role'] == 'system' %} {{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }} {%- else %} {{- '<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n' }} {%- endif %} {%- endif %} {%- for message in messages %} {%- if (message.role == "user") or (message.role == "system" and not loop.first) %} {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }} {%- elif (message.role == "assistant" and not message.tool_calls) %} {% generation %} {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }} {% endgeneration %} {%- elif message.role == "assistant" %} {% generation %}{{- '<|im_start|>' + message.role }} {%- if message.content %} {{- '\n' + message.content }} {%- endif %} {%- for tool_call in message.tool_calls %} {%- if tool_call.function is defined %} {%- set tool_call = tool_call.function %} {%- endif %} {{- '\n<tool_call>\n{\"name\": \"' }} {{- tool_call.name }} {{- '\", \"arguments\": ' }} {{- tool_call.arguments | tojson }} {{- '}\n</tool_call>' }} {%- endfor %} {{- '<|im_end|>\n' }}{% endgeneration %} {%- elif message.role == "tool" %} {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %} {{- '<|im_start|>user' }} {%- endif %} {{- '\n<tool_response>\n' }} {{- message.content }} {{- '\n</tool_response>' }} {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %} {{- '<|im_end|>\n' }} {%- endif %} {%- endif %} {%- endfor %} {%- if add_generation_prompt %} {% generation %}{{- '<|im_start|>assistant\n' }}{% endgeneration %} {%- endif %} """

it works for my code, thanks!

May 06 '25 12:05 TankNee

apply_chat_template return_assistant_tokens_mask not work for Qwen2.5

System Info

Who can help?

Information

Tasks

Reproduction

Expected behavior