CogVLM2 on oobabooga webui generate void tokens

[Grimmins]

System Info / 系統信息

Cuda 12.5 Trasnformers 4.41.2 Python 3.12 Ubuntu 22.04

Who can help? / 谁可以帮助到您？

No response

Information / 问题信息

• [X] The official example scripts / 官方的示例脚本
• [ ] My own modified scripts / 我自己修改的脚本和任务

Reproduction / 复现过程

Steps to reproduce (yes, you have to use the oobabooga webui, but that's not their fault - it may be a mistake on my part, in which case, it's not really an issue - ).

1. Launch oobabooga webui
2. In Model tab, load THUDM_cogvlm2 with Transformers
3. go in chat tab
4. ask a question Q something about an image that you can upload

Here the traceback :

Loading checkpoint shards: 100%|██████████████████| 8/8 [00:06<00:00,  1.33it/s]
10:53:31-241178 INFO     Loaded "THUDM_cogvlm2-llama3-chat-19B" in 6.97 seconds.
10:53:31-241757 INFO     LOADER: "Transformers"                                 
10:53:31-242045 INFO     TRUNCATION LENGTH: 8192                                
10:53:31-242332 INFO     INSTRUCTION TEMPLATE: "Custom (obtained from model     
                         metadata)"                                             
Traceback (most recent call last):
  File "/media/nabab/NVME2/AI/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/queueing.py", line 527, in process_events
    response = await route_utils.call_process_api(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/nabab/NVME2/AI/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/route_utils.py", line 261, in call_process_api
    output = await app.get_blocks().process_api(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/nabab/NVME2/AI/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/blocks.py", line 1786, in process_api
    result = await self.call_function(
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/nabab/NVME2/AI/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/blocks.py", line 1350, in call_function
    prediction = await utils.async_iteration(iterator)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/nabab/NVME2/AI/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/utils.py", line 583, in async_iteration
    return await iterator.__anext__()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/nabab/NVME2/AI/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/utils.py", line 576, in __anext__
    return await anyio.to_thread.run_sync(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/nabab/NVME2/AI/text-generation-webui/installer_files/env/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/nabab/NVME2/AI/text-generation-webui/installer_files/env/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2177, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/media/nabab/NVME2/AI/text-generation-webui/installer_files/env/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 859, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/nabab/NVME2/AI/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/utils.py", line 559, in run_sync_iterator_async
    return next(iterator)
           ^^^^^^^^^^^^^^
  File "/media/nabab/NVME2/AI/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/utils.py", line 742, in gen_wrapper
    response = next(iterator)
               ^^^^^^^^^^^^^^
  File "/media/nabab/NVME2/AI/text-generation-webui/modules/chat.py", line 414, in generate_chat_reply_wrapper
    for i, history in enumerate(generate_chat_reply(text, state, regenerate, _continue, loading_message=True, for_ui=True)):
  File "/media/nabab/NVME2/AI/text-generation-webui/modules/chat.py", line 382, in generate_chat_reply
    for history in chatbot_wrapper(text, state, regenerate=regenerate, _continue=_continue, loading_message=loading_message, for_ui=for_ui):
  File "/media/nabab/NVME2/AI/text-generation-webui/modules/chat.py", line 280, in chatbot_wrapper
    visible_text = html.escape(text)
                   ^^^^^^^^^^^^^^^^^
  File "/media/nabab/NVME2/AI/text-generation-webui/installer_files/env/lib/python3.11/html/__init__.py", line 19, in escape
    s = s.replace("&", "&amp;") # Must be done first!
        ^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'replace'
Traceback (most recent call last):
  File "/media/nabab/NVME2/AI/text-generation-webui/modules/callbacks.py", line 61, in gentask
    ret = self.mfunc(callback=_callback, *args, **self.kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/nabab/NVME2/AI/text-generation-webui/modules/text_generation.py", line 376, in generate_with_callback
    shared.model.generate(**kwargs)
  File "/media/nabab/NVME2/AI/text-generation-webui/installer_files/env/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/media/nabab/NVME2/AI/text-generation-webui/installer_files/env/lib/python3.11/site-packages/transformers/generation/utils.py", line 1622, in generate
    result = self._sample(
             ^^^^^^^^^^^^^
  File "/media/nabab/NVME2/AI/text-generation-webui/installer_files/env/lib/python3.11/site-packages/transformers/generation/utils.py", line 2788, in _sample
    model_inputs = self.prepare_inputs_for_generation(input_ids, **model_kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: CogVLMForCausalLM.prepare_inputs_for_generation() missing 1 required positional argument: 'token_type_ids'
Output generated in 0.34 seconds (0.00 tokens/s, 0 tokens, context 46, seed 1920598604)

We can see the model is loaded and the last line said that the token has been generated. But the token is void (the answer si ""). I'm sure this is caused by the errors that appear

Expected behavior / 期待表现

have an answer to my question Q

[zRzRzRzRzRzRzR]

You cannot directly use the conversation template; you need to process the input according to our demo. You seem to be missing a step in the dtype conversion process. Please carefully review how the input is handled before chatting in our CLI demo.

[Grimmins]

do you have any idea how to fix the problem via the WebUI interface? just change the dtype? that shouldn't be too complicated...