generation_utils.py NoneType 错误问题

[3rd-Musketeer]

tokenizer: moka-ai/m3e-base

错误：

Traceback (most recent call last):
  File "webui_xbl_stable.py", line 449, in <module>
    model_status = init_model()
  File "webui_xbl_stable.py", line 166, in init_model
    answer_result_stream_result = local_doc_qa.llm_model_chain(
  File "/usr/local/lib/python3.8/dist-packages/langchain/llms/base.py", line 291, in __call__
    self.generate([prompt], stop=stop, callbacks=callbacks)
  File "/usr/local/lib/python3.8/dist-packages/langchain/llms/base.py", line 191, in generate
    raise e
  File "/usr/local/lib/python3.8/dist-packages/langchain/llms/base.py", line 185, in generate
    self._generate(prompts, stop=stop, run_manager=run_manager)
  File "/usr/local/lib/python3.8/dist-packages/langchain/llms/base.py", line 407, in _generate
    else self._call(prompt, stop=stop)
  File "/pkugpt/langchain-ChatGLM/models/baichuan_llm.py", line 50, in _call
    response, _ = self.checkPoint.model.chat(
  File "/root/.cache/huggingface/modules/transformers_modules/baichuan2-7b/modeling_baichuan.py", line 774, in chat
    input_ids = build_chat_input(self, tokenizer, messages, generation_config.max_new_tokens)
  File "/root/.cache/huggingface/modules/transformers_modules/baichuan2-7b/generation_utils.py", line 25, in build_chat_input
    max_input_tokens = model.config.model_max_length - max_new_tokens
TypeError: unsupported operand type(s) for -: `int` and `NoneType`

经检查是因为 generation_utils.py 中，max_new_tokens 有可能为 None

Traceback (most recent call last):
  File "webui_xbl_stable.py", line 449, in <module>
    model_status = init_model()
  File "webui_xbl_stable.py", line 166, in init_model
    answer_result_stream_result = local_doc_qa.llm_model_chain(
  File "/usr/local/lib/python3.8/dist-packages/langchain/llms/base.py", line 291, in __call__
    self.generate([prompt], stop=stop, callbacks=callbacks)
  File "/usr/local/lib/python3.8/dist-packages/langchain/llms/base.py", line 191, in generate
    raise e
  File "/usr/local/lib/python3.8/dist-packages/langchain/llms/base.py", line 185, in generate
    self._generate(prompts, stop=stop, run_manager=run_manager)
  File "/usr/local/lib/python3.8/dist-packages/langchain/llms/base.py", line 407, in _generate
    else self._call(prompt, stop=stop)
  File "/pkugpt/langchain-ChatGLM/models/baichuan_llm.py", line 50, in _call
    response, _ = self.checkPoint.model.chat(
  File "/root/.cache/huggingface/modules/transformers_modules/baichuan2-7b/modeling_baichuan.py", line 774, in chat
    input_ids = build_chat_input(self, tokenizer, messages, generation_config.max_new_tokens)
  File "/root/.cache/huggingface/modules/transformers_modules/baichuan2-7b/generation_utils.py", line 26, in build_chat_input
    assert max_new_tokens is not None
AssertionError

但是 build_chat_input 函数中 max_new_tokens: int=0 有默认值，因此有可能是 modeling_baichuan.py 在调用中传入了 None

[kaixuanjames]

请问这个问题解决了吗？我也遇到了

[BillWSY]

只是为了能够推理的话，我这样解决的

from peft import PeftModel

    model = AutoModelForCausalLM.from_pretrained(
        "baichuan-inc/Baichuan2-7B-Chat",
        trust_remote_code=True
    )

    peft_path = '/path/to/output/checkpoint-max'
    model = PeftModel.from_pretrained(model, peft_path)

    model.half()
    model.cuda()

[skyfaker]

请问这个问题解决了吗？我也遇到了

这个问题是由于加载baichuan2模型的时候没有同时加载baichuan2的generation_config，而transformers初始化的generation_config里面max_new_tokens=None。

解决方法是在加载模型之后指定generation_config，参考： model.generation_config = GenerationConfig.from_pretrained("baichuan-inc/Baichuan2-13B-Base")

[zky001]

我也遇到这个问题了，还没有解决

[zky001]