[Question]: ChatGLM无法调用问题

[henryhust]

请提出你的问题

环境： windows x64 cpu环境 python3.7 paddlepaddle==2.5.0 paddlenlp==2.6.0rc0

import paddle
from paddlenlp.transformers import (
    ChatGLMConfig,
    ChatGLMForConditionalGeneration,
    ChatGLMTokenizer,
)

#读取原始的chatglm-6b模型
model_name_or_path = r'E:\NLP\chatglm-6b-int4' # 使用该路径会自动下载和加载模型
# model_name_or_path = 'data/data217141' # 本地路径，无需下载，运行更快
tokenizer = ChatGLMTokenizer.from_pretrained(model_name_or_path)

config = ChatGLMConfig.from_pretrained(model_name_or_path)
paddle.set_default_dtype(config.dtype)

model = ChatGLMForConditionalGeneration.from_pretrained(
    model_name_or_path,
    tensor_parallel_degree=paddle.distributed.get_world_size(),
    tensor_parallel_rank=0,
    load_state_as_np=True,
    dtype=config.dtype,
)

model.eval()

报错： warnings.warn("Setuptools is replacing distutils.") [2023-07-12 23:06:06,594] [ INFO] - loading configuration file E:\NLP\chatglm-6b-int4\config.json [2023-07-12 23:06:06,641] [ INFO] - Model config ChatGLMConfig { "_name_or_path": "THUDM/chatglm-6b-int4", "activation": "gelu", "architectures": [ "ChatGLMModel" ], "attention_scale": true, "auto_map": { "AutoConfig": "configuration_chatglm.ChatGLMConfig", "AutoModel": "modeling_chatglm.ChatGLMForConditionalGeneration", "AutoModelForSeq2SeqLM": "modeling_chatglm.ChatGLMForConditionalGeneration" }, "bos_token_id": 130004, "dtype": "float16", "eos_token_id": 130005, "gmask_token_id": 130001, "hidden_size": 4096, "inner_hidden_size": 16384, "layernorm_epsilon": 1e-05, "mask_token_id": 130000, "max_sequence_length": 2048, "model_type": "chatglm", "num_attention_heads": 32, "num_hidden_layers": 28, "num_image_tokens": 0, "output_predict": true, "pad_token_id": 3, "paddlenlp_version": null, "position_encoding_2d": true, "pre_seq_len": null, "prefix_projection": false, "quantization_bit": 4, "quantization_embeddings": false, "recompute": false, "transformers_version": "4.27.1", "use_cache": true, "vocab_size": 130528 }

[2023-07-12 23:06:06,641] [ INFO] - loading configuration file E:\NLP\chatglm-6b-int4\config.json [2023-07-12 23:06:06,641] [ INFO] - Model config ChatGLMConfig { "_name_or_path": "THUDM/chatglm-6b-int4", "activation": "gelu", "architectures": [ "ChatGLMModel" ], "attention_scale": true, "auto_map": { "AutoConfig": "configuration_chatglm.ChatGLMConfig", "AutoModel": "modeling_chatglm.ChatGLMForConditionalGeneration", "AutoModelForSeq2SeqLM": "modeling_chatglm.ChatGLMForConditionalGeneration" }, "bos_token_id": 130004, "eos_token_id": 130005, "gmask_token_id": 130001, "hidden_size": 4096, "inner_hidden_size": 16384, "layernorm_epsilon": 1e-05, "mask_token_id": 130000, "max_sequence_length": 2048, "model_type": "chatglm", "num_attention_heads": 32, "num_hidden_layers": 28, "num_image_tokens": 0, "output_predict": true, "pad_token_id": 3, "paddlenlp_version": null, "position_encoding_2d": true, "pre_seq_len": null, "prefix_projection": false, "quantization_bit": 4, "quantization_embeddings": false, "recompute": false, "tensor_parallel_degree": 1, "transformers_version": "4.27.1", "use_cache": true, "vocab_size": 130528 }

[2023-07-12 23:06:06,641] [ INFO] - start to convert pytorch weight file<E:\NLP\chatglm-6b-int4\pytorch_model.bin> to paddle weight file<E:\NLP\chatglm-6b-int4\model_state.pdparams> ... Traceback (most recent call last): File "C:/Users/Henry/PycharmProjects/PaddleNlpProject/main.py", line 21, in dtype=config.dtype, File "D:\Anaconda\envs\paddlenlp\lib\site-packages\paddlenlp\transformers\model_utils.py", line 1287, in from_pretrained model_state_dict = cls.convert(model_weight_file, config, cache_dir) File "D:\Anaconda\envs\paddlenlp\lib\site-packages\paddlenlp\transformers\conversion_utils.py", line 889, in convert name_mappings = cls._get_name_mappings(config) File "D:\Anaconda\envs\paddlenlp\lib\site-packages\paddlenlp\transformers\conversion_utils.py", line 926, in _get_name_mappings raise NotImplementedError NotImplementedError

没有技巧，全是感情。 简单的代码，不太理解为何无法加载

[w5688414]

看日志似乎您在转pytorch模型为paddle,请使用最新的方式加载chatglm推理。

https://github.com/PaddlePaddle/PaddleNLP/tree/develop/llm#4-%E6%8E%A8%E7%90%86