LongWriter [rank4]: Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=15099494.4, input

我使用官方提供的脚本和数据集先后运行了python pre_tokenize_glm4.py python sort_and_group.py --group_size 8 --train_file /home/hnjj/diskdata/yuanshi/media/szf/llm/glm_longwrite/LongWriter/train/datasets 得到了attention_masks_pack.json ，inputs_pack.npy等文件运行训练脚本 ./glm4_longwriter.sh 时，遇到与 DeepSpeedZeroConfig 配置相关的 ValidationError。错误是由于 stage3_prefetch_bucket_size 的输入类型无效，期望为整数但接收到浮点数。

训练日志： [2024-08-26 09:58:48,719] [INFO] [comm.py:683:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl [2024-08-26 09:58:49,793] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-08-26 09:58:50,631] [INFO] [config.py:733:init] Config mesh_device None world_size = 8 [2024-08-26 09:58:50,737] [INFO] [config.py:733:init] Config mesh_device None world_size = 8 [2024-08-26 09:58:50,784] [INFO] [config.py:733:init] Config mesh_device None world_size = 8 [2024-08-26 09:58:50,799] [INFO] [config.py:733:init] Config mesh_device None world_size = 8 [2024-08-26 09:58:51,320] [INFO] [comm.py:652:init_distributed] cdb=None [2024-08-26 09:58:52,754] [INFO] [config.py:733:init] Config mesh_device None world_size = 8 [2024-08-26 09:58:52,859] [INFO] [config.py:733:init] Config mesh_device None world_size = 8 [2024-08-26 09:58:53,039] [INFO] [config.py:733:init] Config mesh_device None world_size = 8 [2024-08-26 09:58:53,301] [INFO] [config.py:733:init] Config mesh_device None world_size = 8 [2024-08-26 09:59:10,505] [INFO] [partition_parameters.py:345:exit] finished initializing model - num_params = 283, num_elems = 9.40B Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:11<00:00, 1.15s/it] Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:11<00:00, 1.16s/it] Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:11<00:00, 1.16s/it] Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:11<00:00, 1.16s/it] Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:11<00:00, 1.16s/it] Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:11<00:00, 1.16s/it] Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:11<00:00, 1.16s/it] Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:11<00:00, 1.18s/it] loading data... loading data... loading data... loading data... loading data... loading data... loading data... loading data... finish loading data Using /home/hnjj/.cache/torch_extensions/py310_cu121 as PyTorch extensions root... Emitting ninja build file /home/hnjj/.cache/torch_extensions/py310_cu121/cpu_adam/build.ninja... Building extension module cpu_adam... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) ninja: no work to do. Loading extension module cpu_adam... Time to load cpu_adam op: 3.158402919769287 seconds [rank4]: Traceback (most recent call last): [rank4]: File "/home/hnjj/diskdata/yuanshi/media/szf/llm/glm_longwrite/LongWriter/train/main.py", line 130, in [rank4]: train() [rank4]: File "/home/hnjj/diskdata/yuanshi/media/szf/llm/glm_longwrite/LongWriter/train/main.py", line 126, in train [rank4]: trainer.train(resume_from_checkpoint=False) [rank4]: File "/home/hnjj/anaconda3/envs/szf-longwrite/lib/python3.10/site-packages/transformers/trainer.py", line 1938, in train [rank4]: return inner_training_loop( [rank4]: File "/home/hnjj/anaconda3/envs/szf-longwrite/lib/python3.10/site-packages/transformers/trainer.py", line 2095, in _inner_training_loop [rank4]: model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer) [rank4]: File "/home/hnjj/anaconda3/envs/szf-longwrite/lib/python3.10/site-packages/accelerate/accelerator.py", line 1303, in prepare [rank4]: result = self._prepare_deepspeed(*args) [rank4]: File "/home/hnjj/anaconda3/envs/szf-longwrite/lib/python3.10/site-packages/accelerate/accelerator.py", line 1779, in _prepare_deepspeed [rank4]: engine, optimizer, _, lr_scheduler = deepspeed.initialize(**kwargs) [rank4]: File "/home/hnjj/anaconda3/envs/szf-longwrite/lib/python3.10/site-packages/deepspeed/init.py", line 179, in initialize [rank4]: config_class = DeepSpeedConfig(config, mpu, mesh_device=mesh_device) [rank4]: File "/home/hnjj/anaconda3/envs/szf-longwrite/lib/python3.10/site-packages/deepspeed/runtime/config.py", line 797, in init [rank4]: self._initialize_params(copy.copy(self._param_dict)) [rank4]: File "/home/hnjj/anaconda3/envs/szf-longwrite/lib/python3.10/site-packages/deepspeed/runtime/config.py", line 817, in _initialize_params [rank4]: self.zero_config = get_zero_config(param_dict) [rank4]: File "/home/hnjj/anaconda3/envs/szf-longwrite/lib/python3.10/site-packages/deepspeed/runtime/zero/config.py", line 71, in get_zero_config [rank4]: return DeepSpeedZeroConfig(**zero_config_dict) [rank4]: File "/home/hnjj/anaconda3/envs/szf-longwrite/lib/python3.10/site-packages/deepspeed/runtime/config_utils.py", line 57, in init [rank4]: super().init(**data) [rank4]: File "/home/hnjj/anaconda3/envs/szf-longwrite/lib/python3.10/site-packages/pydantic/main.py", line 193, in init [rank4]: self.pydantic_validator.validate_python(data, self_instance=self) [rank4]: pydantic_core._pydantic_core.ValidationError: 1 validation error for DeepSpeedZeroConfig [rank4]: stage3_prefetch_bucket_size [rank4]: Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=15099494.4, input_type=float] [rank4]: For further information visit https://errors.pydantic.dev/2.8/v/int_from_float

Aug 26 '24 02:08 sunzhufeng12345

而且如果我把stage3.json中的 "stage3_prefetch_bucket_size": "auto",改为 "stage3_prefetch_bucket_size": 15099494,运行会出现如下错误： [2024-08-26 10:00:37,155] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-08-26 10:00:37,222] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-08-26 10:00:37,235] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-08-26 10:00:37,236] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-08-26 10:00:37,301] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-08-26 10:00:37,331] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-08-26 10:00:37,358] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-08-26 10:00:37,386] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-08-26 10:00:38,665] [INFO] [comm.py:652:init_distributed] cdb=None [2024-08-26 10:00:38,716] [INFO] [comm.py:652:init_distributed] cdb=None [2024-08-26 10:00:38,783] [INFO] [comm.py:652:init_distributed] cdb=None [2024-08-26 10:00:38,791] [INFO] [comm.py:652:init_distributed] cdb=None [2024-08-26 10:00:38,810] [INFO] [comm.py:652:init_distributed] cdb=None [2024-08-26 10:00:38,868] [INFO] [comm.py:652:init_distributed] cdb=None [2024-08-26 10:00:38,891] [INFO] [comm.py:652:init_distributed] cdb=None [2024-08-26 10:00:38,891] [INFO] [comm.py:683:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl [2024-08-26 10:00:39,001] [INFO] [comm.py:652:init_distributed] cdb=None [2024-08-26 10:00:40,846] [INFO] [config.py:733:init] Config mesh_device None world_size = 8 [2024-08-26 10:00:40,934] [INFO] [config.py:733:init] Config mesh_device None world_size = 8 [2024-08-26 10:00:41,119] [INFO] [config.py:733:init] Config mesh_device None world_size = 8 [2024-08-26 10:00:41,127] [INFO] [config.py:733:init] Config mesh_device None world_size = 8 [2024-08-26 10:00:41,138] [INFO] [config.py:733:init] Config mesh_device None world_size = 8 [2024-08-26 10:00:41,236] [INFO] [config.py:733:init] Config mesh_device None world_size = 8 [2024-08-26 10:00:41,240] [INFO] [config.py:733:init] Config mesh_device None world_size = 8 [2024-08-26 10:00:41,249] [INFO] [config.py:733:init] Config mesh_device None world_size = 8 [2024-08-26 10:01:00,375] [INFO] [partition_parameters.py:345:exit] finished initializing model - num_params = 283, num_elems = 9.40B Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:11<00:00, 1.16s/it] Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:11<00:00, 1.19s/it] Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:12<00:00, 1.21s/it] Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:11<00:00, 1.19s/it] Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:11<00:00, 1.19s/it] Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:11<00:00, 1.19s/it] Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:11<00:00, 1.19s/it] Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:12<00:00, 1.23s/it] loading data... loading data... loading data... loading data... loading data... loading data... loading data... loading data... finish loading data Using /home/hnjj/.cache/torch_extensions/py310_cu121 as PyTorch extensions root... Emitting ninja build file /home/hnjj/.cache/torch_extensions/py310_cu121/cpu_adam/build.ninja... Building extension module cpu_adam... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) ninja: no work to do. Loading extension module cpu_adam... Time to load cpu_adam op: 2.942378044128418 seconds finish loading data finish loading data finish loading data finish loading data finish loading data finish loading data finish loading data Using /home/hnjj/.cache/torch_extensions/py310_cu121 as PyTorch extensions root... Emitting ninja build file /home/hnjj/.cache/torch_extensions/py310_cu121/cpu_adam/build.ninja... Building extension module cpu_adam... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) ninja: no work to do. Loading extension module cpu_adam... Time to load cpu_adam op: 2.7372360229492188 seconds Using /home/hnjj/.cache/torch_extensions/py310_cu121 as PyTorch extensions root... Using /home/hnjj/.cache/torch_extensions/py310_cu121 as PyTorch extensions root... Emitting ninja build file /home/hnjj/.cache/torch_extensions/py310_cu121/cpu_adam/build.ninja... Building extension module cpu_adam... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) ninja: no work to do. Loading extension module cpu_adam... Time to load cpu_adam op: 2.803518056869507 seconds Loading extension module cpu_adam... Time to load cpu_adam op: 2.814899444580078 seconds Using /home/hnjj/.cache/torch_extensions/py310_cu121 as PyTorch extensions root... Emitting ninja build file /home/hnjj/.cache/torch_extensions/py310_cu121/cpu_adam/build.ninja... Building extension module cpu_adam... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) ninja: no work to do. Loading extension module cpu_adam... Time to load cpu_adam op: 2.760847568511963 seconds Using /home/hnjj/.cache/torch_extensions/py310_cu121 as PyTorch extensions root... Using /home/hnjj/.cache/torch_extensions/py310_cu121 as PyTorch extensions root... Using /home/hnjj/.cache/torch_extensions/py310_cu121 as PyTorch extensions root... Emitting ninja build file /home/hnjj/.cache/torch_extensions/py310_cu121/cpu_adam/build.ninja... Building extension module cpu_adam... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) ninja: no work to do. Loading extension module cpu_adam... Time to load cpu_adam op: 2.765498161315918 seconds Loading extension module cpu_adam... Time to load cpu_adam op: 2.718514919281006 seconds Loading extension module cpu_adam... Time to load cpu_adam op: 2.747807502746582 seconds Parameter Offload: Total persistent parameters: 516096 in 121 params wandb: W&B API key is configured. Use wandb login --relogin to force relogin wandb: Tracking run with wandb version 0.17.7 wandb: Run data is saved locally in /home/hnjj/diskdata/yuanshi/media/szf/llm/glm_longwrite/LongWriter/train/wandb/run-20240826_100248-p37tgoc6 wandb: Run wandb offline to turn off syncing. wandb: Syncing run glm4_longwriter_szf wandb: ⭐️ View project at https://wandb.ai/beijingdaxue/huggingface wandb: 🚀 View run at https://wandb.ai/beijingdaxue/huggingface/runs/p37tgoc6 0%| | 0/2752 [00:00<?, ?it/s][rank7]: Traceback (most recent call last): [rank7]: File "/home/hnjj/diskdata/yuanshi/media/szf/llm/glm_longwrite/LongWriter/train/main.py", line 130, in [rank7]: train() [rank7]: File "/home/hnjj/diskdata/yuanshi/media/szf/llm/glm_longwrite/LongWriter/train/main.py", line 126, in train [rank7]: trainer.train(resume_from_checkpoint=False) [rank7]: File "/home/hnjj/anaconda3/envs/szf-longwrite/lib/python3.10/site-packages/transformers/trainer.py", line 1938, in train [rank7]: return inner_training_loop( [rank7]: File "/home/hnjj/anaconda3/envs/szf-longwrite/lib/python3.10/site-packages/transformers/trainer.py", line 2279, in _inner_training_loop [rank7]: tr_loss_step = self.training_step(model, inputs) [rank7]: File "/home/hnjj/anaconda3/envs/szf-longwrite/lib/python3.10/site-packages/transformers/trainer.py", line 3318, in training_step [rank7]: loss = self.compute_loss(model, inputs) [rank7]: File "/home/hnjj/anaconda3/envs/szf-longwrite/lib/python3.10/site-packages/transformers/trainer.py", line 3363, in compute_loss [rank7]: outputs = model(**inputs) [rank7]: File "/home/hnjj/anaconda3/envs/szf-longwrite/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl [rank7]: return self._call_impl(*args, **kwargs) [rank7]: File "/home/hnjj/anaconda3/envs/szf-longwrite/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl [rank7]: return forward_call(*args, **kwargs) [rank7]: File "/home/hnjj/anaconda3/envs/szf-longwrite/lib/python3.10/site-packages/deepspeed/utils/nvtx.py", line 18, in wrapped_fn [rank7]: ret_val = func(*args, **kwargs) [rank7]: File "/home/hnjj/anaconda3/envs/szf-longwrite/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1899, in forward [rank7]: loss = self.module(*inputs, **kwargs) [rank7]: File "/home/hnjj/anaconda3/envs/szf-longwrite/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl [rank7]: return self._call_impl(*args, **kwargs) [rank7]: File "/home/hnjj/anaconda3/envs/szf-longwrite/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1603, in _call_impl [rank7]: result = forward_call(*args, **kwargs) [rank7]: File "/home/hnjj/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/modeling_chatglm.py", line 994, in forward [rank7]: transformer_outputs = self.transformer( [rank7]: File "/home/hnjj/anaconda3/envs/szf-longwrite/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl [rank7]: return self._call_impl(*args, **kwargs) [rank7]: File "/home/hnjj/anaconda3/envs/szf-longwrite/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1603, in _call_impl [rank7]: result = forward_call(*args, **kwargs) [rank7]: File "/home/hnjj/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/modeling_chatglm.py", line 882, in forward [rank7]: full_attention_mask = self.get_masks(input_ids, past_key_values, padding_mask=attention_mask) [rank7]: File "/home/hnjj/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/modeling_chatglm.py", line 784, in get_masks [rank7]: full_attention_mask = full_attention_mask * padding_mask.unsqueeze(1)

Aug 26 '24 02:08 sunzhufeng12345

我还遇到了这个： ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/lib/python3.11/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn ret_val = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1855, in forward loss = self.module(*inputs, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1568, in _call_impl result = forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/modeling_chatglm.py", line 994, in forward transformer_outputs = self.transformer( ^^^^^^^^^^^^^^^^^ File "/root/miniconda3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1568, in _call_impl result = forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/modeling_chatglm.py", line 882, in forward full_attention_mask = self.get_masks(input_ids, past_key_values, padding_mask=attention_mask) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/modeling_chatglm.py", line 784, in get_masks full_attention_mask = full_attention_mask * padding_mask.unsqueeze(1) ~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~ RuntimeError: The size of tensor a (32768) must match the size of tensor b (6) at non-singleton dimension 1

Aug 27 '24 06:08 badarrrr

我还遇到了这个： ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/lib/python3.11/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn ret_val = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1855, in forward loss = self.module(*inputs, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1568, in _call_impl result = forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/modeling_chatglm.py", line 994, in forward transformer_outputs = self.transformer( ^^^^^^^^^^^^^^^^^ File "/root/miniconda3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1568, in _call_impl result = forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/modeling_chatglm.py", line 882, in forward full_attention_mask = self.get_masks(input_ids, past_key_values, padding_mask=attention_mask) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/modeling_chatglm.py", line 784, in get_masks full_attention_mask = full_attention_mask * padding_mask.unsqueeze(1) ~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~ RuntimeError: The size of tensor a (32768) must match the size of tensor b (6) at non-singleton dimension 1

是的，我现在也是到这一步卡住了，目前和你的报错一样

Aug 27 '24 06:08 sunzhufeng12345

(T_T)

Aug 27 '24 06:08 badarrrr

我们目前提供的GLM-4-9B模型训练代码需要transformers==4.33.0的环境，更高的transformers环境可能导致错误。为了支持packing training，请用patch/下提供的modeling_chatglm.py文件替换原始模型的modeling_chatglm.py.

Aug 28 '24 05:08 bys0318

我们目前提供的GLM-4-9B模型训练代码需要transformers==4.33.0的环境，更高的transformers环境可能导致错误。为了支持packing training，请用patch/下提供的modeling_chatglm.py文件替换原始模型的modeling_chatglm.py.

目前已经换成4.33.0，而且modeling_chatglm.py也已替换，但是出现如下报错： rank7]: Traceback (most recent call last): [rank7]: File "/home/hnjj/diskdata/yuanshi/media/szf/llm/glm_longwrite/LongWriter/train/main.py", line 130, in [rank7]: train() [rank7]: File "/home/hnjj/diskdata/yuanshi/media/szf/llm/glm_longwrite/LongWriter/train/main.py", line 110, in train [rank7]: model = AutoModelForCausalLM.from_pretrained(model_args.model_name_or_path, [rank7]: File "/home/hnjj/anaconda3/envs/szf-longwrite/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 558, in from_pretrained [rank7]: return model_class.from_pretrained( [rank7]: File "/home/hnjj/anaconda3/envs/szf-longwrite/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2954, in from_pretrained [rank7]: model = cls(config, *model_args, **model_kwargs) [rank7]: File "/home/hnjj/anaconda3/envs/szf-longwrite/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 506, in wrapper [rank7]: f(module, *args, **kwargs) [rank7]: File "/home/hnjj/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/modeling_chatglm.py", line 919, in init [rank7]: self.transformer = ChatGLMModel(config, empty_init=empty_init, device=device) [rank7]: File "/home/hnjj/anaconda3/envs/szf-longwrite/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 506, in wrapper [rank7]: f(module, *args, **kwargs) [rank7]: File "/home/hnjj/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/modeling_chatglm.py", line 846, in init [rank7]: self.encoder = init_method(GLMTransformer, config, **init_kwargs) [rank7]: File "/home/hnjj/anaconda3/envs/szf-longwrite/lib/python3.10/site-packages/torch/nn/utils/init.py", line 54, in skip_init [rank7]: return module_cls(*args, **kwargs).to_empty(device=final_device) [rank7]: File "/home/hnjj/anaconda3/envs/szf-longwrite/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 506, in wrapper [rank7]: f(module, *args, **kwargs) [rank7]: File "/home/hnjj/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/modeling_chatglm.py", line 676, in init [rank7]: self.layers = torch.nn.ModuleList([build_layer(i + 1) for i in range(self.num_layers)]) [rank7]: File "/home/hnjj/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/modeling_chatglm.py", line 676, in [rank7]: self.layers = torch.nn.ModuleList([build_layer(i + 1) for i in range(self.num_layers)]) [rank7]: File "/home/hnjj/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/modeling_chatglm.py", line 674, in build_layer [rank7]: return GLMBlock(config, layer_number, device=device) [rank7]: File "/home/hnjj/anaconda3/envs/szf-longwrite/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 506, in wrapper [rank7]: f(module, *args, **kwargs) [rank7]: File "/home/hnjj/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/modeling_chatglm.py", line 607, in init [rank7]: self.self_attention = SelfAttention(config, layer_number, device=device) [rank7]: File "/home/hnjj/anaconda3/envs/szf-longwrite/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 506, in wrapper [rank7]: f(module, *args, **kwargs) [rank7]: File "/home/hnjj/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/modeling_chatglm.py", line 416, in init [rank7]: self.core_attention = CORE_ATTENTION_CLASSES[config._attn_implementation](config, self.layer_number) [rank7]: File "/home/hnjj/anaconda3/envs/szf-longwrite/lib/python3.10/site-packages/transformers/configuration_utils.py", line 261, in getattribute [rank7]: return super().getattribute(key) [rank7]: AttributeError: 'ChatGLMConfig' object has no attribute '_attn_implementation'. Did you mean: 'attn_implementation'?

Aug 28 '24 05:08 sunzhufeng12345

你这里应该是没有成功替换，我们训练时的modeling_chatglm.py代码中没有这一行：File "/home/hnjj/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/modeling_chatglm.py", line 416, in init [rank7]: self.core_attention = CORE_ATTENTION_CLASSES[config._attn_implementation](config, self.layer_number)。这是原始hf库中的代码才有的。

Aug 28 '24 06:08 bys0318

请问训练支持glm-4-9b-chat吗？不是glm-4-9b

Aug 28 '24 06:08 badarrrr

我们建议从glm-4-9b（base）模型开始进行混训（通用SFT数据+LongWriter-6k数据）。直接从glm-4-9b-chat训练的效果会大打折扣。

Aug 28 '24 07:08 bys0318

你这里应该是没有成功替换，我们训练时的modeling_chatglm.py代码中没有这一行：File "/home/hnjj/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/modeling_chatglm.py", line 416, in init [rank7]: self.core_attention = CORE_ATTENTION_CLASSES[config._attn_implementation](config, self.layer_number)。这是原始hf库中的代码才有的。

我试了确实是，替换了原来的文件后，运行train文件，就会使用的还是原来的modeling_chatglm.py文件

Aug 29 '24 02:08 sunzhufeng12345

你这里应该是没有成功替换，我们训练时的modeling_chatglm.py代码中没有这一行：File "/home/hnjj/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/modeling_chatglm.py", line 416, in init [rank7]: self.core_attention = CORE_ATTENTION_CLASSES[config._attn_implementation](config, self.layer_number)。这是原始hf库中的代码才有的。

我试了确实是，替换了原来的文件后，运行train文件，就会使用的还是原来的modeling_chatglm.py文件

你需要在load时候传入参数trust_remote_code=True

Aug 29 '24 03:08 bys0318

Traceback (most recent call last): File "/gemini/code/train/main.py", line 130, in train() File "/gemini/code/train/main.py", line 126, in train trainer.train(resume_from_checkpoint=False) File "/root/miniconda3/lib/python3.11/site-packages/transformers/trainer.py", line 1948, in train return inner_training_loop( ^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/lib/python3.11/site-packages/transformers/trainer.py", line 2289, in _inner_training_loop tr_loss_step = self.training_step(model, inputs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/lib/python3.11/site-packages/transformers/trainer.py", line 3328, in training_step loss = self.compute_loss(model, inputs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/lib/python3.11/site-packages/transformers/trainer.py", line 3373, in compute_loss outputs = model(**inputs) ^^^^^^^^^^^^^^^ File "/root/miniconda3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/lib/python3.11/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn ret_val = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1855, in forward loss = self.module(*inputs, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1568, in _call_impl result = forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/.cache/huggingface/modules/transformers_modules/glm-4-9b/modeling_chatglm.py", line 803, in forward transformer_outputs = self.transformer( ^^^^^^^^^^^^^^^^^ File "/root/miniconda3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1568, in _call_impl result = forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/.cache/huggingface/modules/transformers_modules/glm-4-9b/modeling_chatglm.py", line 707, in forward hidden_states, presents, all_hidden_states, all_self_attentions = self.encoder( ^^^^^^^^^^^^^ File "/root/miniconda3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1568, in _call_impl result = forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/.cache/huggingface/modules/transformers_modules/glm-4-9b/modeling_chatglm.py", line 541, in forward layer_ret = torch.utils.checkpoint.checkpoint( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/lib/python3.11/site-packages/torch/_compile.py", line 24, in inner return torch._dynamo.disable(fn, recursive)(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 328, in _fn return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/lib/python3.11/site-packages/torch/_dynamo/external_utils.py", line 17, in inner return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/lib/python3.11/site-packages/torch/utils/checkpoint.py", line 458, in checkpoint ret = function(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1568, in _call_impl result = forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/.cache/huggingface/modules/transformers_modules/glm-4-9b/modeling_chatglm.py", line 454, in forward attention_output, kv_cache = self.self_attention( ^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1568, in _call_impl result = forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/.cache/huggingface/modules/transformers_modules/glm-4-9b/modeling_chatglm.py", line 317, in forward query_layer = apply_rotary_pos_emb(query_layer, rotary_pos_emb) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: The following operation failed in the TorchScript interpreter. Traceback of TorchScript (most recent call last): File "/root/.cache/huggingface/modules/transformers_modules/glm-4-9b/modeling_chatglm.py", line 145, in apply_rotary_pos_emb rope_cache = rope_cache[:sq] xshaped = x.reshape(sq, -1, np, rot_dim // 2, 2) rope_cache = rope_cache.view(sq, -1, 1, xshaped.size(3), 2) ~~~~~~~~~~~~~~~ <--- HERE x_out2 = torch.stack( [ RuntimeError: shape '[32768, -1, 1, 32, 2]' is invalid for input of size 524288

我换成了glm-4-9b模型，也换了modeling_chatglm.py文件，但是现在报了一个新的错

Aug 29 '24 07:08 badarrrr

@sunzhufeng12345 @badarrrr 请看我们在README中的FAQ是否能解决你们遇到的问题。不好意思让你们久等了。

Sep 03 '24 16:09 bys0318