ChatGLM-Finetuning

ChatGLM-Finetuning copied to clipboard

作者您好， 最近在跑流水线并行的代码，并尝试在qwen上复现，但发现一个问题，pipemodel需要基于basemodel加载，以7B模型为例，相当于内存中最多的时候需要加载2 * 14g，是这样吗？ 感觉想在8 * 32G v100上跑qwen 32B的希望破灭了....

ShawnChang-ei

4090显卡有办法跑这个项目吗？

3


2279072142

Traceback (most recent call last): File "/home/fangzhijun2/ChatGLM-Finetuning-master/train.py", line 234, in main() File "/home/fangzhijun2/ChatGLM-Finetuning-master/train.py", line 79, in main deepspeed.init_distributed() File "/home/fangzhijun2/anaconda3/envs/torch/lib/python3.10/site-packages/deepspeed/comm/comm.py", line 670, in init_distributed cdb = TorchBackend(dist_backend, timeout, init_method, rank,...

ysz2000

求助一下大佬，我想做一个特定场景的微调，就是想做一个景区的导游。我该用什么思路去微调呢，就是数据集该如何制作

Franklin-L

Exception: Current loss scale already at minimum - cannot decrease scale anymore. Exiting run.

5

作者您好，微调的时候一直遇到这个报错是什么原因 Exception: Current loss scale already at minimum - cannot decrease scale anymore. Exiting run.

yyy900

在RTX 4090 上微调chatglm3报这个错：Current loss scale already at minimum - cannot decrease scale anymore

1

ss_scaler.py:183:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 4, reducing to 2 [2024-01-17 10:10:24,477] [INFO] [logging.py:96:log_dist] [Rank 0] step=16, skipped=16, lr=[0.0001], mom=[(0.9, 0.95)] [2024-01-17 10:10:24,478] [INFO] [timer.py:260:stop] epoch=0/micro_step=64/global_step=16,...

450586509

ChatGLM3四卡训练出错了

1

[2024-02-04 17:56:47,007] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False Using /root/.cache/torch_extensions/py311_cu116 as PyTorch extensions root... Using /root/.cache/torch_extensions/py311_cu116 as PyTorch extensions root... Using /root/.cache/torch_extensions/py311_cu116 as PyTorch extensions root... Using...

eanfs

只有训练没有验证

1

能补充验证的代码吗

NanZhang1991

RuntimeError: Error building extension 'fused_adam'

2

deepspeed==0.12.6 pytorch==1.13.0+cu116 cuda版本11.0 用的模型是glm3，单卡的   请问这个问题怎么处理呀，能帮忙解决下吧，谢谢了

LeFuGang

File "/root/anaconda3/envs/llm/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 847, in __init__ setattr(self, key, value) AttributeError: can't set attribute 'eos_token'

2

你好，想问问这个是什么原因？

zzdxjtu