chen333

Results 24 comments of


                                            chen333

怎么去除“Building prefix dict from the default dictionary ...”的提示

> 您好，每一次我用jieba进行分词的时候，都会有 > > ``` > Building prefix dict from the default dictionary ... > Loading model from cache /tmp/jieba.cache > Loading model cost 0.128 seconds. > Prefix dict has...

在运行微调脚本报错 torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0

> @MrRexy-Ling ，就是把微调训练的指令放到 shell脚本里，用bash来启动 run.py, 因为需要开启多个进程，所以一般用 .sh文件来执行模型的训练启动，截图上面就是我的 run.sh的内容，我将运行的指令放到shell脚本运行，依然报这种错误，怎么解决呢？

哪位大佬帮忙看看，微调出现维度不一致问题； RuntimeError: The size of tensor a (25165824) must match the size of tensor b (12288) at non-singleton dimension 0

> 这个应该是deepspeed配置的问题，有一个类似的issue：#43 > > 查了一下可能的解决方案： > > * `apt-get update; apt-get install ninja-build` > * 把cuda版本从10.1升级到10.2（[https://github.com/microsoft/DeepSpeed/issues/694）](https://github.com/microsoft/DeepSpeed/issues/694%EF%BC%89) 我的CUDA版本是12.0 也是这个问题

哪位大佬帮忙看看，微调出现维度不一致问题； RuntimeError: The size of tensor a (25165824) must match the size of tensor b (12288) at non-singleton dimension 0

> > /opt/conda/lib/python3.10/site-packages/torch/include/ATen/cuda/CUDAContext.h:10:10: fatal error: cusolverDn.h: No such file or directory > > #include > > 问题解决了，可以训练啦！！主要是cusolverDn.h: No such file or directory 找不到导致；添加环境变量，export PATH=/usr/local/cuda/bin:$PATH 在哪添加呢

哪位大佬帮忙看看，微调出现维度不一致问题； RuntimeError: The size of tensor a (25165824) must match the size of tensor b (12288) at non-singleton dimension 0

> pip uninstall deepspeed DS_BUILD_FUSED_ADAM=1 pip install deepspeed 以上不行的话再试试 git clone https://github.com/microsoft/DeepSpeed.git cd DeepSpeed DS_BUILD_FUSED_ADAM=1 pip3 install . 还是不行的话，提出你的错误 pip uninstall deepspeed DS_BUILD_FUSED_ADAM=1 pip install deepspeed 进行了上述操作依然出现这个报错 File "/home/nbicc/data/anaconda3/envs/visualglm/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line...

哪位大佬帮忙看看，微调出现维度不一致问题； RuntimeError: The size of tensor a (25165824) must match the size of tensor b (12288) at non-singleton dimension 0

> > tokenizer的问题可以参考这里：[#111 (comment)](https://github.com/THUDM/VisualGLM-6B/issues/111#issuecomment-1579019781) > > tokenzier重新运行是正常； ![image](https://private-user-images.githubusercontent.com/37685989/245088950-8e975551-9e79-46c6-93ca-2651bcc7d022.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTA0NjYzOTgsIm5iZiI6MTcxMDQ2NjA5OCwicGF0aCI6Ii8zNzY4NTk4OS8yNDUwODg5NTAtOGU5NzU1NTEtOWU3OS00NmM2LTkzY2EtMjY1MWJjYzdkMDIyLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAzMTUlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMzE1VDAxMjgxOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTBkZmQxNTUyMmIxZjIxZTNiNzVkNGEwZDFhMzQyYzhlNzQwYWQ2ZmVlODU1NjdkN2JjMWJkNzRkODg0NTZjYjImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.-7gSUxj7-qFHxCfdpQCmYW2_owmfrgm6maGO73akFWY) > > 主要是后面的问题： RuntimeError: Error building extension 'fused_adam'，详情见上面；问题已全部解决，微调成功

哪位大佬帮忙看看，微调出现维度不一致问题； RuntimeError: The size of tensor a (25165824) must match the size of tensor b (12288) at non-singleton dimension 0

> > > tokenizer的问题可以参考这里：[#111 (comment)](https://github.com/THUDM/VisualGLM-6B/issues/111#issuecomment-1579019781) > > > > > > tokenzier重新运行是正常； ![image](https://private-user-images.githubusercontent.com/37685989/245088950-8e975551-9e79-46c6-93ca-2651bcc7d022.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTA0NjYzOTgsIm5iZiI6MTcxMDQ2NjA5OCwicGF0aCI6Ii8zNzY4NTk4OS8yNDUwODg5NTAtOGU5NzU1NTEtOWU3OS00NmM2LTkzY2EtMjY1MWJjYzdkMDIyLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAzMTUlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMzE1VDAxMjgxOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTBkZmQxNTUyMmIxZjIxZTNiNzVkNGEwZDFhMzQyYzhlNzQwYWQ2ZmVlODU1NjdkN2JjMWJkNzRkODg0NTZjYjImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.-7gSUxj7-qFHxCfdpQCmYW2_owmfrgm6maGO73akFWY) > > 主要是后面的问题： RuntimeError: Error building extension 'fused_adam'，详情见上面； > > 问题已全部解决，微调成功推理微调后的模型权重文件时出现： File "/home/nbicc/data/anaconda3/envs/lm/lib/python3.8/site-packages/transformers/utils/hub.py", line 469, in...

哪位大佬帮忙看看，微调出现维度不一致问题； RuntimeError: The size of tensor a (25165824) must match the size of tensor b (12288) at non-singleton dimension 0

> > > > tokenizer的问题可以参考这里：[#111 (comment)](https://github.com/THUDM/VisualGLM-6B/issues/111#issuecomment-1579019781) > > > > > > > > > tokenzier重新运行是正常； ![image](https://private-user-images.githubusercontent.com/37685989/245088950-8e975551-9e79-46c6-93ca-2651bcc7d022.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTA0NjYzOTgsIm5iZiI6MTcxMDQ2NjA5OCwicGF0aCI6Ii8zNzY4NTk4OS8yNDUwODg5NTAtOGU5NzU1NTEtOWU3OS00NmM2LTkzY2EtMjY1MWJjYzdkMDIyLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAzMTUlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMzE1VDAxMjgxOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTBkZmQxNTUyMmIxZjIxZTNiNzVkNGEwZDFhMzQyYzhlNzQwYWQ2ZmVlODU1NjdkN2JjMWJkNzRkODg0NTZjYjImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.-7gSUxj7-qFHxCfdpQCmYW2_owmfrgm6maGO73akFWY) > > > 主要是后面的问题： RuntimeError: Error building extension 'fused_adam'，详情见上面； > > > >...

请问在阿里云上部署，连接不上huggingface网站的问题怎么解决呀？

> ![5262d26499e54c1efcfe05c3c892ee12](https://private-user-images.githubusercontent.com/112093823/316390030-119c9b04-e69d-4680-9626-a1aec4989dc1.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTE1MzA1NzQsIm5iZiI6MTcxMTUzMDI3NCwicGF0aCI6Ii8xMTIwOTM4MjMvMzE2MzkwMDMwLTExOWM5YjA0LWU2OWQtNDY4MC05NjI2LWExYWVjNDk4OWRjMS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwMzI3JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDMyN1QwOTA0MzRaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT02MTI5ZTZkZGU5NTRiMTczYzc1NmY2MTE5YTVkMDYzZTU2ZTkyMGY0N2I0ZDgzZjU0ZDcxOWY2ZmYzOTQ2NzcyJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.oab2LXqsPShi7pwkHTCWIR6tx0thNRsPIJutucsZxjM) 请问在阿里云上部署，连接不上huggingface网站的问题怎么解决呀？下载到阿里云服务器使用