inference ---- out of memery

[AugWrite]

请问推理代码并行化demo是哪个？根据 readme这么调的，但是不行一直报错。。 CUDA_VISIBLE_DEVICES=4,5,6,7 python cli_demo_sat.py --from_pretrained /data/CogCoM/CogCoM/cogcom-chat-17b --local_tokenizer /data/CogCoM/CogCoM-main/vicuna-7b-v1.5 --fp16 --quant 8 --english --nproc_per_node 4

（服务器是8卡v100-32G） 报错： kenizer /data/CogCoM/CogCoM-main/vicuna-7b-v1.5 --fp16 --quant 8 --english --nproc_per_node 4 [2024-07-08 18:19:08,531] [INFO] [real_accelerator.py:133:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-07-08 18:19:10,471] [WARNING] Failed to load bitsandbytes:No module named 'bitsandbytes' [2024-07-08 18:19:12,346] [INFO] building CogCoMModel model ... [2024-07-08 18:19:12,348] [INFO] [RANK 0] > initializing model parallel with size 1 [2024-07-08 18:19:12,349] [INFO] [RANK 0] You didn't pass in LOCAL_WORLD_SIZE environment variable. We use the guessed LOCAL_WORLD_SIZE=1. If this is wrong, please pass the LOCAL_WORLD_SIZE manually. [2024-07-08 18:19:12,349] [INFO] [RANK 0] You are using model-only mode. For torch.distributed users or loading model parallel models, set environment variables RANK, WORLD_SIZE and LOCAL_RANK. [2024-07-08 18:19:27,219] [INFO] [RANK 0] > number of parameters on model parallel rank 0: 17639685376 [2024-07-08 18:19:45,973] [INFO] [RANK 0] CUDA out of memory. Tried to allocate 86.00 MiB. GPU [2024-07-08 18:19:45,974] [INFO] [RANK 0] global rank 0 is loading checkpoint /data/CogCoM/CogCoM/cogcom-chat-17b/50000/mp_rank_00_model_states.pt [2024-07-08 18:20:19,633] [INFO] [RANK 0] > successfully loaded /data/CogCoM/CogCoM/cogcom-chat-17b/50000/mp_rank_00_model_states.pt [2024-07-08 18:20:21,178] [INFO] [RANK 0] > Quantizing model weight to 8 bits [rank0]: Traceback (most recent call last): [rank0]: File "/data/CogCoM/CogCoM-main/cogcom/demo/cli_demo_sat_zd.py", line 167, in [rank0]: main() [rank0]: File "/data/CogCoM/CogCoM-main/cogcom/demo/cli_demo_sat_zd.py", line 70, in main [rank0]: quantize(model.transformer, args.quant) [rank0]: File "/data/anaconda3/envs/CogCOM/lib/python3.11/site-packages/sat/quantization/kernels.py", line 282, in quantize [rank0]: replace_linear(model) [rank0]: File "/data/anaconda3/envs/CogCOM/lib/python3.11/site-packages/sat/quantization/kernels.py", line 280, in replace_linear [rank0]: replace_linear(sub_module) [rank0]: File "/data/anaconda3/envs/CogCOM/lib/python3.11/site-packages/sat/quantization/kernels.py", line 280, in replace_linear [rank0]: replace_linear(sub_module) [rank0]: File "/data/anaconda3/envs/CogCOM/lib/python3.11/site-packages/sat/quantization/kernels.py", line 280, in replace_linear [rank0]: replace_linear(sub_module) [rank0]: File "/data/anaconda3/envs/CogCOM/lib/python3.11/site-packages/sat/quantization/kernels.py", line 247, in replace_linear [rank0]: setattr(module, name, QuantizedColumnParallelLinear( [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/data/anaconda3/envs/CogCOM/lib/python3.11/site-packages/sat/quantization/kernels.py", line 158, in init [rank0]: super(QuantizedColumnParallelLinear, self).init(*args, **kwargs) [rank0]: File "/data/anaconda3/envs/CogCOM/lib/python3.11/site-packages/sat/mpu/layers.py", line 256, in init [rank0]: self.weight = Parameter(torch.empty(self.output_size_per_partition, [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 96.00 MiB. GPU

[terryII]

可以通过如下方式调用torchrun --standalone --nnodes=1 --nproc-per-node=4 cogcom/demo/cli_demo_sat.py --from_pretrained /data/CogCoM/CogCoM/cogcom-chat-17b --local_tokenizer /data/CogCoM/CogCoM-main/vicuna-7b-v1.5 --fp16 --quant 8，是否量化按需添加 @AugWrite

[AugWrite]

哇谢谢！解决了！！（^ ▽ ^）