[BUG]: 运行 train_prompts.py prompts.csv --strategy naive 失败

[exceedzhang]

🐛 Describe the bug

I download prompts.csv and run： python train_prompts.py prompts.csv --strategy naive --lora_rank 16 Traceback (most recent call last): File "train_prompts.py", line 122, in main(args) File "train_prompts.py", line 32, in main actor = GPTActor().cuda() File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 688, in cuda return self._apply(lambda t: t.cuda(device)) File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 578, in _apply module._apply(fn) File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 578, in _apply module._apply(fn) File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 578, in _apply module._apply(fn) File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 601, in _apply param_applied = fn(param) File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 688, in return self._apply(lambda t: t.cuda(device)) RuntimeError: CUDA error: out of memory CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. root@autodl-container-98b3119d3c-f07453a6:~/autodl-tmp/ColossalAI/applications/ChatGPT/examples#

RuntimeError: CUDA error: out of memory 我使用 A5000 GPU 24GB 显存，训练使用GPU内存需要多少？是否我运行参数有问题？请求大家帮助！

Environment

使用ChatGPT0.1.0版本

[Issues-translate-bot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

Title: [BUG]: Failed to run train_prompts.py prompts.csv --strategy naive

[exceedzhang]

[Issues-translate-bot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

[ht-zhou]

Thanks for your feedback. We suggest you to use colossalai_zero2 strategy to train instead of naive which may save GPU mem for you. You can also use train_prompt.sh for training demo.

[exceedzhang]

Thank you！I'll try again！

[chingfeng2021]

@ht-zhou Could not find 'RANK' in the torch environment 这个策略需要多少的显存？我试了一下，发现了额这个报错

Traceback (most recent call last): File "train_prompts.py", line 122, in main(args) File "train_prompts.py", line 25, in main strategy = ColossalAIStrategy(stage=2, placement_policy='cuda') File "/opt/conda/lib/python3.8/site-packages/chatgpt/trainer/strategies/colossalai.py", line 77, in init super().init(seed) File "/opt/conda/lib/python3.8/site-packages/chatgpt/trainer/strategies/ddp.py", line 25, in init super().init() File "/opt/conda/lib/python3.8/site-packages/chatgpt/trainer/strategies/base.py", line 23, in init self.setup_distributed() File "/opt/conda/lib/python3.8/site-packages/chatgpt/trainer/strategies/colossalai.py", line 110, in setup_distributed colossalai.launch_from_torch({}, seed=self.seed) File "/opt/conda/lib/python3.8/site-packages/colossalai/initialize.py", line 215, in launch_from_torch raise RuntimeError( RuntimeError: Could not find 'RANK' in the torch environment, visit https://www.colossalai.org/ for more information on launching with torch root@8b69ccfeec2f:/gpt/applications/ChatGPT/examples#

[ht-zhou]

Hi @JThh , can you help to answer this question?

[ac0ra]

Rank error may be fixed by executing as torchrun rather than directly as python.

torchrun --standalone --nproc_per_node=2 train_prompts.py <Insert args here>

[JThh]

@chingfeng2021, has the issue been resolved?

[binmakeswell]

We have updated a lot. Please check the latest code. This issue was closed due to inactivity. Thanks.