train error with exitcode: -4

[tiancaitzp]

The base model and dataset are loaded successfully, when training start, it will get error after a few seconds. The error is as follow:

[JThh]

May I know your command for training? Have you changed any line in your training codes?

[tiancaitzp]

command: torchrun --nnodes=1 --nproc_per_node=1 scripts/train.py configs/opensora/train/16x256x256.py --data-path /home/tanzhipeng/open-sora/Open-Sora/datasets/travel.csv

changes in code :

[JThh]

May I know why you set use_reentrant = True? It might be not recommended when there are nested modules (link).

[tiancaitzp]

When I set use_reentant = False，I got the same error.

[JThh]

May I know your outputs of nvidia-smi? The error codes of -4 are often flagged with Out-Of-Memory (OOM) issue.

[tiancaitzp]

[tiancaitzp]

May I know your outputs of nvidia-smi? The error codes of -4 are often flagged with Out-Of-Memory (OOM) issue.

Hi, is there any update about this issue?

[FrankLeeeee]

Hi @tiancaitzp , can you try it again with export CUDA_LAUNCH_BLOCKING=1? It might raise more specific errors.

[zhengzangw]

Your code modification seems correct.

[zhengzangw]

Please try OpenSora 1.2. If the problem still exists, feel free to re-open.