colian

Why attention_mask should repeat 'sp_size' here?

3

Thanks for your great work! https://github.com/PKU-YuanGroup/Open-Sora-Plan/blob/3c08cb2fef031dc659fbcfb33acac4c515d7bc51/opensora/utils/communications.py#L119 Since the input size of attention_mask is (1, 24, 60, 80), why still to repeat sp_size times here?

Nan loss after 20k+ steps of training.

2

Hi, have you ever seen this happened?

"closing signal SIGTERM", seems like RAM OOM?

2

Hi, when i run the latest v1.3 code for fine-tuning, it fails the training every-time when the program tries to save the checkpoint, as shown below. I have never met...