Integrates DeepSpeed-Ulysses sequence parallel, enabling efficient training of large language models with ultra-long sequences.

[githisw]

Reminder

• [x] I have read the above rules and searched the existing issues.

System Info

This project is an extension of LLaMA-Factory that integrates DeepSpeed-Ulysses sequence parallel technology, enabling efficient training of large language models with ultra-long sequences. https://github.com/githisw/LLaMA-Factory I have test it, an will release report later.

Reproduction

Put your message here.

Others

No response