Guangyao Zhang

## 📌 Checklist before creating the PR - [ ] I have created an issue for this PR for traceability - [ ] The title follows the standard format: `[doc/gemini/tensor/...]:...

## 📌 Checklist before creating the PR - [x] I have created an issue for this PR for traceability - [x] The title follows the standard format: `[doc/gemini/tensor/...]: A concise...

[FEATURE]: Support SP+PP in Llama etc.

1

### Describe the feature Currently most models like Llama does not support SP together with PP. Please add support for this.

### Describe the feature Please add Ulysses Sequence Parallelism support for Command-R, Qwen2 and ChatGLM

### Is there an existing issue for this bug? - [X] I have searched the existing issues ### 🐛 Describe the bug Main repo test_shard_llama fails for these configs: ```...

[BUG]: Pipeline Parallelism fails when input shape varies

1

### Is there an existing issue for this bug? - [X] I have searched the existing issues ### 🐛 Describe the bug Pipeline parallelism fails when input size is different....

### Is there an existing issue for this bug? - [X] I have searched the existing issues ### 🐛 Describe the bug TP+Split Gather(Acc) 4GPU Original FP16 Model: 0.755 FP8...