yzwu

How to use gradient accumulate in BytePS torch DDP?

5

Did you have the demo for gradient accumulate in BytePS torch DDP? I can not find it in `byteps/torch/example`.

Did BytePS Support multiple NICs now?

13

Hello! Did BytePS implement multiple NICs internally?

the share memory optimization of RDMA in single machine

3

Hello, as shown in Figure, the paper said that the byteps did the share memory optimization of RDMA in single machine. Where is the relevant code? 

the question about byteps's timeline

20

Hello, according to https://github.com/bytedance/byteps/blob/master/docs/timeline.md, I get the timeline and find the grad tensor start time is unordered, which seems enable the priority scheduler. However, I check the code and find...

The performance of many execute times are different

3

**Phenomenon** Thanks for your excellent works! Recently I ran `example pytorch/benchmark_byteps.py` with RDMA distributed traning based on https://github.com/bytedance/byteps/blob/master/docs/step-by-step-tutorial.md and found a strange phenomenon, which is the speed perfomance with many...

run distributed training with RDMA reports the libibverbs warning

35

**Describe the bug** Excuse me, based on https://github.com/bytedance/byteps/blob/master/docs/step-by-step-tutorial.md, when I run distributed training with RDMA, the scheduler will print the following warning: ``` BytePS launching scheduler [19:05:14] byteps/server/server.cc:339: BytePS server...

### Is there an existing issue for this? - [X] I have searched the existing issues ### Current Behavior none ### Expected Behavior _No response_ ### Steps To Reproduce none...

[QUESTION] Is there a plan to introduce torch.compile into Megatron?

3

**Your question** Is there a plan to introduce `torch.compile` into Megatron to speed up the training performance?

run sh train.sh hangs in resnet50 benchmark with 4 and 8 gpus of single machine

1

### Question Hi, recently I build the oneflow environment and run the resnet50 of OneFlow-Benchmark, it runs successfully when use 1 gpus of single machine and 2 gpus of single...

[Iluvatar] Support V1_KVCACHE_SCHEDULER and paddleocr-vl rope mode

4

## Motivation 为了适配paddleocr-vl模型，特在天数硬件上支持V1_KVCACHE_SCHEDULER和paddle ocr vl的rope模式。除此之外，还验证了打开V1_KVCACHE_SCHEDULER后，之前适配的ERNIE纯文模型和ERNIE VL模型系列精度均正常 ## Modifications Pass ## Usage or Command Pass ## Accuracy Tests Pass ## Checklist - [x] Add at least a tag in the PR...