stylable icon indicating copy to clipboard operation
stylable copied to clipboard

How to run communication scheduling with BytePS

Open Rivendile opened this issue 4 years ago • 12 comments

Hello, I would like to run communication scheduling with BytePS. What parameters should I set? Is setting BYTEPS_SERVER_ENABLE_SCHEDULE=1, BYTEPS_SCHEDULING_CREDIT and BYTEPS_PARTITION_BYTES enough? What's the difference between server schedule and bytescheduler? Any help would be appreciated. Thanks a lot.

Rivendile avatar Jan 05 '21 12:01 Rivendile

The three parameters are enough. BYTEPS_SERVER_ENABLE_SCHEDULE is for the server process, but it is enabled by default though. The other two are similar to those used in ByteScheduler.

What's the difference between server schedule and bytescheduler?

ByteScheduler does not support scheduling at the server side.

ymjiang avatar Jan 05 '21 12:01 ymjiang

Thanks to your timely reply. Does byteps have scheduling at the worker side by default? Is there any parameter to control this feature?

Rivendile avatar Jan 05 '21 13:01 Rivendile

Does byteps have scheduling at the worker side by default? Is there any parameter to control this feature?

Yes, BYTEPS_SCHEDULING_CREDIT controls the scheduling at the worker side.

ymjiang avatar Jan 05 '21 13:01 ymjiang

Thanks a lot. Is the priority used for worker-side scheduling the index of the layers? What is the priority used for server-side scheduling?

Rivendile avatar Jan 06 '21 00:01 Rivendile

For both sides, the priority is determined by the tensor index.

You can refer to here if you are interested in how it works: https://github.com/bytedance/byteps/blob/master/byteps/server/server.cc#L473

ymjiang avatar Jan 06 '21 01:01 ymjiang

Hi, @ymjiang ,I tried to set these parameters to control scheduling behavior, which is successful while using MxNet BytePS. However, when I set byteps_server_enable_schedule=1 and byteps_scheduling_credit=4, the timeline looks like this: image It seems that there is no scheduling at all. Any suggestions?

Rivendile avatar Jan 07 '21 07:01 Rivendile

It seems that there is no scheduling at all. Any suggestions?

Actually I am not sure how you get this conclusion from this figure.. Enabling the scheduling only means that former layers are preferentially selected for communication, but this is not guaranteed.

ymjiang avatar Jan 07 '21 10:01 ymjiang

I'm a little confused. First of all, in MxNet, when I use these parameters, the former layers are selected for communication, and the latter ones that are not sent while the former one comes will be delayed. image As shown above, in MxNet, the tensor of gradient 28 wait for former layers. But in the figure https://github.com/bytedance/byteps/issues/348#issuecomment-755952360, the tensors of features 28, 26, 24, 17 show the similar pattern. They all have pull operations that are done early or late. On the other hand, don't the credit size and priority queue used in https://github.com/bytedance/byteps/blob/249006c9105d7b4fd09962eb133c3e76de1c8656/byteps/common/scheduled_queue.cc guarantee the preference, at least in the range of the credit size? Because the GetTask function gets the ready tensor with the highest priority.

Rivendile avatar Jan 07 '21 12:01 Rivendile

I think there is a problem with the for loop in GetTask function in https://github.com/bytedance/byteps/blob/249006c9105d7b4fd09962eb133c3e76de1c8656/byteps/common/scheduled_queue.cc. For example, when a large tensor with high priority is not chosen because of credit limitation, some small tensors with lower priority may be chosen. Thus the priority is not guaranteed. So why does not BytePS use true priority queue used in ByteScheduler (https://github.com/bytedance/byteps/blob/33fe89f5a6a691ec562ad2b0167f0192fd8ced7d/bytescheduler/bytescheduler/common/bytecore.py#L213) to get rid of this problem?

Rivendile avatar Jan 07 '21 12:01 Rivendile

For example, when a large tensor with high priority is not chosen because of credit limitation, some small tensors with lower priority may be chosen. Thus the priority is not guaranteed.

We partition the large tensors to equal size to avoid this. (small tensors do not matter much according to our tests)

ymjiang avatar Jan 08 '21 09:01 ymjiang

Thanks to your explanation! Could you please tell me what else makes the priority unguaranteed?

Rivendile avatar Jan 09 '21 02:01 Rivendile

Does byteps have scheduling at the worker side by default? Is there any parameter to control this feature?

Yes, BYTEPS_SCHEDULING_CREDIT controls the scheduling at the worker side.

Hi, since scheduling at the worker side is the default manner, could you share the differences between import byteps.torch as bps and import byteps.torch.cross_barrier as bps ?

lucasleesw avatar Jun 17 '21 02:06 lucasleesw