How model parallelize across GPUs?

[Gumpest]

Could you introduce more details in parallelizing across GPUs, like how to implement through PyTorch.

[RahulBhalley]

You can use PyTorch Lightning instead. It automatically parallelizes the model training across GPUs and also supports TPU with just a single argument.

[imankgoyal]

We use the nccl backend with PyTorch to parallelize the stream while inference (testing). For training, we use usual distributed setup.

[Gumpest]

So we need at least three GPUs to inference three streams in ParNet?

[imankgoyal]

Yes if you want to do the multi-GPU inference. Otherwise, you can also do single gpu inference but it will be slower.

[twmht]

@imankgoyal

foe the edge devcie, using mulit-gpu for inference is expensive, what is your opinion?