still on the same GPU when iter size is above 1 ?

[hli2020]

Hi, when iter_size is over 1, say 3, it still computes on one GPU, right? some confusing here in the code. I guess the default GPU_id is 1, aka the first one? loading existing model would choose gpu=1, while training from scratch doesn't have such a setting in the code.

[gcr]

oops! that comment, and the one near line 159, is left-over from a broken multi-GPU version.

The entire thing runs on a single GPU. You can set the GPU to use with cutorch.setDevice(i). See Facebook's multi-GPU code for a great example of multi-GPU training.