set random seed for a multi-GPU model

[Galaxy-Husky]

Hi, I have a question about how to set seed when using multi gpus.

The following code snippet uses torch.cuda.manul_seed funciton.

if is_cuda and seed > 0:
        # These ensure same initialization in multi gpu mode
        torch.cuda.manual_seed(seed)

But I found a warning in the document of torch.cuda.manual_seed, which says "If you are working with a multi-GPU model, this function is insufficient to get determinism. To seed all GPUs, use manual_seed_all()."

Would it be better to change torch.cuda.manual_seed(seed) to torch.cuda.manual_seed_all(seed) ?

[vince62s]

Thanks. Have you tried to run twice the same run? not getting the same results in multi gpus ?