Stochastic Depth Determinism

[ajaysaini725]

There are a few problems with StochasticDepth determinism right now:

• use_same_gpu_seed assumes each process has exactly the same seed when instead each process has seed = user provided seed + global_rank
• The generator object created in apply_stochastic_depth needs to be seeded otherwise the use_same_depth_across_gpus functionality is broken because the generator for each process is initialized with a random seed.

** Environment ** Any multi-GPU environment

** To reproduce Train a model with StochasticDepth and the seed set on multiple GPUs.

Expected behavior

use_same_gpu_seed and use_same_depth_across_gpus should work deterministically.