training icon indicating copy to clipboard operation
training copied to clipboard

[Unet3d] - Add infinite data loader to align epochs->samples transition

Open mmarcinkiewicz opened this issue 1 year ago • 1 comments

The main change is to replace standard dataloaders (finishing at each epoch) to "infinite" ones - uniformly sampling from the dataset till the program terminates. This technically might change the order of samples, but:

  1. Everyone is using fairly large batch size (i.e. 56, which is 1/3 of the dataset)
  2. I checked the distribution of samples and throughout the training it is indistinguishable from the previous behavior
  3. The new RCPs are very similar to the old ones - I'll open a PR soon. Indeed some RCPs are a bit faster than previously, not sure if it's related to the new behavior, a bug in the old behavior when we switched to samples, or just due to the variance

There is no need to modify the submission code.

mmarcinkiewicz avatar Jan 17 '24 16:01 mmarcinkiewicz

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

github-actions[bot] avatar Jan 17 '24 16:01 github-actions[bot]