neural-pipeline icon indicating copy to clipboard operation
neural-pipeline copied to clipboard

Add batch gradients accumulating (for increase batch without increasing memory usage)

Open toodef opened this issue 6 years ago • 0 comments

For solving this issue these steps needed:

  • Add method enable_grads_acumulation(steps_num: int) to Trainer class
  • Add gradients accumulating like described there
  • Write tests for a simple network, there will be compared 2 losses values:
    1. Calculated without gradients accumulating
    2. Calculated with gradients accumulating

For do this test need to provide same data input to model and same weights in model (last can be done by flushing weights to file).

  • [Optional] Explore how BatchNorm works with gradients accumulating. There says, that it's a problem (but disscussion from pre-relase of PyTorch 1.0 version)

toodef avatar Feb 14 '19 08:02 toodef