neural-pipeline
neural-pipeline copied to clipboard
Add batch gradients accumulating (for increase batch without increasing memory usage)
For solving this issue these steps needed:
- Add method
enable_grads_acumulation(steps_num: int)
toTrainer
class - Add gradients accumulating like described there
- Write tests for a simple network, there will be compared 2 losses values:
- Calculated without gradients accumulating
- Calculated with gradients accumulating
For do this test need to provide same data input to model and same weights in model (last can be done by flushing weights to file).
- [Optional] Explore how BatchNorm works with gradients accumulating. There says, that it's a problem (but disscussion from pre-relase of PyTorch 1.0 version)