Effect of learning rate and batch size on training time

[Shivank1006]

@iperov First of all thanks for this amazing repo. I have been working around this project for a while, and running my experiments on A6000 GPU, which supports maximum of 64 batch size. I want to decease the training time for RTM models. I did some experiments but the results are not that good. I had doubts regarding:

1. Can I increase the learning rate initially (like 8x) and decrease it by half every-time when I delete the inter_AB file ( I delete inter_AB when my loss stops decreasing) ?
2. Can I use multiple GPUs (2-4 ) to increase the batch size and assume that the iterations to delete inter_AB file will also decrease?
3. Is there any other way than the number of iterations to decide when to delete the inter_AB file?
4. Also I am having some wobbling issues with the face when there are extreme camera motions ( like zooming or when the subject comes close to the camera). Does turning off Random Warp after the 4 deletes of inter_AB will decrease this issue?

I am open for any suggestion to decrease the training time of a RTM model. Thanks in advance.