BN parameters followup question

[d33dler]

In a previous issue concerning BN parameter freezing after first epoch in the old DN4 version (https://github.com/WenbinLee/DN4/issues/12) you mentioned it is done in order to prevent performance degradation which I assume is caused by overfitting to the training set distributions. However, in the new version you don't do it anymore. Curiously enough after removing the BN parameter freezing I even obtained a slight increase in accuracy R@1. Could you explain what is happening? I was always skeptical about the approach in your code to freeze after the 1st epoch, since this means we basically treat the rest of the training as fine-tuning - but this doesn't make sense, since we randomly shuffle the dataset in each epoch and randomly construct the mini-batches, plus we don't use a different split for finetuning. Thank you @WenbinLee

Additional notes: Before, when freezing the BN parameters after the first epoch, I was using kaiming init:

def init_weights_kaiming(m):
    if isinstance(m, nn.Conv2d) or isinstance(m, nn.Linear):
        nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='leaky_relu')
        if m.bias is not None:
            nn.init.constant_(m.bias, 0)

When switching to normal init, the performance wouldn't improve after the first epoch (after freezing the BN parameters) and after a 2 epochs it would start degrading. When BN freezing was removed, the training was working well.