[Suggestion] Loss patterns close to typical training paths

[jotaf98]

Very inspiring work!

I was wondering if you thought about checking for arbitrary patterns close to some trained weights (i.e. constrain the weights to stay close to some constant weights, for example with an L2 loss). The trained weights could be snapshots from standard training (e.g. SGD, Adam), either after training or in the middle of it.

The reason is what one could ask how "far from the beaten path" these patterns are; if they're ubiquitous and will be commonly found during training, or if you need to move in weight-space very far to regions that are not very likely to occur with current training practices.

[universome]

Hi! Thanks! No, we have not considered such an idea, but it looks quite interesting. One can say: "Ok, we can find such patterns, but they exist very far from a trajectory you normally encounter during training, so there is no need to be afraid of such irregularity". It's an interesting question if it's true or not. In our experiments on MNIST/FashionMNIST we did find patterns with acceptable accuracy of 90/95 percents in the points of a pattern but it does not necessarily say that one can descend to these points with normal training despite them having a good accuracy.

[jotaf98]

Yes, that's what I was thinking about. Thank you for answering!