Invariance breaking after training

[Allorth]

(I couldn't find your email address so I'm writing my question here). Good morning!

I'm new to both ML and GCNN and I've tried to replicate the results of Cohen on rotated MNIST using your code here. (Which implements Cohen's code if I'm understanding your code correctly.)

My network is the following (schematically, all GConv2D uses 10 filters of size 3x3 except for the last who uses 4x4): GConv2D('Z2' -> 'C4') Activation('relu'), GConv2D('C4'-> 'C4'), Activation('relu'), BatchNormalization(), MaxPooling2D(pool_size=(2, 2)), Activation('relu'), GConv2D('C4' -> 'C4'), Activation('relu'), GConv2D('C4' -> 'C4'), Activation('relu'), GConv2D('C4' -> 'C4'), Activation('relu'), GConv2D('C4' -> 'C4'), Activation('relu'), GConv2D('C4' -> 'C4'), Activation('relu'), GroupPool('C4'), GlobalAveragePooling2D(), Activation('softmax')

This network is invariant before training (entering the same image rotated gives the same output) and it trains (training set is not rotated) to around an accuracy of 97-99% but when running on the test set containing randomly rotated images I get an accuracy of about 70% and plotting the output for a single image rotated all four ways I see that the output of the network is no longer invariant to the input rotations.

I've got an idea that might effect the result: Stepping through the trained network with specific input (once again one image rotated through all different rotations) it looks as though there are numerical errors along the way (rounding). I tested to add a layer that just rounded all data to 3 decimals precisely before each GConv2D layer and this seemed to fix the invariance problem but the accuracy now dropped to 70% for training and was the same for testing.

But if I have understood things correctly for the invariance to break the transformed filters must be changed independently and I don't see how rounding errors could do this. (I haven't done a deep-dive in the code and studied exactly how the filters are updated but maybe it's time to do that.)

I'm confused of how it's possible for the invariance of this to break, any tips on as to why would be appreciated. :)

Thanks, Oscar