PreActivations for Shortcut Projection

[RishabGargeya]

Hi,

Great work here! I have a quick question-- Based on my understanding of your code and the MSRA paper, shouldn't your type B projection shortcuts in the full and bottleneck preactivation models take bn_pre_relu as their input, and not l (excluding the first block)?

As the identity mapping paper states, "For the bottleneck ResNets, when reducing the feature map size we use projection shortcuts [1] for increasing dimensions, and when pre- activation is used, these projection shortcuts are also with pre-activation."

Did I misunderstand something? Thanks for your time!

[FlorianMuellerklein]

I think that you might be right. I've been thinking recently that I have a mistake in the projections. I will look into it, thank you.