Support Recurrent BatchNormalization with @recurrent

[dwf]

From @cooijmanstim:

I tried implementing it [recurrent batch normalization] in blocks at some point, but it became a mess because the batch statistics are in the inner graph so you can't use them directly to update the population statistics. AFAIK you would have to include them as scan outputs and it would break the whole @recurrent mechanism because whoever calls the recurrent apply (e.g. RecurrentStack) would have to be aware of the extra outputs. I don't understand how they do it in the Keras code.

This seems like it could be fixed with some extension of the @recurrent mechanism. I don't know enough about it though.

[dwf]

One idea: is it possible to update an external shared variable from within the scan inner graph? You could have an update that stashes the batch statistics in an external shared variable, and make sure that update runs before the population updates (@nouiz might know if this is viable).

[cooijmanstim]

I suppose the update attribute might be useful here, but a complicating factor is that at each timestep a different shared variable or a different subtensor of the shared variable should be updated.