Support for Spatial Transformer Networks

[beniz]

Spatial Transformers act as a warping / attention mechanism useful for a variety of tasks.

There's a handful of repositories implementing STN for Caffe, see https://github.com/BVLC/caffe/issues/3114

However, STN have not made it into official Caffe yet, so it would need to be added to dd Caffe backend https://github.com/beniz/caffe/tree/master_dd_integ and rebased as Caffe recently changed the distribution of layers in C++ files.

Original paper: http://arxiv.org/abs/1506.02025

Implementations of interest:

• https://github.com/happynear/caffe-windows/blob/master/src/caffe/layers/transformer_layer.cpp
• https://github.com/XiaoxiaoGuo/caffe-stn with https://github.com/XiaoxiaoGuo/caffe-stn/blob/master/include/caffe/custom_layers.hpp
• https://github.com/daerduoCarey/SpatialTransformerLayer

Applications of interest:

• Efficient downsampling
• Learning of affine transformation for image classification
• Attention mechanism for fine-grained image classification

[beniz]

This now appears to be readily available for Caffe, see https://github.com/matthieudelaro/caffeBVLCplus/tree/stn and https://github.com/BVLC/caffe/issues/3114#issuecomment-225789373

This is not part of official Caffe, so there's need for it to be merged in a branch of our Caffe backend, then test on a variety of architectures.

[ayumiymk]

Do you know some examples about STN used as attention mechanism？

[beniz]

This task has remained unattended, so no.

[beniz]

https://github.com/daerduoCarey/SpatialTransformerLayer appears to be an almost up to date version, the CPU code should be replaced with the more optimized version from the BVLCplus repository above.

Useful blog post: http://christopher5106.github.io/big/data/2016/04/18/spatial-transformer-layers-caffe-tensorflow.html