pix2pix

pix2pix copied to clipboard

Sorry if this is a bit off-topic but this seems to be the best place to ask:

Do you think that adding one or more spatial transformer layers to the pix2pix architecture could increase the spatial range of the model and reduce "mosaic" artifacts? Right now it seems that there is a certain limit on how far the network can "see" around each pixel - so if the structures in a pair are very different it will not learn something meaningful. If I understand STLs correctly they allow to add moving, scaling and rotation of the input so I would imagine that this could help extending the range or finding the related structures in the paired image.

Interesting idea. If I understood you correctly, you mean adding a STL on top of the images before feeding them to the discriminator, right? This essentially introduces (learned) data augmentation to the training, which could potentially improve the results. Please let us know your findings if you decide to give it a try.