Spectral Positional Encoding

[BrianPugh]

I see in your FourierUnit you added an optional spectral_pos_encoding argument. Have you experimented at all with this? Has it improved/reduced performance?

[windj007]

Wow! Great that you've noticed that :)

We experimented with positional encoding in spectral domain just a little bit. It did not help for the inpainting on our benchmarks - but might work in other cases. But we did not explore that feature thoroughly enough to say something for sure.

I'll be happy to hear back if this feature helps :)

[BrianPugh]

Interesting, I would expect positional encoding (possibly a different encoding than a simple linear mesh) would have helped.

So, this suggests a few possible outcomes (1x1 filter/conv here will always refer to the conv in the frequency domain inside the Spectral Transform block):

1. The 1x1 filter doesn't take frequency into account/is frequency agnostic (this also applies to the original FFC paper).
2. Some sort of spatial information is latently encoded in the featuremaps. In this case, the 1x1 convolution takes frequency into account, but the positional encoding is redundant.
3. The frequency/phase domain isn't really important; each pixel in the spectral image is to just be interpreted as a different hash of the entire input featuremap.
4. The 1x1 filter doesn't actually do anything. Perhaps the real power just comes from applying BN and relu on the spectral image before applying the iFFT. Or, perhaps its inappropriate to perform the BN/relu since it limits what the post-iFFT transform image looks like.

[Marcelo5444]

Hi @BrianPugh, Have you done any further research on that?

[BrianPugh]

i have not had a chance/the resources to perform experiments with these changes.

[Mengmengbai]

Interesting, I would expect positional encoding (possibly a different encoding than a simple linear mesh) would have helped.

So, this suggests a few possible outcomes (1x1 filter/conv here will always refer to the conv in the frequency domain inside the Spectral Transform block):

1. The 1x1 filter doesn't take frequency into account/is frequency agnostic (this also applies to the original FFC paper).
2. Some sort of spatial information is latently encoded in the featuremaps. In this case, the 1x1 convolution takes frequency into account, but the positional encoding is redundant.
3. The frequency/phase domain isn't really important; each pixel in the spectral image is to just be interpreted as a different hash of the entire input featuremap.
4. The 1x1 filter doesn't actually do anything. Perhaps the real power just comes from applying BN and relu on the spectral image before applying the iFFT. Or, perhaps its inappropriate to perform the BN/relu since it limits what the post-iFFT transform image looks like.

Great idea！I agree with you. Maybe I can do some experiments.

[Ellohiye]

Interesting, I would expect positional encoding (possibly a different encoding than a simple linear mesh) would have helped. So, this suggests a few possible outcomes (1x1 filter/conv here will always refer to the conv in the frequency domain inside the Spectral Transform block):

1. The 1x1 filter doesn't take frequency into account/is frequency agnostic (this also applies to the original FFC paper).
2. Some sort of spatial information is latently encoded in the featuremaps. In this case, the 1x1 convolution takes frequency into account, but the positional encoding is redundant.
3. The frequency/phase domain isn't really important; each pixel in the spectral image is to just be interpreted as a different hash of the entire input featuremap.
4. The 1x1 filter doesn't actually do anything. Perhaps the real power just comes from applying BN and relu on the spectral image before applying the iFFT. Or, perhaps its inappropriate to perform the BN/relu since it limits what the post-iFFT transform image looks like.

Great idea！I agree with you. Maybe I can do some experiments.

Hi, have you done an experiment? What was the result?