[Paper] Neural Plasticity-Inspired Foundation Model for Observing the Earth Crossing Modalities

[jacobbieker]

Arxiv/Blog/Paper Link

https://arxiv.org/abs/2403.15356

Detailed Description

In this they use the central wavelength of the image sensor channel to create an embedding for different EO sensors, and modify the weights of a transformer model depending on what input is happening. Quite interesting results, it does well across different sensors and number of channels for multiple tasks.

Context

I've had the idea of using a wavelength embedding, like a positional embedding, with transformers for a while now, but haven't been able to try it out. My idea was more have an embedding of the central wavelength and the full-width half-maximum, or, alternatively, the two end wavelengths for a given channel. This doesn't quite do that, but does do something similar, and has good results. Would be quite interested to try this out.