Paper - Implicit Neural Representations with Periodic Activation Functions

[NicolaBernini]

Overview

Paper overview

Implicit Neural Representations with Periodic Activation Functions

Arxiv: https://arxiv.org/abs/2006.09661 Project Web: https://vsitzmann.github.io/siren/

Images as NN

Say you have a Signal represented on a discrete grid and you want to use it to learn an implicit continuous and differentiable representation of it

You can use neural networks for this purpose since they represent continuous and differentiable functions and since its representation is not directly explainable then we can refer to it as an implicit representation

Practically, you can think at the input image as a discrete mapping between

• the pixel coordinates
• the RGB color

You can then use that single image as a Training Set for a NN to learn this mapping

If you do so, the NN representation, contrary to the original grid based representation has the following advantages

• it is differentiable
• it could be more memory efficient, preserving the details

How can we do that

To test this idea, we can start with a simple NN so let's start asking ourselves

Can a simple MLP with ReLU activations learn an image?

Before doing any experiment we immediately see a fundamental limitation in this kind of NN: the ReLU activation

It is (almost) everywhere differentiable (since the NN uses it to learn by backpropagating the gradient) however its derivatives define its representation capability for high frequency and since

• its first derivative is 0 or 1 and
• its second derivative is 0 (almost) everywhere

then we immediately see it can't represent signals where a significant part of the information is in the higher derivatives

This equates to applying a low pass filter to the image and in fact, this is confirmed empirically: see Fig.1 Col 2

So we can try to use more nonlinear activation functions like tanh and softplus having non-zero higher-order derivatives (letting aside the Vanishing and Exploding Gradient challenges in training them) however here we can see the experimental results, even if better than the ReLU, are not that great

The main contribution of the paper is then to propose an activation function allowing to learn an implicit neural representation of an image better than any other activation considered