compare with relu

[shuaizzZ]

Just updated 4 hours ago, you are really amazing! Have you tried to make some comparisons with relu on some practical tasks?

[scart97]

In the original paper they have comparisons with other activation functions, but it's very focused on the modeling aspect. I tried image classification on imagenette (small subset of the full imagenet) using a xResnet18 architecture with both ReLu and Siren activations. I found that Siren enables training with higher learning rates, but requires stronger regularization and don't match the accuracy achieved with ReLu (87.4% ReLu vs 83.8% Siren). I will not take these numbers as conclusive, as batchnorm doesn't seem to play nice with Siren (when I tried to add it to the image fitting experiment strange artifacts appeared).

If you want to experiment with it, it's the notebook Siren_imagenette.ipynb

[scart97]

Sample of image fitting using the model with batchnorm, notice the strange patterns that are not present in the original model: