Adaptive activation function

[KirillZubov]

https://arxiv.org/abs/1906.01170

[atiyo]

Also related is their follow up work which uses neuron-wise adaptive activations: https://arxiv.org/abs/1909.12228

[atiyo]

Where is the best place for adaptive activations to live? Architectural aspects such as FastDense are in DiffEqFlux as far as I can tell. But does it make sense to add adaptive activations in there if their only use is in PINNs?

[ChrisRackauckas]

This PINN-only stuff can make sense here, like https://github.com/SciML/NeuralPDE.jl/pull/336 . We need to expand the docs to include a section for it.

[killah-t-cell]

https://arxiv.org/abs/2006.09661 this is also relevant.

[NeuralPDE]

Where is the best place for adaptive activations to live? Architectural aspects such as FastDense are in DiffEqFlux as far as I can tell. But does it make sense to add adaptive activations in there if their only use is in PINNs?

not just use in PINNs, Adaptive activation function's core idea is adding the loss function

to ensure the minimum $S(a)$, that is, the maximum $a$

to increase the gradient

to increase speed， avoid “Gradient disappearance”

so, It is suitable for all neural network related optimization problems

[YichengDWu]

It looks like they used a lot of effort to do very little, i.e., make the activation function have a large enough derivative. In fact simply using a Gaussian activation function has the same effect.