Feature Request: Add Mish activation

[digantamisra98]

Mish is a new novel activation function proposed in this paper. It has shown promising results so far and has been adopted in several packages including:

• TensorFlow-Addons
• SpaCy (Tok2Vec Layer)
• Thinc - SpaCy's official NLP based ML library
• Echo AI
• Eclipse's deeplearning4j
• Hasktorch
• CNTKX - Extension of Microsoft's CNTK
• FastAI-Dev
• Darknet
• Yolov3
• BeeDNN - Library in C++
• Gen-EfficientNet-PyTorch
• dnet
• DL4S

All benchmarks, analysis and links to official package implementations can be found in this repository

It would be nice to have Mish as an option within the activation function group.

This is the comparison of Mish with other conventional activation functions in a SEResNet-50 for CIFAR-10:

[extremety1989]

function mish(x) { return x * (Math.exp(Math.log(1 + Math.exp(x))) - Math.exp(-Math.log(1 + Math.exp(x))))/(Math.exp(Math.log(1 + Math.exp(x))) + Math.exp(-Math.log(1 + Math.exp(x)))); }

function derivativeOfMish(y) { let omega = Math.exp(3 * y)+4 * Math.exp(2 * y) + (6+4 * y) * Math.exp(y) + 4 * (1 + y); let delta = 1 + Math.pow((Math.exp(y) + 1), 2); return Math.exp(y) * omega / Math.pow(delta, 2); }

[extremety1989]

Can you please add mish function,that i provided, i already tested it on my custom neural network and it works great,better than sigmoid and tanh ! on XOR

[mubaidr]

That sounds great. Would love to see as contributor though.

[robertleeplummerjr]

Keep in mind we have the GPU implementations as well.

[extremety1989]

@mubaidr how can i apply this function as contributor?

[digantamisra98]

@extremety1989 are you planning to submit a PR?

[extremety1989]

@digantamisra98 no, sometimes it returns NaN when learning rate is 0.1, i do not know what is the probleme,maybe javascript

[digantamisra98]

@extremely1989 Mish has a Softplus operator which needs proper threshold to fix that NaN issue you might be facing.

[extremety1989]

@digantamisra98 my threshold is 0.5, i how much should i turn ?

[digantamisra98]

@extremety1989 the Softplus operator thresholds that Tensorflow use is in the range of [0,20]