No default epsilon of 1e-16 for all optimizers

[JackTemaki]

#508

This is old Theano behavior.

[albertz]

So what do you suggest instead? No default at all, so require to be explicitly set? Or change the default to sth else? To what? Or just let it use the TF default if it is not set?

[JackTemaki]

Or just let it use the TF default if it is not set?

I would vote for this one, but everything is better than the current default which is far off compared to usual values. (1e-8 to 1e-10)

[albertz]

@JackTemaki (or anyone): Did you actually ever compare directly? Can you really say that e.g. Adam or Nadam is better with 1e-8?

[JackTemaki]

@JackTemaki (or anyone): Did you actually ever compare directly? Can you really say that e.g. Adam or Nadam is better with 1e-8?

No I did not compare it directly, all my setups of the last 2 years use 1e-8.