DiffRules.jl icon indicating copy to clipboard operation
DiffRules.jl copied to clipboard

rule for tanh has catastrophic cancellation for |x| > 20

Open tpapp opened this issue 1 year ago • 2 comments

One could simply revert #30 to fix this, but if one wants CSE then it can be calculated from first principles. I started a discussion which has numerical examples.

tpapp avatar Dec 13 '23 13:12 tpapp

ChainRules should suffer from the same problem, possibly the rule was just copied to it from DiffRules: https://github.com/JuliaDiff/ChainRules.jl/blob/40b9058c1a6798a4f72bda1227786d584b44c822/src/rulesets/Base/fastmath_able.jl#L41

devmotion avatar Dec 13 '23 13:12 devmotion

Note that the catastrophic cancellation happens way sooner than |x|=20 … by x = 4 the 1 - tanh(x)^2 formula has already lost 2 digits, by x = 6 it has lost 4 digits, and by x = 14 it has lost 10 digits.

See also @tpapp's discourse thread on this.

stevengj avatar Dec 13 '23 14:12 stevengj