DiffRules.jl
                                
                                 DiffRules.jl copied to clipboard
                                
                                    DiffRules.jl copied to clipboard
                            
                            
                            
                        rule for tanh has catastrophic cancellation for |x| > 20
One could simply revert #30 to fix this, but if one wants CSE then it can be calculated from first principles. I started a discussion which has numerical examples.
ChainRules should suffer from the same problem, possibly the rule was just copied to it from DiffRules: https://github.com/JuliaDiff/ChainRules.jl/blob/40b9058c1a6798a4f72bda1227786d584b44c822/src/rulesets/Base/fastmath_able.jl#L41
Note that the catastrophic cancellation happens way sooner than |x|=20 … by x = 4 the 1 - tanh(x)^2 formula has already lost 2 digits, by x = 6 it has lost 4 digits, and by x = 14 it has lost 10 digits.
See also @tpapp's discourse thread on this.