Formatting.jl icon indicating copy to clipboard operation
Formatting.jl copied to clipboard

Returns dramatically wrong results

Open goerz opened this issue 2 years ago • 2 comments

julia> using Formatting
julia> Formatting.fmt(".1e", 0.0003)
"2.10e-04"

goerz avatar Feb 02 '23 14:02 goerz

For comparison, the Printf stdlib returns the correct result:

julia> using Printf

julia> @sprintf "%.1e" 0.0003
"3.0e-04"

Seelengrab avatar Feb 02 '23 14:02 Seelengrab

Another example:

julia> using Formatting
julia> Formatting.fmt(".1e", 0.0006)
"5.10e-04"

~I've only seen this for these two particular numbers.~ Same error with 3e-8 and 6e-8. Probably an unfortunate interaction between the truncation algorithm and certain inexact floating point representations.

danielwe avatar Feb 21 '23 17:02 danielwe