wdte icon indicating copy to clipboard operation
wdte copied to clipboard

wdte: use `big.Float` instead of `float64` for `Number`

Open DeedleFake opened this issue 7 years ago • 2 comments

A bit of testing has found that big.NewFloat(n).Text('f', -1) seems to, not surprisingly, produce significantly cleaner textual representations of numbers than just plain fmt.Printf("%v") does, allowing, for example, much higher Fibonacci numbers to be printed without resulting in scientific notation. While I'm still debating whether or not it makes sense for wdte.Number to actually be a wrapper for big.Float directly, I think it makes sense to at least give it a String() method that uses the above for now.

DeedleFake avatar Sep 20 '18 15:09 DeedleFake

After a bit more experimenting, it might not make sense to use just Text('f', -1). I'd like it to round large numbers of decimals but display large integers cleanly. I'm thinking something like the following:

if n.IsInt() {
  return n.Text('f', -1)
}

return n.Text('g', 10)

DeedleFake avatar Sep 20 '18 16:09 DeedleFake

While I'd like numbers to just be float64s to make usage of them a bit easier from the Go side, I'd also kind of like to somehow make Number capable of transparent precision-less math.

DeedleFake avatar Sep 20 '18 19:09 DeedleFake