arbitrary[BigDecimal] generates numbers that cause NumberFormatExceptions when serialized and deserialized

[2rs2ts]

I had arbitrary[BigDecimal] generate the following: "0E+2147483648" in version 1.11.3. That scale is bigger than Int.MaxValue, so when you serialize the BigDecimal as a String and then try to reconstruct it with that String you will get a NumberFormatException. I understand why this is, but it's troublesome.

scala> BigDecimal(0, Int.MinValue)
res6: scala.math.BigDecimal = 0E+2147483648

According to the Javadocs, when you use this constructor, the value of the BigDecimal is going to be unscaledValue * (10^(-1 * scale)). So Int.MinValue actually becomes the maximum scale, and thanks to two's complement, this actually results in impossible 32-bit integer scale values.

A quick fix for this case is just to choose between Int.MinValue + limit + 1, but I think there's more than meets the eye to this issue. For instance, when I ran this generator several times to see what was going on by using this print call: println(s"unscaledVal is $unscaledVal, scale is $scale, limit is $limit, mc is $mc") I found that it'll make other problematic numbers too:

unscaledVal is -28334198897217871282176, scale is -2147483640, limit is 7, mc is precision=16 roundingMode=HALF_EVEN

So what's wrong with this?

scala> val orig = BigDecimal(BigInt("-28334198897217871282176"), -2147483640, UNLIMITED)
orig: scala.math.BigDecimal = -2.8334198897217871282176E+2147483662

scala> BigDecimal(orig.toString)
java.lang.NumberFormatException
  at java.math.BigDecimal.<init>(BigDecimal.java:554)
  at java.math.BigDecimal.<init>(BigDecimal.java:383)
  at java.math.BigDecimal.<init>(BigDecimal.java:806)
  at scala.math.BigDecimal$.exact(BigDecimal.scala:125)
  at scala.math.BigDecimal$.apply(BigDecimal.scala:283)
  ... 33 elided

I realize we can blame java.math.BigDecimal for its inconsistent constructors, but, it's still problematic here. Should there be a generator provided that doesn't suffer from this gotcha? Here is one that works for me:

lazy val genBigDecimal: Gen[BigDecimal] = for {
    unscaledVal <- arbitrary[BigInt]
    scale <- Gen.chooseNum(Int.MinValue + unscaledVal.abs.toString.length, Int.MaxValue)
} yield BigDecimal(unscaledVal, scale)

[rickynils]

@2rs2ts Thank you for the detailed analysis of this weird behavior. Would you say your genBigDecimal implementation would make a good candidate for Arbitrary[BigDecimal]? I think it looks fine, but this is not my area of expertise.

[2rs2ts]

@rickynils I'm not an expert on number formats and floating point arithmetic myself, and I had to stumble through this discovery to be able to describe it properly. The existing function makes use of the different MathContexts available and that's probably good for testing, but I don't know a lot about them. I don't know how they can affect the results, but I'm loathe to just use UNLIMITED. I'll see if I can get an answer on StackOverflow. Here is the question I have posted.