How to decode numbers with more than 128 bits precision?

[zawlazaw]

The JSON standard allows for decimal/integer numbers of arbitrary precision. In contrast to that, zio-json introduces a strict maximum precision at Lexer.NumberMaxBits = 128 bits. The motivation for some limit bound is given in the scaladoc of SafeNumbers as follows: "BigInteger, BigDecimal, Float and Double have a configurable bit limit on the size of the significand, to avoid OOM style attacks, which is 128 bits by default."

Numbers beyond that precision will not just loose precision, but even abort JSON decoding and fail.

Unfortunately, I did not find out any nice way for configuring zio-json in such a way that this limit is ignored, or at least that it can be set to a higher value, in order to let String#fromJson[Json] succeed.

So my question is: is there any nice, intended way for decoding higher-precision number literals with zio-json? For example, could I somehow re-configure the limit value, or introduce custom Decoders for BigDecimal?

If not, I could try to come up with some PR on that.

Also, another option to be discussed may be to simply increase this value to, say, 4096 bits (as long as this value is only applied to numbers that actually require such a high precision, and not to every number).

The only workaround I found out is the following really dirty hack, which redefines the static final int limit to a higher value via Java reflection (incl. redefining attribute modifiers). So, for those of you who need a workaround solution (like me, for interoperation with some geocoding API), here is the (evil) code (note: does work for openjdk-11, but no longer for openjdk-16):

import zio._
import zio.json._
import zio.json.ast.Json
import zio.json.internal.Lexer

import java.lang.reflect.{Field, Modifier}

object TempTest extends ZIOAppDefault {

  override def run = for {
    json <- ZIO.succeed("""{"longitude":13.38885989999999992505763657391071319580078125}""")
    _    <- Console.printLine("FAILS (Left): " + json.fromJson[Json])
    _    <- ZIO.attempt(setLexerNumberMaxBitsViaDirtyHack(256))
    _    <- Console.printLine("SUCCEEDS (Right): " + json.fromJson[Json])
  } yield ()

  def setLexerNumberMaxBitsViaDirtyHack(newValue: Int): Unit = {
    // really dirty hack (and may not work on some JVMs/SecurityManagers) to update static final int field Lexer.NumberMaxBits
    if (Lexer.NumberMaxBits != newValue) {
      val field: Field = Lexer.getClass.getDeclaredField("NumberMaxBits")
      field.setAccessible(true)
      val modifiers = classOf[Field].getDeclaredField("modifiers")
      modifiers.setAccessible(true)
      modifiers.setInt(field, field.getModifiers & ~Modifier.FINAL);
      field.setInt(Lexer, newValue)
      assert(Lexer.NumberMaxBits == newValue)
    }
  }
}

[plokhotnyuk]

Also, setLexerNumberMaxBitsViaDirtyHack cannot be cross compiled with Scala.js (and Scala Native in case of future support) and could require an additional configuration for the native-image compiler of GraalVM.

[zawlazaw]

I fully agree. To avoid a potential misunderstanding: I do definitely not propose to implement this hack or any reflection to zio-json codebase :) It is just a temporary workaround that may temporarily help out some of us. Instead of such a hack, we should either:

1. find a way how such numbers can already be decoded with existing functionality (I might simply not be aware of a good existing solution, perhaps you are? Perhaps one can replace the BigDecimal decoder within some local scope or so?)
2. if 1. does not work out, provide a PR that addresses this issue in a clean way (the solution approach should be briefly discussed before, as I see already 3 very different alternatives).

Regarding 1., I will wait one week or so for whether somebody has an idea. Afterwards, I would propose some alternatives, and then probably provide a small PR. It's probably not a big deal to solve this, but it can be a show-stopper for using zio-json when you have to process (valid) JSON documents with such high-precise and/or very large numbers.

[visox]

Hi, any update on this ? having an error in scala.js because of this ....value(expected a 128 BigDecimal)