flatbuffers icon indicating copy to clipboard operation
flatbuffers copied to clipboard

[C++] parser: Add zero_on_float_to_int logic and improve json parsing.

Open Dwordcito opened this issue 5 months ago • 2 comments

Motivation

This change addresses a fundamental issue when parsing JSON serialized using libraries like cJSON that represent all numbers as double.

Problem Statement

  1. Precision limitation: double values in JSON can only represent integers exactly up to 53 bits (2^53 - 1 = 9,007,199,254,740,991)
  2. Precision loss: When an integer field in FlatBuffers contains values exceeding this limit, cJSON converts them to double, causing precision loss ex "1.8446744073709552e+19"
  3. Parsing failure: The current parser rejects these float values for integer fields, causing the entire parsing operation to fail

Implemented Solution

Instead of failing the entire parsing operation, this change introduces the zero_on_float_to_int option that:

  • Detects when a float value is received for an integer field
  • Preserves the validity of the parsing event
  • Assigns a default value (0) to the problematic field
  • Emits a warning to inform about precision loss

Use Cases

This is particularly useful for systems that:

  • Process large volumes of JSON data with 64-bit integers
  • Require fault tolerance in parsing operations
  • Prefer partially valid data over complete parsing failure
  • Integrate with legacy systems using cJSON or similar libraries

Behavior

// With zero_on_float_to_int = true
"{ id: 1.8446744073709552e+19 }" → id field = 0 (with warning)

// With zero_on_float_to_int = false (default)
"{ id: 1.8446744073709552e+19 }" → parsing fails

Test Coverage

The change includes comprehensive test coverage in ZeroOnFloatToIntTest() to ensure the feature works correctly in both enabled and disabled states.

Dwordcito avatar Jun 27 '25 14:06 Dwordcito

@dbaileychess I hope you're okay, we're using this change in our product, I think it would be good to have it in the library (it would help us a lot)

Dwordcito avatar Jun 27 '25 18:06 Dwordcito

hey @Dwordcito - looking over this change, it looks like it is only introduced as a compile time constant at the moment? It may be worth exposing this functionality in a way that doesn't require source code modifications -- either a cmake flag or a command line option (defaulting to the old behavior). Thoughts?

jtdavis777 avatar Nov 27 '25 00:11 jtdavis777