gNMI Specification: JSON encoding of ±infinity and NaN

[tsuna]

I have a problem with this:

The JSON type indicates that the value included within the bytes field of the node value message is encoded as a JSON string. This format utilises the specification in RFC7159.

RFC7159 Section 6 says:

Numeric values that cannot be represented in the grammar below (such as Infinity and NaN) are not permitted.

This detail has been a recurring source of pain for us. We worked around this by doing this:

func jsonFloat(v float64) interface{} {
        switch {
        case math.IsInf(v, 1):
                return "+Infinity"
        case math.IsInf(v, 0):
                return "-Infinity"
        case math.IsNaN(v):
                return "NaN"
        default:
                return v
        }
}

Which sucks but given the RFC I'm not sure how we can do better. What I would like to see, however, is an agree-upon solution to this problem in the gNMI spec. If the above (or a variant thereof) sounds reasonable, then I would vote for that, but I'm also open to hearing alternatives.

[tsuna]

In a similar vein, we've had problems with large unsigned values, which occur frequently in interface counters. JSON implementations typically represent all numbers as floating point values (with double precision), so values beyond 2^53 - 1 (or less than -2^53 + 1) are tricky to transport correctly. It would be nice to get some clarification on that too in the spec (let me know if you think this should be discussed in a separate issue).

In Go this means that using UseNumber() is essential when deserializing the values, and then one must test for the value to know how to correctly turn it into a Go value.

[robshakir]

I think there's also some further complexity here that comes from (RFC7951)[https://tools.ietf.org/html/rfc7951] too. In this specification, YANG decimal64, uint64 and int64 are encoded as strings. This means that in JSON serialisation/deserialisation there is specific handling needed for these types, rather than simply representing them as a number. For JSON_IETF I think we probably have to live with this decision in the RFC, in the name of compatibility between implementations.

I presume the decision here is in the specific JSON encoding? To me, with the first option here, we need to have some specific parsing logic to deal with the fact that a schema node of some numeric type can actually be represented as a string, such that we try to cast it to the relevant type. In (pyangbind)[https://github.com/robshakir/pyangbind), I think that the strings above would actually throw errors if you returned them for any schema node that has a numeric type. In that case, do we simply choose a single type (string?) that can be used to represent these 'difficult to transport' numbers, such that one can always use strconv.ParseToXXX against it?

So, yes, I agree that we need to figure out how to handle this. LMK your thoughts on the above - we can schedule a hangout if it's easier.

[robshakir]

The other question, apologies, where would we encounter Infinity or NaN in a context that these are valid values for a leaf in a schema? It wasn't immediately obvious for YANG, so I was wondering whether there are other contexts where it occurs.

[ashu-ciena]

Hello @robshakir @gcsl

I am missing any conclusion on this thread. So, should we encode all numeric type data into strings as we have only int64 and uint64 datatypes in gnmi.proto and as per json RFC, these needs to be encoded in strings! Why not add an additional data type int32 which takes care of this confusion.

[gcsl]

I am going to close this old comment as the predates the inclusion of TypedValue for PROTO encoding of values which does allow for precise encoding of double and signed and unsigned integers. JSON does not support +/-Infinity or NaN and all numeric types are coerced to double precision losing precision for 64bit integers. If one cares about precise encoding and simplicity of handling, PROTO encoding should be used.

TypedValue has int_val, uint_val and double_val for encoding of numeric types which should cover all options. There should be no need to add types of lower precision such as int32 as int64 can carry an int32 with no loss.

https://github.com/openconfig/gnmi/blob/master/proto/gnmi/gnmi.proto#L108 https://github.com/openconfig/reference/blob/master/rpc/gnmi/gnmi-specification.md#223-node-values https://github.com/openconfig/reference/blob/master/rpc/gnmi/gnmi-specification.md#23-structured-data-types