RFC: 64-bit Integer Type

[carterkozak]

render

[sfackler]

How are we going to handle the serialization/deserialization in Jackson, since we need SafeLong to continue serializing as a number and normal longs to serialize as strings?

[carterkozak]

We can implement simple serializers in the conjure-java-runtime ObjectMappers factory, similar to the way we serialize java long values safely in our logging library.

[sfackler]

Handling the serialization in conjure-rust requires a bit of hackery, but it's dooable: https://github.com/palantir/conjure-rust/pull/20

[markelliot]

I'd be curious if the common-case for 64 bit integers is that they're used as identifiers or actually used for math. If for math, it seems important we resolve the JavaScript/TypeScript question, because AFAICT there's no common way to do long-math in JS/we may not want to force that into JS.

If used as identifiers (and thus important to represent minimally, where possible to avoid memory bloat), we might want to consider a specialized ID-type like uuid, where, abstractly, the wire-value accurately represents the uniqueness/can be used directly for equality, and where some in-memory representations can be more efficient (by using a semantically equivalent value in fewer bytes).

At any rate, if it's intended to represent identifiers, I think this proposal is perhaps too general.

[carterkozak]

I don't think we should add another identifier type beyond rid and uuid, which are both strong identifier types, I would not want to recommend or endorse using 64-bit integers as identifiers.

Furthermore using an identifier wrapper around a 64-bit integer would significantly increase memory overhead over a language-specific primitive 64-bit integer.

It's relatively uncommon to do math on large numbers in the frontend, making these values effectively IDs already. The most common operations are equality comparison and sorting. Math operations can be used on these types using the linked javascript bigint proposal which is already implemented in chrome has has polyfills for other/older browsers. There are other libraries for javascript which provide this functionality, however we should focus on the one which we and google anticipate being adopted into ecmascript.

[markelliot]

It's relatively uncommon to do math on large numbers in the frontend, making these values effectively IDs already.

That's kind of my point -- why are we adding this type?

[sfackler]

The internal service that I work on uses 64 bit integers as timestamps in its API. Safelongs aren't sufficient since 2^53 nanoseconds is only ~104 days. We don't expect any frontends to ever be talking directly to this service, so the level of support in Typescript isn't really all that important to my use case in particular.

We're moving this service's APIs slowly over to Conjure, but lack of 64 bit integer support is going to be a pretty hard blocker on being able to handle even the majority of the API surface.

[carterkozak]

why are we adding this type?

Most of our code is neither typescript nor front end, between backend services large number values are more common and helpful, and the usage patterns mentioned above do not apply.

[uschi2000]

For that use-case, @sfackler, it seems sufficient to define int64 == alias of string, internal to your service, right? I'm a little skeptical of adding a half-baked implementation of a thing that would be used by one consumer (in one language) and for which a viable and wire-format-forwards-compatible alternative exists.

[sfackler]

My particular thing would be dealing with two languages, not one, but in general 64 bit integers should work just fine in all the languages we care about except one.

[uschi2000]

Yep, and I believe that that ("except one") is a deal-breaker.

[sfackler]

But what level of "fine-ness" are we okay with? Would a polyfilled BigInt be sufficient in Typescript?

Any API that wants to be consistent across a large variety of languages is going to have some amount of impedance mismatch. As an extreme example, set<double> isn't even representable in conjure-rust! It seems like we have to decide what level of support is minimally required across all the languages we care about.

[markelliot]

@iamdanfox I think there's a pretty big difference between rid, uuid and integer64. There's never going to be an expectation that rid and uuid have more than equality/ordering as types, and so it's natural to render as strings in typescript, whereas there should be an expectation for our numeric types that one can do math. If we're intent on adding a similar type, I'd again propose we target something more narrow, such as i64id, a 64-bit integer id with a natural representation in languages that support 64 bit numbers, and string otherwise.

Alternatively, we could wait for a common typescript polyfill for math, as noted in the other comments.

[iamdanfox]

@markelliot I think the intention of this integer64 proposal is that it is sent over the wire as a quoted string but in all practicable languages (Java, Golang, Python, Rust), you'd interact with a conjure 'integer64' type as a number (allowing sensible +, -, < etc).

In typescript, we could choose to bundle any one of the existing polyfills, BigInt, bn.js, bigi, yaffle or silentmatt-biginteger. However, I think it's actually preferable to represent these as javascript strings initially so that consumers are free to experiment with these polyfills... I'd only want to impose a specific one when we better understand performance & compatibility characteristics. Until then, I think it's appropriate to let consumers choose (they can even just use BigInt("1234") in Chrome today, with no polyfill at all!).

[stale[bot]]

This PR has been automatically marked as stale because it has not been touched in the last 14 days. If you'd like to keep it open, please leave a comment or add the 'long-lived' label, otherwise it'll be closed in 7 days.

[jhowarth-pal]

wanted to show interest from golang rpc side of things