msgpack-javascript Add BigInt support

Add BigInt support

Open jasonpaulos opened this issue 2 years ago • 5 comments

This PR adds native support for JavaScript BigInts.

I didn't want to cause a breaking change in any way, so I've introduced a few different decoding options that essentially allow library users to opt into this feature. Additionally, since no BigInts are returned by this library by default, everything else should remain functional in environments where BigInt is not supported.

Encoding

Similar to Numbers, BigInts are encoded using the smallest possible MessagePack int type. The added benefit of encoding a BigInt is that integers larger than Number.MAX_SAFE_INTEGER can be encoded with the correct precision.

If you attempt to encode a BigInt larger than the maximum uint64 value or smaller than the minimum int64 value, an error will be thrown, since MessagePack does not support larger integer types.

Decoding

Decoding BigInts is more nuanced. I believe there is no single correct way to do this, since consumers of this library will want different behavior depending on their use cases. For example, some may never want to see a BigInt from this library, some may only want to see BigInts if the value being decoded cannot fit into a Number, and some may want to always receive BigInts for consistency.

Instead of choosing just a single way to decoding integers, I've offered the following possibilities, which can be specified with the new DecodeOptions.intMode option:

IntMode.UNSAFE_NUMBER: Always returns the value as a number. Be aware that there will be a loss of precision if the value is outside the range of Number.MIN_SAFE_INTEGER to Number.MAX_SAFE_INTEGER.
IntMode.SAFE_NUMBER: Always returns the value as a number, but throws an error if the value is outside of the range of Number.MIN_SAFE_INTEGER to Number.MAX_SAFE_INTEGER.
IntMode.MIXED: Returns all values inside the range of Number.MIN_SAFE_INTEGER to Number.MAX_SAFE_INTEGER as numbers and all values outside that range as bigints.
IntMode.BIGINT: Always returns the value as a bigint, even if it is small enough to safely fit in a number.

For backwards compatibility, IntMode.UNSAFE_NUMBER is the default value for decoding. As a result, there should be no change in behavior for users who do not specify this option.

At Algorand, we've had success adopting a similar approach for JSON integer decoding: https://github.com/algorand/js-algorand-sdk/pull/260

Tests

New tests have been added to ensure the new functionality works as intended. Additionally, I've enabled the bignum tests from msgpack-test-js.

Feedback is appreciated, please let me know if there are any questions or suggested changes!

Closes #115

May 05 '22 00:05 jasonpaulos

Hi @gfx, is this PR something you would consider adding to the library?

Jun 17 '22 21:06 jasonpaulos

We'd also love it if this could be merged. The current ExtensionCodec example isn't really usable since it encodes large integers as strings, which would require special casing on the decoding side (which, in our case, isn't in JS).