add sparse option to drop null/undefined from objects

[Matt-Esch]

I would like to drop keys with null/undefined values from objects. It would be preferable to not need to strip these values first and simply skip over them when packing. When unpacking into fixed data types, missing keys can be initialised with defaults. I've proposed adding a sparse option in this pr.

[kriszyp]

Sorry this kind of got lost in my notifications and I missed it. I would like to do some testing on how much of performance impact this would have, as that is my concern with this.

[philz]

I'm interested in dropping undefined as well! Cheers!

[PS1TD]

Need this too)))

[teetering]

Any updates on this? In particular, I think it's pretty critical to allow this for undefined at the very least so that it replicates the behavior of round-trip JSON stringify/parse...

Is it possible to do this currently using addExtension?

Thank you!

[kriszyp]

Still need some performance regression testing to see if this would negatively impact the majority of users that don't use this option.

[kriszyp]

Any updates on this? In particular, I think it's pretty critical to allow this for undefined at the very least so that it replicates the behavior of round-trip JSON stringify/parse...

Actually to clarify this: this PR is for skipping undefined and nulls. This is not the behavior of JSON.stringify. Are looking for an option to skip undefined, or both undefined and null? I would actually think that skipping undefined is probably more useful than null+undefined as in this PR? (although options could be added for both)

[teetering]

Actually to clarify this: this PR is for skipping undefined and nulls. This is not the behavior of JSON.stringify. Are looking for an option to skip undefined, or both undefined and null? I would actually think that skipping undefined is probably more useful than null+undefined as in this PR? (although options could be added for both)

Thank you for clarifying. I'm ambivalent about skipping null. As you say, it is the skipping serialization of undefined values that would be required to allow msgpackr as a drop-in-replacement for JSON stringify as serialization method.

The motivation: We have several services that communicate with each other via JSON-serialized payloads. Obviously this quite typical. It would be really nice if we could just switch to using msgpack where we care to do so. Unfortunately, some assumptions get broken where the parsing client is not expecting the undefined values to be received. Obviously this could be worked around in various ways but would require quite a bit of work. It would be nice if there was an option that replicated the JSON behavior since it is surely one of the most common use-cases of msgpack (as an alternative to JSON serialization).

FWIW, other libraries offer this option (I assume for the same reason).

[kriszyp]

Merged, but I have made some significant changes to improve readability, flexibility, and performance:

• To avoid performance regressions, this option is more narrowly applied to the code branch that handles the size minimizing variable map size for serializing to MessagePack maps. It doesn't affect the fast path for static sized maps, and it is also does not impact, nor is it available for serializing objects as records (which I believe makes sense, records are more appropriately used for more statically typed structures).
• I renamed this to skipValues, and it accepts an array of values to skip. This allows you to provide skipValues: [undefined] to only skip undefined values, skipValues: [undefined, null] to skip undefined and nulls (or anything else you want to skip). I think this is also more descriptive ("sparse" usually refers to arrays, which this doesn't affect at all).

[teetering]

Thank you so much, that is fantastic!

[teetering]

@kriszyp I assume based on the above, the following should work? I'm trying it with the latest, but seems that the undefined values aren't actually skipped...

const msgpackr =   require('msgpackr');
const packr = new msgpackr.Packr({skipValues:[undefined]});
const {unpack} = msgpackr;

const testThing = {testing:123, test:undefined, hi:'bye'};
for(const k in testThing) {
    console.log('before: '+k+' : '+testThing[k]);
}
const packed = packr.pack(testThing);
const testThing2 = unpack(packed);
for(const k in testThing2) {
    console.log('after: '+k+' : '+testThing2[k]);
}

Output:

before: testing : 123
before: test : undefined
before: hi : bye

after: testing : 123
after: test : undefined
after: hi : bye

[kriszyp]

You probably need: new msgpackr.Packr({useRecords: false, skipValues:[undefined]}); (See in the docs that that this can't be used with records, only with MessagePack maps.)

[teetering]

You probably need: new msgpackr.Packr({useRecords: false, skipValues:[undefined]}); (See in the docs...

Thank you, that did it. For what it's worth I had read the docs already. Not sure if that makes it more or less embarassing, but after re-reading I'm still not super-clear about the reprecussions of disabling useRecords. Adding to the confusion, if I'm reading correctly, useRecords defaults to false for the basic pack (in the docs mentions "like the standalone pack and unpack functions") but defaults to true for an instantiated Packr, is that right?

Thanks again.

[kriszyp]

useRecords is an extension that provides better performance and more compact serialization, but is not part of standard MessagePack, so it is not interoperable with other MessagePack libraries.

useRecords defaults to false for the basic pack (in the docs mentions "like the standalone pack and unpack functions") but defaults to true for an instantiated Packr, is that right?

Yes, that's correct. The standalone functions are basic/standard MessagePack (de)serializer functions, and the constructor is for constructor specific/optimized serializations.