Omit property names

[hpx7]

trafficstars

I'm using msgpack to transmit objects with a fixed schema, so it feels wasteful to encode the property names with each payload. Is there some way to provide some schema information to the encoder/decoder during initialization so that property names can be excluded from the serialization output?

Proposal of what this api could look potentially like:

import { ValueOnlyEncoder, ValueOnlyDecoder } from "@msgpack/msgpack";

const schema = ... // some representation of your object schema, contains the property names
const encoder = new ValueOnlyEncoder(schema);
const decoder = new ValueOnlyDecoder(schema);

const data = { foo: [1, 2, 3], bar: "baz" };

const encoded = encoder.encode(data); // fails if data doesn't conform to schema
const decoded = decoder.decode(encoded);

[hpx7]

From a quick search, this project seems to be the closest match for the api above: https://github.com/sitegui/js-binary

[gfx]

As you said, decoding keys is very slow, so this library already has a mechanism to cache short keys contributed by @sergeyzenchenko:

• https://github.com/msgpack/msgpack-javascript/blob/master/src/CachedKeyDecoder.ts

You can see a cache-hit rate by running benchmark/benchmark-from-msgpack-lite.ts with CACHE_HIT_RATE=1:

CACHE_HIT_RATE=1 npx ts-node benchmark/benchmark-from-msgpack-lite.ts
...
CACHE_HIT_RATE: cache hit rate in CachedKeyDecoder: hit=36318175, miss=25, hit rate=0.9999993116398941

Of course, better implementation is always welcome.

[gfx]

Schema-based serialization should be useful for robust systems, by the way. I'd like to design it after https://github.com/tc39/proposal-decorators is standardized (see also #91).

[hpx7]

@gfx https://github.com/msgpack/msgpack-javascript/pull/54 is a really nice performance boost, but if I understand correctly it still requires sending keys over the wire? I'm mostly interested in being able to reduce the payload size by not including the keys at all, and instead reconstruct the keys from a static schema provided to the decoder. For my use case, this would greatly optimize network bandwidth usage!

[gfx]

I see. It makes sense, but in such a case, I recommend ProtocolBuffers, which is designed in the very case and has tools to maintain the schema.

[hpx7]

Yeah I've also been meaning to explore https://github.com/mtth/avsc, which has some pretty impressive benchmark numbers in comparison to protobufjs

[o-alexandrov]

@hpx7 could you please share what you picked in the end? And whether it worked well for you.

Also, you might be interested in msgpackr which claims better performance than this library and avsc (btw it’s not surprising avsc isn’t fast given their desire to keep additional dependencies to support node 0.x)