clp icon indicating copy to clipboard operation
clp copied to clipboard

ffi: Add support to serialize msgpack array/map into a JSON string.

Open LinZhihao-723 opened this issue 8 months ago • 0 comments

Description

In the next IR stream format, we use msgpack to represent a structured log event before CLP IR serialization. To handle the arrays inside the structured log event, we store them as unstructured arrays represented by a JSON string (same as clp-s). This requires methods to turn a msgpack object into JSON string. There are two existing ways to do this:

  1. nlohmann JSON library provides this feature by turning the msgpack bytes into an in-memory JSON object and then dumping the object into a raw string. However, from our experimental benchmark, this introduces significant run time overhead. It's more efficient to directly turn a msgpack in-memory objects into JSON string.
  2. Using << operator on an in-memory msgpack object will also return a JSON string: [https://github.com/msgpack/msgpack-c/wiki/v_2_0_cpp_json].

This PR adds methods to support serialization from msgpack array/map to JSON string without turning it into other intermediate representations. Even though msgpack-c already has support on JSON serialization, a microbenchmark shows that our implementation is 3 times faster than using <<.

Validation performed

  1. Add unit tests to ensure the generated strings are valid JSON strings, and have the same schema and values before/after serialization.
  2. Workflow passed in all platforms.

LinZhihao-723 avatar Jun 28 '24 21:06 LinZhihao-723