clp
clp copied to clipboard
ffi: Add support to serialize msgpack array/map into a JSON string.
Description
In the next IR stream format, we use msgpack
to represent a structured log event before CLP IR serialization. To handle the arrays inside the structured log event, we store them as unstructured arrays represented by a JSON string (same as clp-s
). This requires methods to turn a msgpack object into JSON string. There are two existing ways to do this:
-
nlohmann JSON
library provides this feature by turning the msgpack bytes into an in-memory JSON object and then dumping the object into a raw string. However, from our experimental benchmark, this introduces significant run time overhead. It's more efficient to directly turn amsgpack
in-memory objects into JSON string. - Using
<<
operator on an in-memorymsgpack
object will also return a JSON string: [https://github.com/msgpack/msgpack-c/wiki/v_2_0_cpp_json].
This PR adds methods to support serialization from msgpack
array/map to JSON string without turning it into other intermediate representations. Even though msgpack-c
already has support on JSON serialization, a microbenchmark shows that our implementation is 3 times faster than using <<
.
Validation performed
- Add unit tests to ensure the generated strings are valid JSON strings, and have the same schema and values before/after serialization.
- Workflow passed in all platforms.