hearth icon indicating copy to clipboard operation
hearth copied to clipboard

Migrate to CBOR as the de facto serialization format

Open marceline-cramer opened this issue 1 year ago • 9 comments

We've been using JSON, which bloats message size, is inefficient to encode and decode floats and integers, and requires conversion of blobs to base64 strings in order to have reasonable fast serialization. This kinda sucks!

A new message format that alleviates these problems should have the following properties:

  • Binary format. Efficient data representation, especially for raw byte arrays.
  • Schema-less. Can be made human-readable without the need for an external schema definition. Can be directly imported into a scripting environment as an object type.
  • Simple. Easy to work with.

CBOR satisfies all of these goals. No more base64! No more processes bottlenecked by ser/de!

To-do:

  • #278
  • add ciborium as a workspace dep
  • remove serde_with from schema and DON'T base64-encode anything
  • add ciborium to runtime and use it to serialize events in PubSub, deserialize messages in SinkProcess, and serialize responses in RequestResponseProcess.
  • swap out serde_json for ciborium in init
  • swap out serde_json for ciborium in run_wasm example
  • remove serde_json dep from workspace, schema, ctl, init, wasm, server, fs, and runtime.
  • swap out serde_json for ciborium in guest
  • add ciborium to kindling workspace deps
  • update all of the kindling crates that use serde_json to ciborium
  • remove ciborium from kindling

Then, after #197:

  • update ByteVec with custom Deserialize and Serialize implementations that read and write the serde bytes data type directly
  • rework JsonAssetLoader into CborAssetLoader

marceline-cramer avatar Nov 29 '23 19:11 marceline-cramer