csp icon indicating copy to clipboard operation
csp copied to clipboard

Add fast `csp.Struct` to Arrow record batch conversion

Open timkpaine opened this issue 1 year ago • 0 comments

Now that @arhamchopra has taken care of fast json conversion, we should move on to fast Apache Arrow conversion. We should have 2 mechanisms for conversion:

  • converting an individual struct, necessary but probably less efficient overall
  • collecting lists of structs and converting to record batches on either a timer or after a certain window size

Native arrow conversion will make certain use cases substantially faster, including in our csp-based webserver framework, but should also have benefits for other adapters like Kafka and websockets.

timkpaine avatar Jun 19 '24 01:06 timkpaine