synth
synth copied to clipboard
Lossless Sampling
Required Functionality
Currently the XExportStrategy and Sampler::sample functions work with vectors of JSON values.
This is handy, but it is loses information.
In fact for any data sink which has types which are a superset of the JSON data model (both Postgres and Mongo) you will be losing information as most types get serialized to a string (for example timestamps).
This can be a problem, as at insertion time you don't know what type to use on the client library.
Proposed Solution
- It feels like export strategy is doing too much + too tightly coupled to the sampler
- The sampler returns JSON values, which is not ideal. We want the sampler to return a vector of
Valuetypes incore::graph. - The type mapping has to be redone. We currently have JSON -> PG types & JSON -> Mongo. This needs to be re-implemented for
core::graph::Value-> X.
While we're at it, we should probably look into #28 regarding escaping (at least for Postgres).