piker icon indicating copy to clipboard operation
piker copied to clipboard

tsdb editing and mgmt toolz

Open goodboy opened this issue 3 years ago • 0 comments

with #308 landing shortly we'll have daemon support for storing large time series as part of a trader's normal toolset. one of the main utilities i'm sure most would expect is the ability to edit data sets in a way that's both intuitive and simple.

UX ideas

  • allow user selection of datums on a chart and export to a repl (likely ipython shell) which has all the user's computational tools as well as a standard set of tsdb mgmt APIs for "fixing" data that was either written wrong or for overlaying data sets on a whim
  • the mgmt APIs will likely want to include:
    • [ ] timeseries diffing and duplication detection (the latter for when bad writes happen due to bugs or mistakes)
    • [ ] tsdb write/delete ability and "reload" functionality, possibly by context-menu style right click
    • [ ] export to a format suitable for network transport?
    • [ ] gap detection for attempting to find missing history frame segments
  • a repl (prolly ipython) which wraps all this in a suitable env (with some extensions?) for doing all the above easily from cli (so likely we'll need https://github.com/goodboy/tractor/pull/306)

main holdups

  • [ ] marketstore doesn't seem to have any obvious way to delete series segments (https://github.com/alpacahq/marketstore/issues/264)
  • [ ] it also has no way to query granularity (https://github.com/alpacahq/marketstore/issues/207) though we are not really planning to ever require dowsampling more then 1m period

Probably more to come as this gets researched more.

goodboy avatar May 11 '22 13:05 goodboy