piker icon indicating copy to clipboard operation
piker copied to clipboard

TSDB ingest of historical data sets: a dedicated `piker` tool to run parallelized history gathering

Open goodboy opened this issue 7 years ago • 2 comments

Once we get to the point where we're pulling both historical and real-time data (after #247 lands) there will be an obvious need for a local caching system to avoid needless network usage/latency especially when displaying real-time charts for massive sets of ever increasing price data.


TODO:

  • [ ] a piker ingest command to run a parallelized history gatherer program based on a default symbol set and the user's config
  • [ ] some kind of crawler system to obtain all the symbols?

goodboy avatar Mar 28 '18 00:03 goodboy

ultra-finance seems to have an example of this in its stockCrawler script.

goodboy avatar Mar 28 '18 03:03 goodboy

For financial data time series I just found alpaca's MarketStore.

It looks like just the thing to back distributed charting and real time algorithmics. The provide a python client though it'll likely need to be trio-ized for streaming.

goodboy avatar Mar 22 '19 03:03 goodboy