Backfilling kraken historical OHLC

[goodboy]

Back filling OHLC shm arrays has been added for the IB broker backend and we need this same functionality for any broker(s) we add.

The next natural one in line is kraken since they are our major NA supported crypto broker.

Notes:

• we currently already support streaming ohlc quotes via the ws api which are used to update the current bar
• we need to add backfilling using the public OHLC endpoint (note not part of the ws api)
  • our current api code for this defined on the internal Client.bars()
  • iirc this historical data can get a max of 720 bars per request, but we can likely batch with async once startpoint logic is determined using the since parameter defined for the endpoint

Hot tips for the implementation:

• piker's nascent "data feed api" can be viewed starting in piker/data/__init__.py where one will find the use of tractor's streaming system
• the tldr breakdown is that streaming clients create a Feed using piker.data.open_feed() and stream from its .stream attr - an ex of this in the charting code where the stream is passed down to an update task which real-time updates the graphics on the chart
• the opposite end of this stream it the "producer" side provided by the broker and it's data retrieved from APIs: each broker currently provides a module defined stream_quotes() as is requested in the data feed client code "streaming routine" (as defined in tractor) (this is the "IPC endpoint" that the Feed.stream maps to from above). So, each broker backend defines this stream_quotes() to provide real-time data to consumers of tractor's IPC system.
  • kraken's implementation is here
  • ib's implementation is here
  • another note: the way this is currently implementated for the ib backend is by spawning a new task which does the backfilling into an underlying shared mem array directly - this machinery currently does not exist in the kraken backend and likely figuring out a way to generalize this backgroung-backfiller-task as an api for brokers would be useful to accomplish as part of this work. Further, the kraken stream_quotes() implementation will likely have to move to using tractor's context streaming push api just like now used in the ib equivalent since we can't spawn tasks from an async generator function due to conflicts with trio's nursery semantics (and to some degree SC in general).
• be mindful of kraken's request limits - there might have to be some limit logic added as part of this to avoid throttling

[goodboy]

@guilledk apologies for leading you astray here. The post on kraken's support page says it all: https://support.kraken.com/hc/en-us/articles/218198197-How-to-retrieve-historical-time-and-sales-trading-history-using-the-REST-API-Trades-endpoint-

TLDR:

• only 720 (12 h) worth of 1m OHLC bars
• you can get the entire history of tick data!

So this is actually even better news then I had hoped since we'll now be able to retrieve data usable for real forward testing and well simulated backtests.

I'd like to instead focus on getting real-time 1s OHLC sampled charting and move away from using the OHLC ws endpoint to instead use the trade stream; this will get us to basically the same results as from IB and traditional market feeds :)

Let me know what you think. I'm also cool to start this effort.