backfill not fetching the wished days of history (interrupts, no error)

[firepol]

System information

• Have I written custom code (as opposed to using zenbot vanilla): vanilla
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 17.10
• Zenbot version: unstable, latest
• NodeJS version: 8.9.4
• Python version: 2.7.12
• Exact command to reproduce:

./zenbot.sh backfill binance.TRX-BTC -d 30

here what I get:

binance.XRP-BTC saved 10091 trades 30 days left ....................... binance.XRP-BTC saved 111559 trades 29 days left ....................... binance.XRP-BTC saved 179491 trades 28 days left ....................... binance.XRP-BTC saved 169093 trades 27 days left ....................... binance.XRP-BTC saved 88678 trades 26 days left ....................... binance.XRP-BTC saved 71651 trades 25 days left ............... download complete!

Then e.g. if I try to run:

./zenbot.sh sim binance.XRP-BTC --days 5

I get:

no trades found! try running zenbot backfill binance.XRP-BTC first

This was confusing, because I did a backfill of 30 days. Then checking the output I noticed that after "25 days left" it stopped.

Expected behavior: backfill MUST fetch the amount of days specified in the parameter, else it should display an error in red, so the user can actually see the problem and run the command again.

I did run it again with a smaller amount of days

./zenbot.sh backfill binance.XRP-BTC -d 5

here what I got:

binance.XRP-BTC saved 7815 trades 5 days left ....................... binance.XRP-BTC saved 182378 trades 4 days left ....................... binance.XRP-BTC saved 192550 trades 3 days left ....................... binance.XRP-BTC saved 205436 trades 2 days left ...... ................. binance.XRP-BTC saved 310715 trades 1 days left ................An error occurred { Error: binance GET https://api.binance.com/api/v1/aggTrades?symbol=XRPBTC&startTime=1515681192051&endTime=1515684792051 request timed out (10000 ms) at timeout.catch.e (/home/paul/tools/zenbot/node_modules/ccxt/js/base/Exchange.js:293:27) at constructor: [Function: RequestTimeout] } ....... binance.XRP-BTC saved 378790 trades 0 days left ...... download complete!

This time I could see an error (request timed out) but still I wish it would display a clear error message, not just "download complete!" (if there was an error, it's not complete).

Expected behavior: on errors it should display an error message instead of "download complete", ideally in red.

UPDATE: I think binance is experiencing problems/timeouts, so to whoever can investigate this, I suggest to test with binance.

[firepol]

When fixing this, it would be nice to check also the past data and allow to fetch it. I have "holes" in my backfilles and can't find a way to fill the data for those missing periods within a day. So e.g. in my simulations I see that a day is shorter than 24 hours because several hours of trades in that day are missing...

[jorisw]

Did some tests with this. It seems to be related to gaps in the data over at the exchange, and zenbot's resume mechanism. Example, from trying to download 2019's BTC-USDT history entirely from Binance, with some debug added:

// Data collected previously
skipping 3779 hrs of previously collected data 

// Picks the correct resume date
options: { product_id: 'BTC-USDT', from: 2019-06-07T20:00:24.265Z }

// Correctly skips over existing data
binance.BTC-USDT saved 5782 trades 259 days left 

// Gets no further trades at day 259, probably because at
// specified date, there were no more trades (outage?)
trades: []

// Since there's no further trades in the result, zenbot thinks it's done.
// Instead it should try the next day/interval period.
download complete! (!trades.length)

When I try the next day as a start date, manually, it continues to download:

% zenbot backfill binance.BTC-USDT --days 258

binance.BTC-USDT saved 8233 trades 258 days left
.....

So the (faulty) assumption in the backfill command seems to be that receiving an empty trade array for an API call, means we're done, without trying a later date. I think zenbot should just try the next day, until the date range (-days to now, or start - end) has been exhausted.