bt-ccxt-store icon indicating copy to clipboard operation
bt-ccxt-store copied to clipboard

Published 20 hours ago • verified publisher icon Dave-Vallance

can't resample data

Open mr-m0nst3r opened this issue 5 years ago • 8 comments

Hi there,

I'm trying to add 1 minute data to CCXTFeed, and then resample it to 5 minutes, so I can get MACD of 1m and 5m. But failed.

Here's the code:

# !/usr/bin/env python
# -*- coding: utf-8; py-indent-offset:4 -*-
from __future__ import (absolute_import, division, print_function,
                        unicode_literals)
import sys
import time
from datetime import datetime, timedelta

import backtrader as bt
from ccxtbt import CCXTFeed

class TestStrategy(bt.Strategy):
    params = (
        ('printlog', True),
    )

    def log(self, txt, dt=None, doprint=False):
        ''' Logging function fot this strategy'''
        if self.params.printlog or doprint:
            dt = dt or bt.num2date(self.data.datetime[0])
            print('%s, %s' % (dt, txt))

    def start(self):
        self.counter = 0
        print('START')

    def prenext(self):
        self.counter += 1
        print('prenext len %d - counter %d' % (len(self), self.counter))

    def __init__(self):
        self.macd = bt.indicators.MACDHisto(self.datas[0])
        self.macd2 = bt.indicators.MACDHisto(self.datas[1])


    def next(self):
        self.counter += 1
        price_txt = "Counter: " + str(self.counter) + " Open/Close/High/Low/Volume: " + str(self.data0.open[0]) + " - "+ str(self.data0.close[0]) + " - " + str(self.data0.high[0]) + " - " + str(self.data0.low[0])  + " - " + str(self.data0.volume[0]) + " Len: "+ str(len(self.data0))# + " Time Frame:" + bt.TimeFrame.getname(self.data0._timeframe) + " Len: "+ str(len(self.data0))
        self.log(price_txt)
        macd_txt = "MACD: {:.2f}, Histo: {:.2f}".format(self.macd.macd[0],self.macd.histo[0])
        self.log("MACD#1: " + macd_txt)
        macd2_txt = "MACD: {:.2f}, Histo: {:.2f}".format(self.macd2.macd[0],self.macd2.histo[0])
        self.log("MACD#2: " + macd2_txt)
    
    def notify_data(self, data, status, *args, **kwargs):
        dn = data._name
        dt = datetime.now()
        msg= 'Data Status: {}'.format(data._getstatusname(status))
        print(dt,dn,msg)
        if data._getstatusname(status) == 'LIVE':
            self.live_data = True
        else:
            self.live_data = False

if __name__ == '__main__':
    #cerebro = bt.Cerebro(quicknotify=True)
    cerebro = bt.Cerebro()

    

    #exchange = sys.argv[1] if len(sys.argv) > 1 else 'gdax'
    exchange = sys.argv[1] if len(sys.argv) > 1 else 'binance'
    symbol = sys.argv[2] if len(sys.argv) > 2 else 'ETH/USDT'

    #store = CCXTStore(exchange=exchange, currency='LTC', config=config, retries=5, debug=False)

    hist_start_date = datetime.utcnow() - timedelta(minutes=10)
    print('UTC NOW: ', datetime.utcnow())
    print('hist_start_data: ', hist_start_date)
    print('Using symbol: ', symbol)

    data = CCXTFeed(exchange=exchange,
                             dataname=symbol,
                             timeframe=bt.TimeFrame.Minutes,
                             fromdate=hist_start_date,
                             #todate=datetime(2019, 1, 1, 0, 2),
                             compression=1,
                             ohlcv_limit=2,
                             currency='USDT',
                             retries=5,

                             # 'apiKey' and 'secret' are skipped
                             config={'enableRateLimit': True, 'nonce': lambda: str(int(time.time() * 1000))})

    cerebro.adddata(data)
    cerebro.resampledata(data, timeframe=bt.TimeFrame.Minutes, compression=2)

    cerebro.addstrategy(TestStrategy)
    # Run the strategy
    cerebro.run()

And here's the output:

python ccxtbttest.py             
UTC NOW:  2019-06-21 07:19:54.687534
hist_start_data:  2019-06-21 07:09:54.687358
Using symbol:  ETH/USDT
START
2019-06-21 15:20:01.656669 ETH/USDT Data Status: DELAYED
prenext len 1 - counter 1
prenext len 2 - counter 2
2019-06-21 15:20:03.363956 ETH/USDT Data Status: LIVE
prenext len 3 - counter 3
prenext len 4 - counter 4
prenext len 5 - counter 5
prenext len 6 - counter 6
prenext len 7 - counter 7
prenext len 8 - counter 8
prenext len 9 - counter 9
prenext len 10 - counter 10
prenext len 11 - counter 11
prenext len 12 - counter 12
Traceback (most recent call last):
  File "ccxtbttest.py", line 90, in <module>
    cerebro.run()
  File "/Users/michael/.venv/bt-ccxt-store/lib/python3.6/site-packages/backtrader-1.9.74.123-py3.6.egg/backtrader/cerebro.py", line 1127, in run
    runstrat = self.runstrategies(iterstrat)
  File "/Users/michael/.venv/bt-ccxt-store/lib/python3.6/site-packages/backtrader-1.9.74.123-py3.6.egg/backtrader/cerebro.py", line 1298, in runstrategies
    self._runnext(runstrats)
  File "/Users/michael/.venv/bt-ccxt-store/lib/python3.6/site-packages/backtrader-1.9.74.123-py3.6.egg/backtrader/cerebro.py", line 1557, in _runnext
    dt0 = min((d for i, d in enumerate(dts)
ValueError: min() arg is an empty sequence

Any idea?

Thank you.

mr-m0nst3r avatar Jun 21 '19 07:06 mr-m0nst3r

Hi @mr-m0nst3r

Thanks for the submission and report. Yes, resampling currently does not work from my testing. It is an item to tackle in the future. I have marked this as a bug and will keep it open.

Dave-Vallance avatar Jun 21 '19 07:06 Dave-Vallance

Okay, thank you.

Btw, I guess I have to calculate the macd without using the bt.Indicators.

Thanks and hope someone could make this bug fixed.

mr-m0nst3r avatar Jun 21 '19 07:06 mr-m0nst3r

@Dave-Vallance

I find that if we remove fromdate parameter, the script runs well, but started at about 1 day before, without the error described above.

Any help with this finding?

Here's the question I posted on the backtrader community: https://community.backtrader.com/topic/1914/resample-live-feed-from-ccxt-problem/4

mr-m0nst3r avatar Jun 24 '19 06:06 mr-m0nst3r

After testing it seems that this bug only triggers when a bar which needs to be resampled is only a few seconds old. I haven't been able to trigger it while walking through the code.

I came up with this ugly fix: Inside of ccxtfeed.py -> _load

                    start = datetime.utcnow()
                    if start.second < 15:
                        time.sleep(15 - start.second)

                    self._fetch_ohlcv()

Log snippets from some testing I did with 1 and 2 minute bars:

2020-03-16 16:22:22.377627 - fetch_ohlcv - Attempt 0
03-16 16:22 - custom_classes.f_cerebro: WARNING  works as intended
2020-03-16 16:24:22.207071 - fetch_ohlcv - Attempt 0
03-16 16:24 - custom_classes.f_cerebro: WARNING  works as intended
2020-03-16 16:26:21.639287 - fetch_ohlcv - Attempt 0
03-16 16:26 - custom_classes.f_cerebro: WARNING  works as intended
2020-03-16 16:32:11.773225 - fetch_ohlcv - Attempt 0
03-16 16:32 - custom_classes.f_cerebro: WARNING  works as intended
2020-03-16 16:42:17.367415 - fetch_ohlcv - Attempt 0
03-16 16:42 - custom_classes.f_cerebro: WARNING  works as intended
2020-03-16 16:46:17.382720 - fetch_ohlcv - Attempt 0
03-16 16:46 - custom_classes.f_cerebro: WARNING  works as intended



2020-03-16 16:34:08.293936 - fetch_ohlcv - Attempt 0
03-16 16:34 - custom_classes.f_cerebro: WARNING  resampled bug triggered
2020-03-16 16:40:03.369245 - fetch_ohlcv - Attempt 0
03-16 16:40 - custom_classes.f_cerebro: WARNING  resampled bug triggered
2020-03-16 16:44:05.686620 - fetch_ohlcv - Attempt 0
03-16 16:44 - custom_classes.f_cerebro: WARNING  resampled bug triggered
2020-03-16 16:48:07.024158 - fetch_ohlcv - Attempt 0
03-16 16:48 - custom_classes.f_cerebro: WARNING  resampled bug triggered

LatchRazu avatar Mar 16 '20 07:03 LatchRazu

No, at least it is not working for me

@Dave-Vallance

I find that if we remove fromdate parameter, the script runs well, but started at about 1 day before, without the error described above.

Any help with this finding?

Here's the question I posted on the backtrader community: https://community.backtrader.com/topic/1914/resample-live-feed-from-ccxt-problem/4

geraybos avatar Apr 15 '20 05:04 geraybos

For me it worked when I checked while escape condition to: if len(self.data) != 0: break

dogenkigen avatar Jun 16 '20 11:06 dogenkigen

After testing it seems that this bug only triggers when a bar which needs to be resampled is only a few seconds old. I haven't been able to trigger it while walking through the code.

I came up with this ugly fix: Inside of ccxtfeed.py -> _load

                    start = datetime.utcnow()
                    if start.second < 15:
                        time.sleep(15 - start.second)

                    self._fetch_ohlcv()

Log snippets from some testing I did with 1 and 2 minute bars:

2020-03-16 16:22:22.377627 - fetch_ohlcv - Attempt 0
03-16 16:22 - custom_classes.f_cerebro: WARNING  works as intended
2020-03-16 16:24:22.207071 - fetch_ohlcv - Attempt 0
03-16 16:24 - custom_classes.f_cerebro: WARNING  works as intended
2020-03-16 16:26:21.639287 - fetch_ohlcv - Attempt 0
03-16 16:26 - custom_classes.f_cerebro: WARNING  works as intended
2020-03-16 16:32:11.773225 - fetch_ohlcv - Attempt 0
03-16 16:32 - custom_classes.f_cerebro: WARNING  works as intended
2020-03-16 16:42:17.367415 - fetch_ohlcv - Attempt 0
03-16 16:42 - custom_classes.f_cerebro: WARNING  works as intended
2020-03-16 16:46:17.382720 - fetch_ohlcv - Attempt 0
03-16 16:46 - custom_classes.f_cerebro: WARNING  works as intended



2020-03-16 16:34:08.293936 - fetch_ohlcv - Attempt 0
03-16 16:34 - custom_classes.f_cerebro: WARNING  resampled bug triggered
2020-03-16 16:40:03.369245 - fetch_ohlcv - Attempt 0
03-16 16:40 - custom_classes.f_cerebro: WARNING  resampled bug triggered
2020-03-16 16:44:05.686620 - fetch_ohlcv - Attempt 0
03-16 16:44 - custom_classes.f_cerebro: WARNING  resampled bug triggered
2020-03-16 16:48:07.024158 - fetch_ohlcv - Attempt 0
03-16 16:48 - custom_classes.f_cerebro: WARNING  resampled bug triggered

This works for me. Thanks. But are there any specific reasons to use 15 seconds? I used 5 seconds and it seems to work as well.

JiazhengChai avatar Dec 07 '20 07:12 JiazhengChai

This works for me. Thanks. But are there any specific reasons to use 15 seconds? I used 5 seconds and it seems to work as well.

No specific reason. Excessive caution.

LatchRazu avatar Dec 07 '20 10:12 LatchRazu