Bug in datahistorymanager

[bijanx]

New Issue

Context

New installation. Installed database. Ran migrations and seeded exchanges.

When I create a datahistory job it fails because this line is returning nil.

candles, err = m.candleLoader(jobs[i].Exchange, jobs[i].Pair, jobs[i].Asset, jobs[i].Interval, jobs[i].StartDate, jobs[i].EndDate)

./gctcli datahistory addjob savecandles --nickname=gateio-spot-btc-usdt-1m-candles --exchange=gateio --asset=spot --pair=BTC-USDT --interval=60 --start_date="2022-06-02 12:00:00" --end_date="2022-06-04 12:00:00" --request_size_limit=10 --max_retry_attempts=3 --batch_size=3

• Operating System: MacOS 14.3

• GoCryptoTrader version (gocryptotrader -version): GoCryptoTrader v0.1 arm64 go1.22.2 pre-release.

Expected Behavior

I expect it to download fresh data without any preexisting candles in the database.

It presume it should be able to download candles without preexisting data. There is no nil handling so the app fails.

Failure Log

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x80 pc=0x101c31470]

goroutine 264 [running]:
github.com/thrasher-corp/gocryptotrader/engine.(*DataHistoryManager).compareJobsToData(0x14000d6a090, {0x14000efe008, 0x1, 0x1012a2f64?})
        /Users/bijan/gocryptotrader/engine/datahistory_manager.go:201 +0x3b0
github.com/thrasher-corp/gocryptotrader/engine.(*DataHistoryManager).PrepareJobs(0x14000d6a090)
        /Users/bijan/gocryptotrader/engine/datahistory_manager.go:164 +0x120
github.com/thrasher-corp/gocryptotrader/engine.(*DataHistoryManager).runJobs(0x14000d6a090)
        /Users/bijan/gocryptotrader/engine/datahistory_manager.go:268 +0x108
github.com/thrasher-corp/gocryptotrader/engine.(*DataHistoryManager).run.func1.1()
        /Users/bijan/gocryptotrader/engine/datahistory_manager.go:245 +0x20
created by github.com/thrasher-corp/gocryptotrader/engine.(*DataHistoryManager).run.func1 in goroutine 180
        /Users/bijan/gocryptotrader/engine/datahistory_manager.go:244 +0xc0
exit status 2

[gloriousCode]

Thank you for providing the CLI command you are using and the (unfortunate) panic response. It'll make looking into things much easier.

I'll do an investigation soon, post what I've found and from there, look into developing a solution

[bijanx]

More information

It's failing here:

// LoadFromDatabase returns Item from database seeded data
func LoadFromDatabase(exchange string, pair currency.Pair, a asset.Item, interval Interval, start, end time.Time) (*Item, error) {
	retCandle, err := candle.Series(exchange,
		pair.Base.String(), pair.Quote.String(),
		int64(interval.Duration().Seconds()), a.String(), start, end)

	if err != nil {
		return nil, err
	}

err is not nil, so it returns nil, then tries to call here but candles is nil, which causes the panic

err = jobs[i].rangeHolder.SetHasDataFromCandles(candles.Candles)

Not sure what the expected behavior is here. Should it return an empty struct instead? I will give that a shot.

[bijanx]

In database/repository/candle/candle.go in the function Series there's this:

	if len(out.Candles) < 1 {
		return out, fmt.Errorf("%w: %s %s %s %v %s", ErrNoCandleDataFound, exchangeName, base, quote, interval, asset)
	}

Since there's no data in the DB yet, it's returning an error which is causing the issue downstream.

I've commented this out but it seems to be silently failing now because it hasn't ingested any data.

[bijanx]

That fixes this issue actually. It's failing for a different reason:

[ERROR] | DATAHISTORY | 16/04/2024 09:02:37 | GateIO unsuccessful HTTP status code: 400 raw response: {"label":"INVALID_PARAM_VALUE","message":"Candlestick too long ago. Maximum 10000 points ago are allowed"}