recover trades with influx issue

[mattou78400]

Hi,

I can't seem to use the recover.js script when setting influx as the storage solution. It seems to be related to a missing "insert" function in influx.js

Am I missing something?

Thanks a lot for the great work!

[Tucsky]

Hmmm recover.js I barely remember this file dude, look like it's an old file that was used to trigger arbitrary recovery procedure (getMissingTrades function on exchange's class), but I haven't touched it in more than 5 months, and have made good progress on trades recovery since then. It is all automated now (either when an exchange reconnect or at server start), you shouldn't really use those manually anyway

I think i'm just going to delete this file Would be easier if you tell me what you are trying to do 🤔

[mattou78400]

Thanks for the quick response. I didn't know this file was deprecated, but rather the preferred way to build back the trade/candle history

What I'm trying to do is basically get missing data "easily". What's the preferred way to pull the missing data since Jan 1st 2022 for example?

Just launching index.js will only start monitoring trades as of when it's started

Thanks!

[Tucsky]

@mattou78400 I don't think it's possible, because exchanges won't allow you get 6 months of RAW trades from their api lol That why I didn't put much effort into that recover.js file, I just recover the last trades, not more than a couple minutes worst case Maybe some exchanges will tho, haven't checked. so this is what you can try :

What you can do is simulate a situation where you stopped the server on the 1st of Jan 2022, and now just starting it again, triggering a recovery trades procedure for that timespan. You need to remove this condition (that prevents getting more than 4h of trades) and edit products/connections.json file to write the correct timestamp (1640995200000 for 2022-01-01T00:00:00Z)

example :

{
  "BINANCE_FUTURES:btcusdt": {
    "exchange": "BINANCE_FUTURES",
    "pair": "btcusdt",
    "hit": 100,
    "startedAt": 1640908800000,
    "timestamp": 1640995200000
  }
}

You will probably run into 429 errors too (too many requests), the recovery just stops when the exchange return an http error so yeah... good luck 😃

[mattou78400]

Amazing, I was thinking of something like this instead

Thanks for the support, I'll report back with my results :-)

[mattou78400]

Trying to work it out, there seem to be no issues with exchanges and 429

But it does seem to me it'd be easier to fix up the recover.js file to accept influxDB, this script actually worked pretty great, albeit kind of slow, with file storage I'll look into it, thanks

[Tucsky]

I added a config to adjust the delay between request but if we set a too low value it will 429, but I run multiple instances on the same IP too so there is that

[mattou78400]

he's not liking the heap limit

is the script keeping all the pulled candles in cache?

[Tucsky]

Yeah I was thinking about that too As you noticed, the recovered trades stays in an array until the exchange is done recovering all of them. The issue with releasing trades before it's done is that it will mess with the order at the end.

Given the range to recover like this : [range.from ... range.to] As you fetch the exchange api to recover part of the missing trades, some exchange it will fetch from most recent to the most older and some from the most older to the most recent

Fetching from the most recent to the most older is bad especially for influx storage, because every 10s or so it takes the released trades and build 10s bars out of it. If you release in parts, especially in the wrong order, it will build bars made of partial trades, which is not an issue when they are processed in the good order as the last bar is stored in the memory and can be edited without being overrided with another chunk of partial trades, but not on the opposite

[Tucsky]

I can send you the trades maybe I wonder how many gigs of data that would be tho

[mattou78400]

hmm yeah indeed that sounds trickier than I initially thought... It would be incredibly useful if you could send me the data, or a dump of your influxDB 💯 If so do you have a Telegram I could reach you on?

[mattou78400]

I can send you the trades maybe I wonder how many gigs of data that would be tho

Could you check how big it is by any chance? wondering if it's anything "movable"

[Tucsky]

I did checked and it's about 40gb for 1yr of data compressed, BTC and ETH included (about 50markets?). So maybe if you only want 6 months of BTC it could lower it to 15gb 🤔 I don't store raw trades on influx, just bars for a set of timeframes so exporting the db won't help u. If you find a way to transfer 15-20gb for free... I'm willing to help

[mattou78400]

Well, to be honest the easier might be to upload the archive directly on one of my server, I could make you a ssh access? Would that work? Do you have an email or telegram I could reach you on? :)

Thanks a lot for the help!

[Tucsky]

Yes i believe that would work, just dm @tucsky on telegram