Zenbot hanging

[arthurwolf]

Have I written custom code (as opposed to using zenbot vanilla): Yes, a custom strategy, but see bellow
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 20.04, Fresh install in VM
Zenbot version (commit ref, or version): What github clones this day
Zenbot branch: Same
NodeJS version: v10.19.0
Python version (when using a python script): 2.7.17
Exact command to reproduce (include everything): node zenbot.js sim binance.1INCH-USDT --strategy=grid --period=10m --start=202012250000 --end=202101010000 --grid_high=0.001 --grid_low=0.0001 --grid_number=100
Did I make any changes to conf-sample.js?: no

Describe the problem

Zenbot just holds on a regular basis during simulations.

I added tons of console.logs to trace the issue, and it seems like I get to onCollectionCursorEnd in sim.js, with numTrades != 0 it goes into the "else", so the engine.exit(exitSim) isn't hit, and from there it just does nothing anymore.

What is this "else" for and what is supposed to happen once it's hit?

I do not think this has to do with my custom strategy, but I could very well be wrong. Would still appreciate any idea in debugging this so I can fix whatever is happening here.

here's the order things happen during a crash:

1. enter collectionCursorStream.on('data') callback
2. As totaltrades == numtrades, call onCollectionCursorEnd()
3. Enter onCollectionCursorEnd()
4. go into the "else"
5. console.log([reversing, cursor, lastTrade, numTrades, totalTrades]) there shows: [ undefined, 1609032111821, { _id: 'binance.1INCH-USDT-783797', trade_id: '783797', time: 1609032125360, size: 126, price: 1.4184, side: 'buy', id: 'binance.1INCH-USDT-783797', selector: 'binance.1INCH-USDT' }, 100, 100 ]
6. exit collectionEnd(), while calling getNext()
7. enter getNext()
8. do tradesCollection.find() on : { query: { selector: 'binance.1INCH-USDT', time: { '$lte': 1609459200000, '$gt': 1609032125360 } }, sort: { time: 1 }, limit: 100, timeout: false }
9. exit collectionCursorStream.on('data') callback (presumably the one called in (1.), which is surprising to happen now, threads/promises/parralelism stuff?
10. in getNext() still, we haven't left yet, collectionCursor.stream() and collectionCursor.count(true) gets executed at this time (it might be before 9.)
11. Finally, we exit getNext()
12. Nothing happens anymore from now, none of the console.log()s get called anymore, no CPU is used.

[arthurwolf]

it's starting to look like this is a weird re-entry issue.

why is getNext() an async function but await isn't used when it's called?

[arthurwolf]

a dirty fix to this was to change limit: 100 to limit: 100000000, it now works 100% of the time it also had a weird side effect: execution of simulations is MASSIVELY faster why was this limited to 100, was this built to be executed on a raspberry pi originally or something like that? it's a very small limit... or maybe there's an operational reason why you need to do it by small groups? the reverse/symmetry thing?

[arthurwolf]

note: setting limit to very high values does not solve the issue, it just makes it much much more difficult to reproduce also, setting limit to 5 makes it much easier to reproduce

[arthurwolf]

Replacing most of the getNext() function with simply:

    let cur = tradesCollection.find(opts.query).sort(opts.sort);
    let results = await cur.toArray();
    results.forEach((trade, i) => {eventBus.emit('trade', trade)});
    engine.exit(exitSim);

functions great (so far...) I'm loosing the "reverse" feature, which I don't use/understand. If I understood what it is maybe I could re-implement it, will look into it.

[arthurwolf]

Down to two lines, works perfectly. Doesn't have the "symmetry" feature, but I don't understand that well enough to implement it. Any explanations welcome:

        await tradesCollection.find(opts.query).batchSize(10).sort(opts.sort).forEach((trade, i) => {eventBus.emit('trade', trade)});
        engine.exit(exitSim);