A way to reduce the amount of data received through watch* methods in pro

[quakemmo]

Preliminary Checks

• [X] I have already searched for existing issues and confirmed that this issue is not a duplicate

Is your feature request related to a problem? Please describe

Subscribing to something like watchBidsAsks() on a busy exchange yields hundreds, potentially thousands of messages every second. Everyone doesn't necessarily need that much data.

Describe the solution you'd like

A parameter like that would direct CCXT to stop processing updates of a particular kind after N have been processed already less than a second ago and potentially save a LOT of CPU power.

Describe alternatives you've considered

While there's an alternative of just using the fetch* methods periodically, being able to use the websocket paradigm but limit how much messages per second you are willing to receive would be nice. Getting updates through a websocket without waiting for the next fetch* cycle has value as it usually comes in quicker.

[carlosmiei]

@quakemmo Can you try to run the watchBidsAsks in background, ignoring its return, and periodically access the cache directly? using exchange.bidsasks (the processing continues in the background but you don't need to handle so many updates "directly".

[quakemmo]

@carlosmiei

Right now I have something like this:

async function watchBidsAsks(cex, tickers) {
    while (true) {
        try {
            const ret = await cex.watchBidsAsks(tickers);
        } catch (e) {    
            console.error(e)
        }
    }
}


async function main(exchangeName, priceTickers, volumeTickers) {
    const cex = eval(`new ccxt.pro.${exchangeName}({})`);
    await cex.loadMarkets ();

    watchBidsAsks(cex, priceTickers);

    await cex.close()
}


main('binance', priceTickers, volumeTickers);

I just let it run in the background and don't do anything with the data. For Binance, just WATCHING 18 of their highest volume pairs eats up about 50% of a i7-9750H core, I'm not even doing anything with the data. I basically want to get updateson eaach pair's spread, but don't need it 500 times per second (for something like BTC/USDT), like 5 times per second would be more than enough (even if that would mean some info loss of course) and I don't see a way to do it with CCXT.

[sc0Vu]

@quakemmo I'm not sure if it's a good idea to reduce messages from the WebSocket stream, as the data comes from exchanges in real time.

Have you considered using polling with the fetch function?

[quakemmo]

@sc0Vu As I said above, polling is an alternative that works, but watching the message arrive on websocket can be superior in many situations because you get aware of the event earlier than with polling.

Something like a throttle switch for those watch* methods would be cool. Like, throttle that particular channel to 2 updates / second, if there are more that come, just discard them until the next second.

[quakemmo]

There's another issue with using fetch instead of watch.

It seems that the way many (most?) exchanges implement something like fetchTicker(), is by fetching all the tickers from the exchange, then picking and feeding back the one the user asked. The amount of processing is enormous on a big exchange with a lot of pairs. As a matter of fact, completely remplacing watch with fetch yielded only a relatively small CPU usage decrease.