RequestTimeout: timed out due to a ping-pong keepalive missing on time

[nileio]

• OS: linux
• Programming Language version: node v16.17.1
• CCXT version: 2.0.31

client = new ccxt.pro['binance']({
      newUpdates: true
    });
while (true) {
              try {
                const trades = await client.watchTrades(symbol);
                //doing things here
              } catch (e) {
                // debugger;
                console.log(e);
                // do nothing and retry on next loop iteration
                // throw e // uncomment to break all loops in case of an error in any one of them
                // break // you can also break just this one loop if it fails
              }
}

RequestTimeout: Connection to wss://stream.binance.com:9443/ws timed out due to a ping-pong keepalive missing on time

watchTrades on Binance every second minute the error is produced and simply things stop suddenly and can't figure out how to even handle the error gracefully or re-subscribe

[ttodua]

Hi @nileio , can you please tell if you still experience that issue? maybe it was some temporary network-related or etc, as I've tested (running already half an hour) and it doesn break even once.

[wunderb1t]

Hi @ttodua, we had also experienced that issue (from Oct 25, 2022 at 13:27:32.333 to Oct 25, 2022 at 20:40:15.209) but on Bybit and on watchTickers method. Message: Connection to wss://stream.bybit.com/realtime_public timed out due to a ping-pong keepalive missing on time

Also we've noticed that every time after ~10 hours of socket connection was up this issue appears again and only thing that helps to resolve it is restart the node instance. And we use per each exchange one ccxt instance for sockets without recreating them after socket connection lost maybe that is the case too.

[nileio]

@ttodua it is intermittent.. It happens when subscribing to multiple markets which is allowed by Binance for example, we should be able to subscribe to up to 1024 markets with one connection. Now I am trying to dig in the code to understand the ping-pong logic. Binance actually abides by WS standards which is great, it only requires pong frame when ping is received, and the socket should emit both ping and pong.. https://datatracker.ietf.org/doc/html/draft-ietf-hybi-thewebsocketprotocol-07#section-4.5.3 https://datatracker.ietf.org/doc/html/draft-ietf-hybi-thewebsocketprotocol#section-5.5.3

I understand some exchanges dont abide by standards , and sends ping in message,etc. this should be dealt with by exception.

Recommendation: let ccxt handle the standards, and clients override if needed for the specific exchange. In that case, ccxt should internally handle ping-pong and do not depend on the relevant exchange.js library to handle that.. hope this makes sense.

the issue does exist and I am guessing it has to do with 'timers' but I didnt have a chance to diagnose to provide more details.

[nileio]

Hi @nileio , can you please tell if you still experience that issue? maybe it was some temporary network-related or etc, as I've tested (running already half an hour) and it doesn break even once.

Please test it after subscribing for multiple markets, that is , 500 markets or so

[caiusCitiriga]

I am experiencing this issue too. Trying to subscribe to 338 markets trades and intermittently receive this error.

Sometimes appears this error too: connection closed by remote server, closing code 1008;

This can be related to: https://github.com/ccxt/ccxt/issues/12031#issuecomment-1312796717 ?

As @nileio said, Binance should let you subscribe 1024 streams from a single connection. Any idea?

I'm using this script to watch the trades on 338 markets:

async scanExchangeMarkets(markets: Market[]) {
    this.logger.verbose(
      `Exchange will be scanned for ${markets.length} markets`,
    );

    const loop = async (market: Market) => {
      while (true) {
        try {
          const trades = await this.client.watchTrades(market.symbol);
          const start = performance.now();
          this.tradesProcessingHandler(trades).then(() =>
            this.updateMeanTradesProcessingTime(start),
          );
        } catch (e) {
          this.logger.error(`Error getting ${market.symbol} trades data`);
          console.log(JSON.stringify(e.message, undefined, 2));
        }
      }
    };

    for (let market of markets) {
      loop(market);
      await Utils.sleep(1000);
    }
}

[caiusCitiriga]

@ttodua I can confirm that with 100 markets subscriptions from a single connection doesn't throw errors. With that being said, this is kinda critical. I'm aware that Binance is the most generous exchange in terms of data provisioning, but their documentation states that:

So, am I missing something? The ccxt manual says that the websocket connection is reused if already existing. So, since the watchTrades runs in a loop, I expect that ccxt adds a new subscription for a given symbol whenever a new symbol (that is not already subscribed) is passed to the function.

So, why does it throw errors if the number of markets to watch is way lower that the 1024 that binance says you can use?

I've also tried to add a delay at the top of the loop() to throttle the subscriptions, but after a while or in some cases after a minute it blows up anyway.

I've also found this issue that seems related: https://github.com/ccxt/ccxt/issues/9907

where @kroitor says that opening more than 50/100 subscriptions is not safe. But why? I'm really confused

Update:

I've managed to refactor the loop code, and to throttle it, by waiting 1s between a subscription and the next one. All went fine for about a couple of hours. Then the first set of errors appeared: timed out due to a ping-pong keepalive missing on time.

After this one, intermittently the Connection to wss://stream.binance.com:9443/ws timed out due to a ping-pong keepalive missing on time appeared (one for EACH market scanned). After the first appearance it will pop out intermittently but consistently for all the scanned markets.

Seeing this, I'm having a guess: Binance says that no more than 5 messages per 1s can be sent. So if I consider the amount of markets scanned (338) and the fact that all tries to send a subscription or pong message all together. I guess that ccxt is sending consecutively without any throttling the subscribe or pong messages to binance? If this is the case, may it be the cause of the 1008 error message?

If this is true, then I understand why @kroitor says that is not safe to open more than 50/100 subs. But this is a big limitation for us, and I guess that it should be fixed. Or is there maybe a workaround we can do for now?

Update 2:

After various optimizations and refactoring (mostly for performance reasons) I've successfully managed to run the watchTrades() method on 319 markets. BUT:

I had to reduce to half the number of time frames klines built for each market. This thing confuses me, since the code running for the building is async and non blocking. The only thing it does is arrays pushes, shifts and indexes lookups. No heavy cpu intensive operations or weird stuff. Also, everything is managed in an async fashion via rxjs events dispatching:

// The code that runs the infinite loop for watching trades
async scanExchangeMarkets(markets: Market[]) {
    this.logger.verbose(
      `Exchange will be scanned for ${markets.length} markets`,
    );

    const loop = async (market: Market) => {
      while (true) {
        try {
          this.trades$.next(await this.client.watchTrades(market.symbol));
        } catch (e) {
          this.logger.error(`Error getting ${market.symbol} trades data`);
          console.log(JSON.stringify(e.message, undefined, 2));
          throw new Error(e);
        }
      }
    };

    for (let market of markets) {
      loop(market);
      await Utils.sleep(1000);
    }
}

....

// The code that runs in response to $trades updates
async handleTrades(trades: Trade[], supportedTimeFrames: string[]) {
    const { symbol } = trades[0];
    for (let tf of supportedTimeFrames) {
      new Promise<void>(async () => {
        const history = this.getOrCreateHistory(symbol, tf, trades[0]);
        history.tradesBuffer.push(...trades);
        if (!history.fetchedMissingInitializationTrades) return;

        const builtKlines = ExchangeUtils.buildKlines(history.tradesBuffer, tf);
        for (const builtKline of builtKlines) {
          this.processKline(history, builtKline);
        }
        history.tradesBuffer = this.removeOldTrades(
          history.tradesBuffer,
          builtKlines[builtKlines.length - 1][0],
        );
      });
    }
  }

Am I missing something? For what I know about node's event loop this code is non blocking, also PM2 seems to confirm that, since the EventLoop latency is 0.13 ms. Another thing confirming the fact that there are no cpu intensive tasks is the CPU usage graph itself, and I'm running on a DigitalOcean CPU optimized server with 2 VCPUs and 4GB or RAM. So it's nothing too fancy:

At this point I'm kinda lost...

[ttodua]

@carlosmiei @pcriadoperez @sc0Vu do you have any thoughts on this? temporarily I would un-assign myself from this issue.

[pcriadoperez]

Hi,

So I think there are two different issues here, the 1008 "Too many requests" error and the ping-pong keep alive error.

• For the 1008 error, after a bunch of testing I found that the issue is we are only using one stream for all of our requests to binance. So even though they allow 1024 streams, we currently only use one, so when receiving too many messages over the stream it throws the 1008 error. I want to dig in more to this but I will be posting a PR with a solution shortly. This will also solve issue #12031

• Second we have the ping-pong error. Unfortunately I still have not been able to reproduce this error. @nileio , I'm not sure this answers your comment above, but we do change the ping/pong logic depending on the exchange if needed. My suggestion here is, I'll implement the multiple streams on binance which will remove the stress on each client, and as you mention it only happens when subscribing to a lot of markets, hopefully this will also solve the ping-pong issue. If not happy to continue looking into it.

[Kinzowa]

Hi, I just got the ping-pong issue with Bybit while listening to 2 markets watch_tickers and 2 private methods simultanously: watchMyTrades and watchOrders. From my experience with CCXT the websocket wrappers throw error with all exchanges I've tried.

timed out due to a ping-pong keepalive missing on time

[caiusCitiriga]

Hi,

So I think there are two different issues here, the 1008 "Too many requests" error and the ping-pong keep alive error.

• For the 1008 error, after a bunch of testing I found that the issue is we are only using one stream for all of our requests to binance. So even though they allow 1024 streams, we currently only use one, so when receiving too many messages over the stream it throws the 1008 error. I want to dig in more to this but I will be posting a PR with a solution shortly. This will also solve issue Connection closed by remote server, closing code 1008 - compare with cryptofeed library #12031
• Second we have the ping-pong error. Unfortunately I still have not been able to reproduce this error. @nileio , I'm not sure this answers your comment above, but we do change the ping/pong logic depending on the exchange if needed. My suggestion here is, I'll implement the multiple streams on binance which will remove the stress on each client, and as you mention it only happens when subscribing to a lot of markets, hopefully this will also solve the ping-pong issue. If not happy to continue looking into it.

@pcriadoperez Ok, knowing that ccxt uses only one stream for binance, the 1008 error now makes sense. So I'll wait for the PR and then I'll try again. Can you ping me here once it's merged? Thank you!

[caiusCitiriga]

hi @pcriadoperez I've updated the ccxt library to the last version. Indeed the 1008 error seems to have been solved, but unfortunately the "timed out due to a ping-pong keepalive missing on time" error is still present.

any ideas? 😞

[caiusCitiriga]

Update:

It happens even with only 1 market subscription, while building 12 timeframes. Although I'm not totally sure about this, but it seems somehow related to the amount of workload of the process, but why? How could be this related? And why would ccxt miss a ping-pong in time due to this?

[caiusCitiriga]

wow, awesome support. As always

[frosty00]

Update: It happens even with only 1 market subscription, while building 12 timeframes. Although I'm not totally sure about this, but it seems somehow related to the amount of workload of the process, but why? How could be this related? And why would ccxt miss a ping-pong in time due to this?

Yeah the pongs are sent in response to a ping and they are scheduled through the event loop. so any code that does not release the event loop would cause this to happen.

wow, awesome support. As always

We are working on other issues, and fixing bugs elsewhere in the lib. Sorry that your bug was not responded to.

[nileio]

@caiusCitiriga I have a feeling something fundamental is going on there .. throttling subscriptions to Binance might simply be the solution (managed by the backend).. I'd stay away for now and would find time to get back to my project over xmas.. @ttodua what is the update around this thanks!!

[ttodua]

@nileio I am not working on this subject, you might tag @carlosmiei / @pcriadoperez / @sc0Vu to find out who is tracking this issue.

[DOABrownie]

I'm also getting this error when watching only one order books on Kucoin. Ping pong missing error gets through after approx 1-3min of the order book polling in console.

[Kinzowa]

Hi,

So I think there are two different issues here, the 1008 "Too many requests" error and the ping-pong keep alive error.

• For the 1008 error, after a bunch of testing I found that the issue is we are only using one stream for all of our requests to binance. So even though they allow 1024 streams, we currently only use one, so when receiving too many messages over the stream it throws the 1008 error. I want to dig in more to this but I will be posting a PR with a solution shortly. This will also solve issue Connection closed by remote server, closing code 1008 - compare with cryptofeed library #12031
• Second we have the ping-pong error. Unfortunately I still have not been able to reproduce this error. @nileio , I'm not sure this answers your comment above, but we do change the ping/pong logic depending on the exchange if needed. My suggestion here is, I'll implement the multiple streams on binance which will remove the stress on each client, and as you mention it only happens when subscribing to a lot of markets, hopefully this will also solve the ping-pong issue. If not happy to continue looking into it.

I monitor the markets of 18 exchanges and am getting the ping-pong issue after few minutes with all exchanges except Binance. Could the patch you applied for Binance explain the absence of ping pong errors on this exchange?

@carlosmiei / @pcriadoperez / @sc0Vu

edit: and Bybit

[yyqiekenao]

I also encountered the same problem，s there any solution?

[caiusCitiriga]

@yyqiekenao yeah, stop using ccxt

[karambaq]

Same problem. Someone find the solution?

[nileio]

Nop - I moved on already !! using a much simpler implementation only for websocket which is pretty reliable much simpler typescript code but unfortunatley requires people to support each exchange - a heck of a task for one person - will publish the latest fork soon

[ragamufin]

In my case this was happening with watchBalance(). It would happen sporadically giving the error timed out due to a ping-pong keepalive missing on time stopping my code from continuing. Sometimes I would get the error NetworkError connection closed by remote server, closing code 1006 or failed due to a connection timeout

As explained in https://github.com/ccxt/ccxt/issues/7951#issuecomment-724762855 and https://github.com/ccxt/ccxt/issues/17162#issuecomment-1468196614, putting a try catch around it I am able to catch the error and then recover from it similar to the following:

while(true) {
	console.log('*** watchBalance ***')
	try {
		const balances = await exchange.watchBalance()
		console.log('balances:', balances)
	} catch(err) {
		console.warn(err)
	}
}

Since I'm running a loop, if there's an error, it gets caught and then in the next iteration of the loop the connection is restored. You could put any recovery/cleanup code you need in the catch block. This may help anyone having these errors with any of the methods that run through websockets such as in the following issues:

https://github.com/ccxt/ccxt/issues/15338 https://github.com/ccxt/ccxt/issues/14557 https://github.com/ccxt/ccxt/issues/14086 https://github.com/ccxt/ccxt/issues/22493 https://github.com/ccxt/ccxt/issues/7858 https://github.com/ccxt/ccxt/issues/10955

[bznsix]

在我的例子中，这种情况发生在 watchBalance() 时，偶尔会出现 timed out due to a ping-pong keepalive missing on time 错误，导致代码无法继续。有时我还会收到 NetworkError connection closed by remote server, closing code 1006 或 failed due to a connection timeout 的错误信息。

正如在 #7951 (comment) 和 #17162 (comment) 中解释的那样，在它周围加上 try catch 就能捕捉到错误，然后类似于下面这样恢复：

while(true) {
	console.log('*** watchBalance ***')
	try {
		const balances = await exchange.watchBalance()
		console.log('balances:', balances)
	} catch(err) {
		console.warn(err)
	}
}

由于我运行的是一个循环，如果出现错误，就会被捕获，然后在循环的下一次迭代中恢复连接。你可以在 catch 代码块中加入任何需要的恢复/清理代码。这可能会对通过 webockets 运行的任何方法（如以下问题）出现错误的人有所帮助：

#15338 #14557 #14086 #22493 #7858 #10955

Using try catch,will ccxt auto rebuild the connection?

[ragamufin]

Yes, if it's in a loop as in the example. In the catch block you can do whatever you need to do in order to handle other things that might break depending on your code. Since add the try - catch I've seen the errors come up and immediately the connection is re-established

[bznsix]

Yes, if it's in a loop as in the example. In the catch block you can do whatever you need to do in order to handle other things that might break depending on your code. Since add the try - catch I've seen the errors come up and immediately the connection is re-established

Yes,trying to fix my code use try catch,and it will auto reconnect.