emac_esp32: emac_esp32_transmit(223): insufficient TX buffer size

[Scrin]

Today my gateway suddenly stopped sending data (both MQTT to my own server and HTTP to ruuvi cloud) all of a sudden. My gateway runs firmware v1.14.3. After about 20 minutes I power cycled the gateway and it resumed normal operation. The metrics scraped from the /metrics endpoint show no anomalies prior to this.

The logs from the gateway from when this started:

The logs mainly just repeat emac_esp32: emac_esp32_transmit(223): insufficient TX buffer size at a rate of a few times per second, with occasional other stuff in between, such as:

[TheSomeMan]

Hi @Scrin, how many Bluetooth sensors do you have nearby? We recently discovered and fixed an issue (#901) that can cause short-term memory shortages when a large number of sensors are nearby and with MQTT enabled. Also, we decided to switch MQTT QoS level from 1 to 0 to prevent using MQTT buffer (see #905). These fixes will be included in the upcoming v1.15.0 release.

[Scrin]

I have 19 active RuuviTags at the moment, since about two months ago. 16 active tags before that. Most of the tags run a custom firmware with reduced transmission interval, so I'm typically hovering at 4.5-5 advertisements per second received, if that makes any difference

[TheSomeMan]

I have 19 active RuuviTags at the moment, since about two months ago. 16 active tags before that. Most of the tags run a custom firmware with reduced transmission interval, so I'm typically hovering at 4.5-5 advertisements per second received, if that makes any difference

Yes, it looks like your RuuviTags generates about 90 messages per second and under these conditions, the problem is quite common. But the manifestation of this problem depends mostly on the MQTT server and the latency in the network. When MQTT QoS1 is used, the outgoing packets are placed to buffer until an acknowledgment from the server is received. Sometimes MQTT server may delay sending ACK packets for some reason and this results in a rapidly growing buffer of unconfirmed packets (they are stored in this buffer for 30 seconds). If you wish, you can test the alfa version where these problems are fixed. Here are the build artifacts: https://github.com/ruuvi/ruuvi.gateway_esp.c/actions/runs/6719912911 Docs: install custom firmware

[Scrin]

I'll test that if this problem comes up again before the next version reaches beta. The MQTT server I use is a local one, so the network latency is very low under normal conditions so I suspect this one was just a random occurrence