API might overload CLN gRPC API

[fusion44]

trafficstars

refs https://github.com/rootzoll/raspiblitz/issues/3182

This might be the result of a mix of problems:

• /list-all-tx being quite inefficient
• the WebUI polling /list-all-tx every two seconds
• RaspiberryPI being low powered

[cstenglein]

@fusion44 should I poll slower? Like every 10/20s?

[fusion44]

I'm not sure, need to test whether the polling or even the API really is the problem.

FYI: For every /list-all-tx call I fetch:

• every Invoice
• every sent payment
• copy the Sqlite file of CLN to get the on-chain stuff
• sort everything

and, depending on the query params, then send back only a fraction of the data.

Hmm maybe I should just fetch everything from the DB then.

[fusion44]

LND has an astronomical high CPU usage too. So that's a systemic issue I didn't notice on my dev PC as it is quite a bit faster than a PI.

edit As soon as I close the WebUI browser window and the polling to /list-all-tx stops, the CPU usage stays high. This indicates to /list-all-tx not being the culprit here.

[openoms]

Should find a way to not keep calling functions which return an error. Agree @cstenglein should slow down significantly. Max every 20s as we are still building on the RPi as the main platform. And to be clear this polling only goes on when the webUI is open, right?

[fusion44]

Yes, the polling in only going on when a browser tab with the WebUI is open. While I agree that we must get away from polling, I actually believe now that the polling is not the root cause of the problem.

I've tested a bit more tonight and the problem persists even as the polling stops.

edit In fact, I've just experienced this problem with no WebUI running at all. It is one of my background tasks which is polling to often. As soon as I kill the API process the CPU usage is returning to normal.

[fusion44]

@openoms Have you tried to let the node run for a bit? CPU usage tends to get down to normal levels from time to time.

[openoms]

@openoms Have you tried to let the node run for a bit? CPU usage tends to get down to normal levels from time to time.

this https://github.com/fusion44/blitz_api/issues/125#issue-1277958308 looks like that any sent payment record is causing a JSONDecodeError and often leading to this line

E0625 21:44:06.980009804 1907979 fork_posix.cc:76]           Other threads are currently calling into gRPC, skipping fork() handlers

should be rate limited.

[openoms]

Testing Open channel with CLN:

needed to close the WebUI to calm it down.