[Bug] Tons of "WARNING Rate limiting ourselves. message type: respond_peers" messages in log

[jayhohoho2019]

What happened?

After upgrade to 1.5.1 (from 1.5.0), full node shows a large number of "WARNING Rate limiting ourselves. message type: respond_peers" messages in debug.log. Full node is in sync. Most connected peers are in sync, have even farmed a block since the upgrade, but these warning messages are filling up the log.

Version

1.5.1

What platform are you using?

Linux

What ui mode are you using?

CLI

Relevant log output

2022-08-23T22:56:57.376 full_node full_node_server        : WARNING  Rate limiting ourselves. message type: respond_proof_of_weight, peer: 31.19.126.71
2022-08-23T22:56:58.384 full_node full_node_server        : WARNING  Rate limiting ourselves. message type: respond_proof_of_weight, peer: 31.19.126.71
2022-08-23T22:56:59.391 full_node full_node_server        : WARNING  Rate limiting ourselves. message type: respond_proof_of_weight, peer: 31.19.126.71
2022-08-23T22:57:01.625 full_node full_node_server        : WARNING  Rate limiting ourselves. message type: respond_peers, peer: 190.16.85.253
2022-08-23T22:57:01.626 full_node full_node_server        : WARNING  Rate limiting ourselves. message type: respond_peers, peer: 109.111.131.6

Last 5 lines of log

Since upgrade, 1851 lines of this message type has shown up in log, with a total of 14 unique IPs.

[jayhohoho2019]

2022-08-23T16:59:22.007 full_node full_node_server : WARNING Rate limiting ourselves. message type: respond_peers, peer: 88.26.224.173 2022-08-23T16:59:22.263 full_node full_node_server : WARNING Rate limiting ourselves. message type: respond_peers, peer: 88.26.224.173 2022-08-23T17:04:02.144 full_node full_node_server : WARNING Rate limiting ourselves. message type: respond_peers, peer: 24.96.61.222

First 3 lines in log, to provide a timeframe for the total of 1851 lines of messages.

[jayhohoho2019]

As of now Wed 24 Aug 2022 12:51:47 PM EDT, 4293 lines of this “Rate limiting" warning messages in log. So that's during a 20 hour time span.

[alnuweiri]

The same problem here.

[luckeyca]

I think this bug might be long before 1.5.0 because there were tons of the rate limiting errors in my logs dated on July 31st which was the oldest I could go back.

[jayhohoho2019]

My full node log started 2021-07-07 and this message started 2022-08-23 which was yesterday. As of now messages containing "Rate limiting" has accounted for ~8% of the total debug.log lines.

[jayhohoho2019]

As of now they are 9.89% of the total debug.log lines.

[jayhohoho2019]

14% of total log as of now and still growing. 13951 lines

[zhangyongchang88]

截至目前，占总日志的 14%，并且仍在增长。13951行

能正常爆块吗？

[00linglin]

I had the same issue on one of my harvesters and full node, ended up changing outbound_rate_limit_percent in the config file from 30 to 100 for both the harvester and full node. The messages stopped, not sure how it will affect other things though. I was having huge amounts of stale points for the affected harvester.

[wallentx]

Anyone reporting these problems, can you also share what pool you are with?

[jayhohoho2019]

Anyone reporting these problems, can you also share what pool you are with?

I'm self farming.

[jayhohoho2019]

And as of now 16.0% of debug.log line count.

[TheGreatGooglyMoogly]

Anyone reporting these problems, can you also share what pool you are with?

I have 6 Pool NFTs, 4 are with different pools (Spacepool, XCHpool, Chiahub & Spacefarmers), two are self-pooling & a load of non-NFT plots. In my case its only rate-limiting 2 IPs, both are synced full nodes.

[CEduardoGR]

Anyone reporting these problems, can you also share what pool you are with?

I'm farming solo and I have this problem too. I also have a few of this message but it is far less frequent than the respond_peers message:

full_node full_node_server : WARNING Rate limiting ourselves. message type: respond_end_of_sub_slot, peer:

And also this one (repeated with the node ID of each harvester, several times):

farmer chia.plot_sync.receiver : ERROR reset: node_id ebd9821959799dff28ccf5f5d0efeef9b341444131349a1029678f41dc7df54e, current_sync: [state 0, sync_id 0, next_message_id 0, plots_processed 0, plots_total 0, delta [valid +0/-0, invalid +0/-0, keys missing: +0/-0, duplicates: +0/-0], time_done None]

The last message has its own topic already.

The warnings and the error came with version 1.5.1. I took another look in the archived logs to see if the errors where there and the frequency they occurred before but couldn't find them, it is pretty clear to me that 1.5.1 is the culprit for introducing all this problems.

[keliew]

Version: 1.5.1 OG solo, NFT pooling.

inbound_rate_limit_percent: 80
outbound_rate_limit_percent: 80

$ grep 'Rate limiting ourselves' ~/.chia/mainnet/log/debug.log | wc -l
203

My other node also has a same setup, but...

 $ grep 'Rate limiting ourselves' ~/.chia/mainnet/log/debug.log | wc -l
0

Both nodes have more or less the same amount of peers (>50).

[wallentx]

Thanks for the feedback. It seems this does seem to be impacting a spread of people. What I've found so far may give some hint as to what's happening. From https://github.com/Chia-Network/chia-blockchain/pull/11636 which got pulled into both main and 1.5.1:

• Adds a capability RATE_LIMITS_V2 to signal that the node supports the new, more permissive rate limits. This allows wallets to make more requests per second, since the backend has been optimized (in the other PRs). • The lowest compatible rate limit that both peers support is used

I won't speculate as to what is happening beyond guessing that this might be peer dependent.

[skweee]

Before 1.5.1 this may have been already happening but was most likely invisible for most users. A diff between 1.5.1 and 1.5.0 reveals that this message changed severity:

• up to 1.5.0 and including: DEBUG
• starting with 1.5.1: WARNING

This is the relevant commit, which was first released in 1.5.1: https://github.com/Chia-Network/chia-blockchain/commit/ab53d489955863e7fc03caef69b29b09b0184784

[bhorvitz]

I am self-pooling and also seeing lots of these. FWIW, the nodes that I am connected to that are reporting rate limiting are full nodes, not light wallets. get_connections shows them as "type": 1. I believe that a wallet shows up as type 6? I might be misinterpreting that.

[jayhohoho2019]

21734 lines. 20.25% of my debug.log line count as of now. Again this is my full node/wallet/farmer.

[jayhohoho2019]

Version: 1.5.1 OG solo, NFT pooling.

inbound_rate_limit_percent: 80
outbound_rate_limit_percent: 80

$ grep 'Rate limiting ourselves' ~/.chia/mainnet/log/debug.log | wc -l
203

My other node also has a same setup, but...

 $ grep 'Rate limiting ourselves' ~/.chia/mainnet/log/debug.log | wc -l
0

Both nodes have more or less the same amount of peers (>50).

What do these rate_limits mean? Since it is a warning now in v1.5.1 and an error prior to it, it is possible the issue has been there, just not logged. But is it an issue that I need to take care of on my full node/farmer/wallet? Could you advise?

Other than that, the fact that it's filling up the log so fast is an issue I suppose.

[LudmilaAlex]

Windows 11 Version 1.5.1 | 64 bit pooling solo NFT,CLI 08/28/2022 11:50:46.829 updated from v1.5.0 to v1.5.1 From 08/28/2022 11:50:46.829 to 08/30/2022 06:13:43 Total events from logger Full_Node 69520 Including 15879 “WARNING” events, including 15826 “WARNING Rate limiting ourselves. message type: respond_peers, peer:,,," This is 15826/15879=99.67%

Screenshot1 of the LogViewPlus window: https://mega.nz/file/NxQjyLra#LBbijLBwjMznCu_t6yYD1wEsWvVKqwpcg-tHxeq11Y8 (This is not a github link, this is a mega.nz link)

Screenshot2 of the LogViewPlus window. [https://mega.nz/file/Qk5VnTyA#TKBzjYMyVN0VZFUsBfRsa8yr79gjK9EXuxQ3JC-dM9w (This is not a github link, this is a mega.nz link)

Before (Windows 11, Version 1.5.0 | 64 bit pooling solo NFT,CL) this was not even close!

[jayhohoho2019]

23.9% of debug.log line count.

[jayhohoho2019]

33.04% of debug.log line count now.

[Sinkevi4]

windows 10/64 v1.5.1 GUI new problem for v1.5.0

2022-09-03T00:04:01.925 full_node full_node_server        : WARNING  Rate limiting ourselves. message type: respond_peers, peer: 104.48.169.185
2022-09-03T00:05:02.232 full_node full_node_server        : WARNING  Rate limiting ourselves. message type: respond_peers, peer: 71.231.207.161
2022-09-03T00:05:02.234 full_node full_node_server        : WARNING  Rate limiting ourselves. message type: respond_peers, peer: 104.48.169.185
2022-09-03T00:05:02.238 full_node full_node_server        : WARNING  Rate limiting ourselves. message type: respond_peers, peer: 71.231.207.161
2022-09-03T00:05:02.239 full_node full_node_server        : WARNING  Rate limiting ourselves. message type: respond_peers, peer: 71.231.207.161

.>90% of log

[jayhohoho2019]

41.4% of log

[jayhohoho2019]

49.2% of log. Also for the last half day or so my sync'ed peer count has remained under 10. Normally it's above 40. I'm restarting chia full node.

[Motophan]

Hi please fix this

[Wontell]

Im self pooling and have the same issue, i had 300K log just with this, Maybe its a coincidence but the moment i "let the CPU breath" it stopped and 24 hours ago i deleted the log and since than tehre is no new one. Im on 12700 [non K] in windows to save power i use power saver mode on the Chia PC. Yesterday i enabled normal mode and deleted the log, so far no new one. No I re-enabled the power saver power profile will see tomorrow if the error repeats

[Sinkevi4]

hi.I use maximum performance power mode all the time. Over the past week, the number of messages has decreased to <> 10%. 19.09.2022, 03:33, "Wontell" @.>: Im self pooling and have the same issue, i had 300K log just with this,Maybe its a coincidence but the moment i "let the CPU breath" it stopped and 24 hours ago i deleted the log and since than tehre is no new one.Im on 12700 [non K] in windows to save power i use power saver mode on the Chia PC.Yesterday i enabled normal mode and deleted the log, so far no new one.No I re-enabled the power saver power profile will see tomorrow if the error repeats—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.>

[Kikicaj]

It's been one month since, its already 1.6 and the same "error" message appears even though everything works fine. How come its not fixed yet?