ssvlabs
Increased Bandwidth Usage Since dencun Upgrade
Describe the bug Since the recent upgrade of dencun, I have noticed a significant increase in bandwidth usage for the ssv-node program I am running. The bandwidth peaks at 70Mbit/s, with the majority being consumed by the 13001 p2p port.
To Reproduce Use iftop -P to observe bandwidth usage. Wait for a while and check the total peak.
Expected behavior I expect the bandwidth usage to be at the previous level, rather than the current 70Mbit/s.
Node (please complete the following information):
- Operator Version: v1.3.2
- BN Client: Prysm, Lighthouse
- Cloud service provider: Huawei Cloud
Additional context I have already tried restarting the program and the system, but the issue persists.
Yes, I found the same problem on my production environment.After the 3.13 upgrade, the P2P traffic is almost doubled, resulting in severe packet loss. The performance of ssv is also much lower than before. I think this is a general question.need to be taken seriously. @y0sher
Thanks for letting us know, we're investigating this right now.
@haoei Can you please confirm that its solely the SSV node that is causing all of this traffic (70Mbit/s), and it's not a combined traffic of, for example, the SSV + CL + EL nodes together? This is at least an order of magnitude above what we've seen so far.
@hwhe @lilkk-jerry Can you please provide the network usage of one of your SSV nodes?
We have an upcoming upgrade planned that would reduce traffic tremendously.
With that said, we're now brainstorming what can be done in the short-term.
@moshe-blox
you use iftop, can clearly see that more than 90% of the traffic is occupied by the ssv p2p port. very obvious。 This is the result after I sorted it by traffic。
ecs-8416:13001 <=> 18.237.5.4:13021 106MB 304Kb 2.27Mb 1.89Mb ecs-8416:13001 <=> ns3216045.ip-141-95-202.eu:13001 109MB 314Kb 2.48Mb 1.88Mb ecs-8416:13001 <=> ecs-119-8-172-250.compute.hwclouds-dns.com:13001 94.9MB 293Kb 2.08Mb 1.63Mb ecs-8416:13001 <=> 18.237.5.4:13025 80.8MB 223Kb 2.00Mb 1.47Mb ecs-8416:13001 <=> ecs-124-243-150-192.compute.hwclouds-dns.co:13001 86.3MB 169Kb 1.7578 1.43Mb ecs-8416:13001 <=> 34.116.92.116:13000 81.5MB 111Kb 1.87Mb 1.42Mb ecs-8416:13001 <=> 37.120.190.85:13001 28.4MB 88.0Kb 556Kb 515Kb ecs-8416:13001 <=> ns31440147.ip-141-94-143.eu:13002 3001 3 79813.9MB 9.82Kb 517Kb 392b ecs-8416:13001 <=> 103.106.58.1:13001 net.md:13000 :13001 18.3MB 203Kb 430Kb 339Kb ecs-8416:13001 <=> 185-70-190-197.starnet.md:13000 15.3MB 15.6Kb 305Kb 321Kb ecs-8416:13001 <=> mta27.greendeskservers.com:13001 1.75MB 15.6Kb 321Kb 265Kb ecs-8416:13001 <=> 77.35.209.243.bcube.co.uk:12005 s-dns. m:30073 2.54MB 28.4Kb 447b 140Kb ecs-8416:13001 <=> 75.98.207.226:13001 3Kb 161 7.99MB 12.2Kb 258Kb 175Kb ecs-8416:13001 <=> 161.35.83.205:23101 co.uk:12005 7.83MB 80.8Kb 99.3Kb 169Kb ecs-8416:13001 <=> 5.9.189.245:13001 5.20MB 10.8Kb 162Kb 162Kb ecs-8416:13001 <=> 144.76.30.248:13001 6.43MB 16.7Kb 117Kb 129Kb ecs-8416:13001 <=> 45.139.122.58:13001 s.com:30073 6.43MB 10.9Kb 139Kb 127Kb ecs-8416:13001 <=> host.cloudsigma.net:13001 1.79MB 10.2Kb 165Kb 119Kb ecs-8416:13001 <=> ns3179937.ip-51-210-209.eu:13001 4.85MB 13.2Kb 97.4Kb 89.4Kb ecs-8416:13001 <=> static.180.71.21.65.clients.your-server.de:13001 4.86MB 12.6Kb 99.4Kb 88.1Kb ecs-8416:13001 <=> static.91.114.181.135.clients.your-server.d:13001 758KB 4.93Kb 91.5Kb 65.3Kb ecs-8416:13001 <=> static.207.100.181.135.clients.your-server.:13001 311KB 10.9Kb 86.7Kb 62.1Kb ecs-8416:13001 <=> c-24-60-2-83.hsd1.ma.comcast.net:13001 34.3MB 0b 0b 33.985.1Kb 61.6Kb ecs-8416:13001 <=> 34.146.254.37:13001 13001 3.18MB 0b 0b 22.9Kb ecs-8416:ssh <=> 113.137.58.76:4795 586KB 10.6Kb 10.4Kb 11.7Kb ecs-8416:13001 <=> 161.35.18.85:1300213-20.net:13003 567KB 0b 45.2Kb 11.4Kb ecs-8416:13001 <=> 129.213.159.159:13001 0.net:13003 1.09MB 0b 11.4Kb 8.19Kb ecs-8416:13001 <=> static.152.225.109.65.clients.your-server.d:13001 182KB 6.00Kb 2.40Kb 3.46Kb ecs-8416:13001 <=> static.101.230.21.65.clients.your-server.de:13001 16.4KB 0b 13.1Kb 3.28Kb ecs-8416:58690 <=> ecs-190-92-218-36.compute.hwclouds-dns.com:30073 71.8KB 0b 8.23Kb 3.19Kb ecs-8416:13001 <=> 195.14.6.207:13001 139KB 2.09Kb 3.67Kb 3.02Kb
TX: cum: 369MB peak: 25.3Mb rates: 930Kb 7.92Mb 6.42Mb RX: 386MB 22.2M96 8.8 1.04Mb 8.46Mb 6.65Mb TOTAL: 766MB 54.1Mb 1.94Mb 16.4Mb 13.1Mb
@moshe-blox EL and CL nodes are not running with SSV. They run on separate machines.