Unusual CPU usage spike after certain pps count?

Open ozschmoz opened this issue 2 years ago • 1 comments

I have been testing ipt-netflow connection on my Intel(R) Xeon(R) Gold 6330 x 2 CPU (hyperthreading off). I noticed a weird growth in CPU usage after a certain pps threshold is reached. Everything seems to be working correctly until I reach about 7-8mpps after which my CPU cores spike to 100% and start dropping packets. Up until 7mpps the CPU usage is gradual based on mpps count but after passing this threshold it starts spiking up to 100%. Without having NETFLOW rules in my iptables I am able to reach 20mpps throughput without experiencing any packet loss. For testing, I was using t-rex udp 64 byte test. I have tried ubuntu 20.04 and Debian 11 with the same effect. Is this expected behavior with this kernel module?

I am using Ubuntu 22.04.3 LTS 5.15.0-79-generic. Intel ethernet CNA XL710-QDA2 40G NIC.

`all cores 0,7% (1mpps):

ipt_NETFLOW 2.6-17-g0eb2092, Protocol version 10 Timeouts: active 120s, Flows: active 20215 Hash: size 655360 Rate: 376848442 bits/sec, cpu# pps; <search Total 1024000; 125446 8560074 cpu0 19662; 1290 164343 cpu1 19917; 2142 166510 cpu2 19358; 3856 161786 cpu3 19765; 1714 165226 cpu4 19917; 1719 166512 cpu5 20632; 2573 172501 cpu6 19250; 1720 160925 cpu7 21299; 2144 178033 cpu8 18992; 3002 158784 cpu9 17713; 1289 148077 cpu10 16487; 9 137818 cpu11 18274; 860 152788 cpu12 18178; 4717 151950 cpu13 17664; 1289 147678 cpu14 17564; 2251 146814 cpu15 17660; 2147 147650 cpu16 17818; 1291 148946 cpu17 18484; 2576 154509 cpu18 19300; 3002 161360 cpu19 18635; 431 155776 cpu20 18023; 2573 150647 cpu21 17715; 4712 148102 cpu22 18123; 863 151511 cpu23 18897; 2145 157943 cpu24 18789; 3002 157082 cpu25 18331; 3002 153215 cpu26 19760; 1719 165199 cpu27 18232; 3004 152384 cpu28 18535; 2574 154957 cpu29 19303; 3429 161386 cpu30 19810; 2143 165642 cpu31 16644; 2145 139099 cpu32 17765; 2143 148515 cpu33 17255; 1719 144240 cpu34 17404; 3856 145522 cpu35 16283; 4284 136110 cpu36 17401; 3433 145522 cpu37 16490; 2575 137826 cpu38 17969; 3001 150221 cpu39 17613; 1716 147229 cpu40 17103; 2572 142952 cpu41 18739; 2145 156655 cpu42 17664; 2143 147655 cpu43 17767; 1290 148505 cpu44 18074; 1717 151078 cpu45 16948; 2148 141677 cpu46 16233; 1722 135676 cpu47 18124; 1290 151509 cpu48 18075; 1715 151071 cpu49 17869; 1290 149368 cpu50 18638; 3429 155799 cpu51 18077; 2199 151066 cpu52 17306; 2149 144653 cpu53 19405; 2573 162210 cpu54 20375; 1287 170344 cpu55 16692; 1717 139532 Export: Rate 697 bytes/s; sock0: 192.168.0.1:2504, srcversion 9D57791DE7AEF9589512DAB; llist (ipfix), refresh-rate 20, timeout-rate 30, (templates 5, active 5). inactive 15s. Maxflows 4000000 (peak 20217 reached 0d0h0m ago), mem 7962K, worker delay 25/250 [1..25] (28 ms, 0 us, 254:2 0 [cpu44]). (mem 5120K), metric 1.01 [1.00, 1.00, 1.00]. InHash: 8579869 pkt, 385442 K, InPDU 20, 869. 1024022 packets/sec; Avg 1 min: 82374217 bps, 223823 pps; 5 min: 17549765 bps, 47677 pps found new [metric], trunc frag alloc maxflows>, traffic: <pkt, bytes>, drop: <pkt, bytes> 20794 [1.01], 0 0 0 0, traffic: 8580868, 376 MB, drop: 0, 0 K 404 [1.00], 0 0 0 0, traffic: 164747, 7 MB, drop: 0, 0 K 405 [1.00], 0 0 0 0, traffic: 166915, 7 MB, drop: 0, 0 K 389 [1.00], 0 0 0 0, traffic: 162175, 7 MB, drop: 0, 0 K 401 [1.00], 0 0 0 0, traffic: 165627, 7 MB, drop: 0, 0 K 410 [1.00], 0 0 0 0, traffic: 166922, 7 MB, drop: 0, 0 K 418 [1.00], 0 0 0 0, traffic: 172919, 7 MB, drop: 0, 0 K 387 [1.00], 0 0 0 0, traffic: 161312, 7 MB, drop: 0, 0 K 436 [1.00], 0 0 0 0, traffic: 178469, 7 MB, drop: 0, 0 K 379 [1.00], 0 0 0 0, traffic: 159163, 6 MB, drop: 0, 0 K 359 [1.00], 0 0 0 0, traffic: 148436, 6 MB, drop: 0, 0 K 341 [1.00], 0 0 0 0, traffic: 138159, 6 MB, drop: 0, 0 K 368 [1.00], 0 0 0 0, traffic: 153156, 6 MB, drop: 0, 0 K 370 [1.00], 0 0 0 0, traffic: 152320, 6 MB, drop: 0, 0 K 366 [1.00], 0 0 0 0, traffic: 148044, 6 MB, drop: 0, 0 K 362 [1.00], 0 0 0 0, traffic: 147176, 6 MB, drop: 0, 0 K 359 [1.00], 0 0 0 0, traffic: 148009, 6 MB, drop: 0, 0 K 368 [1.00], 0 0 0 0, traffic: 149314, 6 MB, drop: 0, 0 K 372 [1.00], 0 0 0 0, traffic: 154881, 6 MB, drop: 0, 0 K 391 [1.00], 0 0 0 0, traffic: 161751, 7 MB, drop: 0, 0 K 377 [1.00], 0 0 0 0, traffic: 156153, 6 MB, drop: 0, 0 K 365 [1.00], 0 0 0 0, traffic: 151012, 6 MB, drop: 0, 0 K 357 [1.00], 0 0 0 0, traffic: 148459, 6 MB, drop: 0, 0 K 367 [1.00], 0 0 0 0, traffic: 151878, 6 MB, drop: 0, 0 K 389 [1.00], 0 0 0 0, traffic: 158332, 6 MB, drop: 0, 0 K 388 [1.00], 0 0 0 0, traffic: 157470, 6 MB, drop: 0, 0 K 369 [1.00], 0 0 0 0, traffic: 153584, 6 MB, drop: 0, 0 K 397 [1.00], 0 0 0 0, traffic: 165596, 7 MB, drop: 0, 0 K 368 [1.00], 0 0 0 0, traffic: 152752, 6 MB, drop: 0, 0 K 377 [1.00], 0 0 0 0, traffic: 155334, 6 MB, drop: 0, 0 K 387 [1.00], 0 0 0 0, traffic: 161773, 7 MB, drop: 0, 0 K 401 [1.00], 0 0 0 0, traffic: 166043, 7 MB, drop: 0, 0 K 348 [1.00], 0 0 0 0, traffic: 139447, 6 MB, drop: 0, 0 K 366 [1.00], 0 0 0 0, traffic: 148881, 6 MB, drop: 0, 0 K 359 [1.00], 0 0 0 0, traffic: 144599, 6 MB, drop: 0, 0 K 354 [1.00], 0 0 0 0, traffic: 145876, 6 MB, drop: 0, 0 K 331 [1.00], 0 0 0 0, traffic: 136441, 5 MB, drop: 0, 0 K 356 [1.00], 0 0 0 0, traffic: 145878, 6 MB, drop: 0, 0 K 335 [1.00], 0 0 0 0, traffic: 138161, 6 MB, drop: 0, 0 K 363 [1.00], 0 0 0 0, traffic: 150584, 6 MB, drop: 0, 0 K 354 [1.00], 0 0 0 0, traffic: 147583, 6 MB, drop: 0, 0 K 342 [1.00], 0 0 0 0, traffic: 143294, 6 MB, drop: 0, 0 K 382 [1.00], 0 0 0 0, traffic: 157037, 6 MB, drop: 0, 0 K 353 [1.00], 0 0 0 0, traffic: 148008, 6 MB, drop: 0, 0 K 355 [1.00], 0 0 0 0, traffic: 148860, 6 MB, drop: 0, 0 K 362 [1.00], 0 0 0 0, traffic: 151440, 6 MB, drop: 0, 0 K 349 [1.00], 0 0 0 0, traffic: 142026, 6 MB, drop: 0, 0 K 332 [1.00], 0 0 0 0, traffic: 136008, 5 MB, drop: 0, 0 K 365 [1.00], 0 0 0 0, traffic: 151874, 6 MB, drop: 0, 0 K 361 [1.00], 0 0 0 0, traffic: 151432, 6 MB, drop: 0, 0 K 363 [1.00], 0 0 0 0, traffic: 149731, 6 MB, drop: 0, 0 K 381 [1.00], 0 0 0 0, traffic: 156180, 6 MB, drop: 0, 0 K 366 [1.00], 0 0 0 0, traffic: 151432, 6 MB, drop: 0, 0 K 351 [1.00], 0 0 0 0, traffic: 145004, 6 MB, drop: 0, 0 K 395 [1.00], 0 0 0 0, traffic: 162605, 7 MB, drop: 0, 0 K 409 [1.00], 0 0 0 0, traffic: 170753, 7 MB, drop: 0, 0 K 335 [1.00], 0 0 0 0, traffic: 139867, 6 MB, drop: 0, 0 K Total 22 pkts, 0 MB, 560 flows; Errors 0 pkts; Traffic lost 0 pkts, 0 Kbytes, 0 flows. sndbuf 18000000, filled 1, peak 2305; err: sndbuf reached 0, connect 0, cberr 0, other 0

all cores 15% (7mpps):

ipt_NETFLOW 2.6-17-g0eb2092, srcversion 9D57791DE7AEF9589512DAB; llist Protocol version 10 (ipfix), refresh-rate 20, timeout-rate 30, (templates 5, active 5). Timeouts: active 120s, inactive 15s. Maxflows 4000000 Flows: active 20370 (peak 20419 reached 0d0h0m ago), mem 7984K, worker delay 25/250 [1..25] (44 ms, 0 us, 253:5 0 [cpu41]). Hash: size 655360 (mem 5120K), metric 0.48 [0.50, 0.88, 0.94]. InHash: 111709794 pkt, 5018388 K, InPDU 50, 4517. Rate: 2637627392 bits/sec, 7167326 packets/sec; Avg 1 min: 973599742 bps, 2645564 pps; 5 min: 224756381 bps, 610726 pps cpu# pps; <search found new [metric], trunc frag alloc maxflows>, traffic: <pkt, bytes>, drop: <pkt, bytes> Total 7167303; 1655937 111689632 20624 [1.01], 0 0 0 0, traffic: 111710255, 4900 MB, drop: 0, 0 K cpu0 137609; 16763 2144435 397 [1.00], 0 0 0 0, traffic: 2144832, 94 MB, drop: 0, 0 K cpu1 139401; 33510 2172343 407 [1.00], 0 0 0 0, traffic: 2172750, 95 MB, drop: 0, 0 K cpu2 135459; 49660 2110900 395 [1.00], 0 0 0 0, traffic: 2111295, 92 MB, drop: 0, 0 K cpu3 138329; 27913 2155628 395 [1.00], 0 0 0 0, traffic: 2156023, 94 MB, drop: 0, 0 K cpu4 139396; 27931 2172352 401 [1.00], 0 0 0 0, traffic: 2172753, 95 MB, drop: 0, 0 K cpu5 144420; 33516 2250520 415 [1.00], 0 0 0 0, traffic: 2250935, 98 MB, drop: 0, 0 K cpu6 134745; 33534 2099760 386 [1.00], 0 0 0 0, traffic: 2100146, 92 MB, drop: 0, 0 K cpu7 149075; 22345 2323148 426 [1.00], 0 0 0 0, traffic: 2323574, 101 MB, drop: 0, 0 K cpu8 132952; 39094 2071879 378 [1.00], 0 0 0 0, traffic: 2072257, 90 MB, drop: 0, 0 K cpu9 123990; 27917 1932209 357 [1.00], 0 0 0 0, traffic: 1932566, 84 MB, drop: 0, 0 K cpu10 115400; 22346 1798277 333 [1.00], 0 0 0 0, traffic: 1798610, 78 MB, drop: 0, 0 K cpu11 127932; 11177 1993689 369 [1.00], 0 0 0 0, traffic: 1994058, 87 MB, drop: 0, 0 K cpu12 127222; 50275 1982559 372 [1.00], 0 0 0 0, traffic: 1982931, 86 MB, drop: 0, 0 K cpu13 123638; 27922 1926657 356 [1.00], 0 0 0 0, traffic: 1927013, 84 MB, drop: 0, 0 K cpu14 122928; 27928 1915546 359 [1.00], 0 0 0 0, traffic: 1915905, 84 MB, drop: 0, 0 K cpu15 123633; 40110 1926653 355 [1.00], 0 0 0 0, traffic: 1927008, 84 MB, drop: 0, 0 K cpu16 124709; 27925 1943393 354 [1.00], 0 0 0 0, traffic: 1943747, 85 MB, drop: 0, 0 K cpu17 129369; 39089 2016005 369 [1.00], 0 0 0 0, traffic: 2016374, 88 MB, drop: 0, 0 K cpu18 135096; 50264 2105333 388 [1.00], 0 0 0 0, traffic: 2105721, 92 MB, drop: 0, 0 K cpu19 130442; 5588 2032707 374 [1.00], 0 0 0 0, traffic: 2033081, 89 MB, drop: 0, 0 K cpu20 126144; 39098 1965729 369 [1.00], 0 0 0 0, traffic: 1966098, 86 MB, drop: 0, 0 K cpu21 123987; 55835 1932163 357 [1.00], 0 0 0 0, traffic: 1932520, 84 MB, drop: 0, 0 K cpu22 126856; 11749 1976887 367 [1.00], 0 0 0 0, traffic: 1977254, 86 MB, drop: 0, 0 K cpu23 132234; 27924 2060667 378 [1.00], 0 0 0 0, traffic: 2061045, 90 MB, drop: 0, 0 K cpu24 131515; 47120 2049493 381 [1.00], 0 0 0 0, traffic: 2049874, 89 MB, drop: 0, 0 K cpu25 128293; 27927 1999289 369 [1.00], 0 0 0 0, traffic: 1999658, 87 MB, drop: 0, 0 K cpu26 138326; 16751 2155656 400 [1.00], 0 0 0 0, traffic: 2156056, 94 MB, drop: 0, 0 K cpu27 127583; 39104 1988125 372 [1.00], 0 0 0 0, traffic: 1988497, 87 MB, drop: 0, 0 K cpu28 129732; 27932 2021608 371 [1.00], 0 0 0 0, traffic: 2021979, 88 MB, drop: 0, 0 K cpu29 135102; 28698 2105378 380 [1.00], 0 0 0 0, traffic: 2105758, 92 MB, drop: 0, 0 K cpu30 138685; 16755 2161194 392 [1.00], 0 0 0 0, traffic: 2161586, 94 MB, drop: 0, 0 K cpu31 116473; 32967 1814978 336 [1.00], 0 0 0 0, traffic: 1815314, 79 MB, drop: 0, 0 K cpu32 124350; 27928 1937835 356 [1.00], 0 0 0 0, traffic: 1938191, 85 MB, drop: 0, 0 K cpu33 120775; 22347 1881943 349 [1.00], 0 0 0 0, traffic: 1882292, 82 MB, drop: 0, 0 K cpu34 121845; 51149 1898729 346 [1.00], 0 0 0 0, traffic: 1899075, 83 MB, drop: 0, 0 K cpu35 113965; 54063 1775933 328 [1.00], 0 0 0 0, traffic: 1776261, 77 MB, drop: 0, 0 K cpu36 121839; 44685 1898774 349 [1.00], 0 0 0 0, traffic: 1899123, 83 MB, drop: 0, 0 K cpu37 115394; 27928 1798196 329 [1.00], 0 0 0 0, traffic: 1798525, 78 MB, drop: 0, 0 K cpu38 125790; 27932 1960168 361 [1.00], 0 0 0 0, traffic: 1960529, 86 MB, drop: 0, 0 K cpu39 123279; 22343 1921080 354 [1.00], 0 0 0 0, traffic: 1921434, 84 MB, drop: 0, 0 K cpu40 119703; 22335 1865287 343 [1.00], 0 0 0 0, traffic: 1865630, 81 MB, drop: 0, 0 K cpu41 131164; 27926 2043922 374 [1.00], 0 0 0 0, traffic: 2044296, 89 MB, drop: 0, 0 K cpu42 123640; 27929 1926737 359 [1.00], 0 0 0 0, traffic: 1927096, 84 MB, drop: 0, 0 K cpu43 124351; 11178 1937862 364 [1.00], 0 0 0 0, traffic: 1938226, 85 MB, drop: 0, 0 K cpu44 126501; 16761 1971353 361 [1.00], 0 0 0 0, traffic: 1971714, 86 MB, drop: 0, 0 K cpu45 118624; 22343 1848561 345 [1.00], 0 0 0 0, traffic: 1848906, 81 MB, drop: 0, 0 K cpu46 113607; 30591 1770338 330 [1.00], 0 0 0 0, traffic: 1770668, 77 MB, drop: 0, 0 K cpu47 126864; 16756 1976986 367 [1.00], 0 0 0 0, traffic: 1977353, 86 MB, drop: 0, 0 K cpu48 126505; 23762 1971340 366 [1.00], 0 0 0 0, traffic: 1971706, 86 MB, drop: 0, 0 K cpu49 125073; 22347 1949027 365 [1.00], 0 0 0 0, traffic: 1949392, 85 MB, drop: 0, 0 K cpu50 130443; 44672 2032777 374 [1.00], 0 0 0 0, traffic: 2033151, 89 MB, drop: 0, 0 K cpu51 126505; 27932 1971283 362 [1.00], 0 0 0 0, traffic: 1971645, 86 MB, drop: 0, 0 K cpu52 121128; 33514 1887526 349 [1.00], 0 0 0 0, traffic: 1887875, 82 MB, drop: 0, 0 K cpu53 135821; 29337 2116580 388 [1.00], 0 0 0 0, traffic: 2116968, 92 MB, drop: 0, 0 K cpu54 142634; 11174 2222646 411 [1.00], 0 0 0 0, traffic: 2223057, 97 MB, drop: 0, 0 K cpu55 116828; 22344 1820532 336 [1.00], 0 0 0 0, traffic: 1820868, 79 MB, drop: 0, 0 K Export: Rate 1721 bytes/s; Total 10 pkts, 0 MB, 233 flows; Errors 0 pkts; Traffic lost 0 pkts, 0 Kbytes, 0 flows. sock0: 192.168.0.1:2504, sndbuf 18000000, filled 1, peak 2305; err: sndbuf reached 0, connect 0, cberr 0, other 0

half cores at 100%, the other half at 30% (8mpps):

ipt_NETFLOW 2.6-17-g0eb2092, srcversion 9D57791DE7AEF9589512DAB; llist Protocol version 10 (ipfix), refresh-rate 20, timeout-rate 30, (templates 5, active 5). Timeouts: active 120s, inactive 15s. Maxflows 4000000 Flows: active 20197 (peak 20255 reached 0d0h0m ago), mem 7960K, worker delay 25/250 [1..25] (76 ms, 0 us, 255:2 0 [cpu43]). Hash: size 655360 (mem 5120K), metric 0.38 [0.30, 0.80, 0.90]. InHash: 180307835 pkt, 8099792 K, InPDU 56, 6386. Rate: 2819165190 bits/sec, 7660743 packets/sec; Avg 1 min: 1349304259 bps, 3666557 pps; 5 min: 332722049 bps, 904124 pps cpu# pps; <search found new [metric], trunc frag alloc maxflows>, traffic: <pkt, bytes>, drop: <pkt, bytes> Total 7660722; 2714059 180288109 20477 [1.01], 0 0 0 0, traffic: 180308586, 7910 MB, drop: 0, 0 K cpu0 125470; 15992 2951815 396 [1.00], 0 0 0 0, traffic: 2952211, 129 MB, drop: 0, 0 K cpu1 130356; 29796 3069156 403 [1.00], 0 0 0 0, traffic: 3069559, 134 MB, drop: 0, 0 K cpu2 130677; 50049 3077880 389 [1.00], 0 0 0 0, traffic: 3078269, 135 MB, drop: 0, 0 K cpu3 130827; 33149 3080824 398 [1.00], 0 0 0 0, traffic: 3081222, 135 MB, drop: 0, 0 K cpu4 131926; 17097 3107064 401 [1.00], 0 0 0 0, traffic: 3107465, 136 MB, drop: 0, 0 K cpu5 131110; 46912 3085030 416 [1.00], 0 0 0 0, traffic: 3085446, 135 MB, drop: 0, 0 K cpu6 131312; 43413 3087425 382 [1.00], 0 0 0 0, traffic: 3087807, 135 MB, drop: 0, 0 K cpu7 131642; 29552 3097370 424 [1.00], 0 0 0 0, traffic: 3097794, 135 MB, drop: 0, 0 K cpu8 130681; 51051 3078611 378 [1.00], 0 0 0 0, traffic: 3078989, 135 MB, drop: 0, 0 K cpu9 131538; 31260 3092493 355 [1.00], 0 0 0 0, traffic: 3092848, 135 MB, drop: 0, 0 K cpu10 131592; 9 3099198 331 [1.00], 0 0 0 0, traffic: 3099529, 135 MB, drop: 0, 0 K cpu11 130048; 17368 3059309 359 [1.00], 0 0 0 0, traffic: 3059668, 134 MB, drop: 0, 0 K cpu12 131652; 96602 3100279 363 [1.00], 0 0 0 0, traffic: 3100642, 136 MB, drop: 0, 0 K cpu13 131196; 28041 3089176 353 [1.00], 0 0 0 0, traffic: 3089529, 135 MB, drop: 0, 0 K cpu14 130387; 37699 3068875 352 [1.00], 0 0 0 0, traffic: 3069227, 134 MB, drop: 0, 0 K cpu15 129831; 53745 3057473 352 [1.00], 0 0 0 0, traffic: 3057825, 134 MB, drop: 0, 0 K cpu16 130901; 26850 3079824 352 [1.00], 0 0 0 0, traffic: 3080176, 135 MB, drop: 0, 0 K cpu17 131710; 60321 3095170 369 [1.00], 0 0 0 0, traffic: 3095539, 135 MB, drop: 0, 0 K cpu18 129875; 48194 3056526 385 [1.00], 0 0 0 0, traffic: 3056911, 134 MB, drop: 0, 0 K cpu19 131849; 7315 3101265 372 [1.00], 0 0 0 0, traffic: 3101637, 136 MB, drop: 0, 0 K cpu20 131549; 52307 3092935 356 [1.00], 0 0 0 0, traffic: 3093291, 135 MB, drop: 0, 0 K cpu21 130722; 97391 3074464 355 [1.00], 0 0 0 0, traffic: 3074819, 134 MB, drop: 0, 0 K cpu22 131043; 24519 3084804 364 [1.00], 0 0 0 0, traffic: 3085168, 135 MB, drop: 0, 0 K cpu23 131353; 41452 3095084 380 [1.00], 0 0 0 0, traffic: 3095464, 135 MB, drop: 0, 0 K cpu24 131326; 65079 3091043 375 [1.00], 0 0 0 0, traffic: 3091418, 135 MB, drop: 0, 0 K cpu25 131655; 25441 3100416 365 [1.00], 0 0 0 0, traffic: 3100781, 136 MB, drop: 0, 0 K cpu26 131247; 40497 3089246 394 [1.00], 0 0 0 0, traffic: 3089640, 135 MB, drop: 0, 0 K cpu27 130856; 61427 3080837 362 [1.00], 0 0 0 0, traffic: 3081199, 135 MB, drop: 0, 0 K cpu28 148277; 57843 3489207 368 [1.00], 0 0 0 0, traffic: 3489575, 153 MB, drop: 0, 0 K cpu29 154417; 57836 3633775 382 [1.00], 0 0 0 0, traffic: 3634157, 159 MB, drop: 0, 0 K cpu30 158510; 57924 3730184 395 [1.00], 0 0 0 0, traffic: 3730579, 163 MB, drop: 0, 0 K cpu31 133119; 38546 3132616 336 [1.00], 0 0 0 0, traffic: 3132952, 137 MB, drop: 0, 0 K cpu32 142128; 65646 3344607 356 [1.00], 0 0 0 0, traffic: 3344963, 146 MB, drop: 0, 0 K cpu33 138033; 28927 3248166 348 [1.00], 0 0 0 0, traffic: 3248514, 142 MB, drop: 0, 0 K cpu34 139262; 96391 3277150 353 [1.00], 0 0 0 0, traffic: 3277503, 143 MB, drop: 0, 0 K cpu35 130255; 86753 3065205 321 [1.00], 0 0 0 0, traffic: 3065526, 134 MB, drop: 0, 0 K cpu36 139266; 67477 3277220 348 [1.00], 0 0 0 0, traffic: 3277568, 143 MB, drop: 0, 0 K cpu37 131885; 63852 3103616 331 [1.00], 0 0 0 0, traffic: 3103947, 136 MB, drop: 0, 0 K cpu38 143772; 86763 3383158 364 [1.00], 0 0 0 0, traffic: 3383522, 148 MB, drop: 0, 0 K cpu39 140897; 49115 3315690 354 [1.00], 0 0 0 0, traffic: 3316044, 145 MB, drop: 0, 0 K cpu40 136808; 67458 3219421 341 [1.00], 0 0 0 0, traffic: 3219762, 141 MB, drop: 0, 0 K cpu41 149921; 48193 3527748 371 [1.00], 0 0 0 0, traffic: 3528119, 154 MB, drop: 0, 0 K cpu42 141312; 48201 3325364 350 [1.00], 0 0 0 0, traffic: 3325714, 145 MB, drop: 0, 0 K cpu43 142124; 38555 3344631 357 [1.00], 0 0 0 0, traffic: 3344988, 146 MB, drop: 0, 0 K cpu44 144583; 38563 3402475 362 [1.00], 0 0 0 0, traffic: 3402837, 149 MB, drop: 0, 0 K cpu45 135580; 57835 3190531 343 [1.00], 0 0 0 0, traffic: 3190874, 139 MB, drop: 0, 0 K cpu46 129840; 68920 3055462 327 [1.00], 0 0 0 0, traffic: 3055789, 134 MB, drop: 0, 0 K cpu47 144998; 28915 3412143 361 [1.00], 0 0 0 0, traffic: 3412504, 149 MB, drop: 0, 0 K cpu48 144589; 51758 3402387 362 [1.00], 0 0 0 0, traffic: 3402749, 149 MB, drop: 0, 0 K cpu49 142943; 38565 3363945 356 [1.00], 0 0 0 0, traffic: 3364301, 147 MB, drop: 0, 0 K cpu50 149089; 96387 3508489 377 [1.00], 0 0 0 0, traffic: 3508866, 153 MB, drop: 0, 0 K cpu51 144580; 57847 3402338 358 [1.00], 0 0 0 0, traffic: 3402696, 149 MB, drop: 0, 0 K cpu52 138434; 57960 3257794 342 [1.00], 0 0 0 0, traffic: 3258136, 142 MB, drop: 0, 0 K cpu53 155231; 57819 3653121 394 [1.00], 0 0 0 0, traffic: 3653515, 160 MB, drop: 0, 0 K cpu54 163018; 28922 3836182 406 [1.00], 0 0 0 0, traffic: 3836588, 168 MB, drop: 0, 0 K cpu55 133520; 38563 3142179 333 [1.00], 0 0 0 0, traffic: 3142512, 137 MB, drop: 0, 0 K Export: Rate 690 bytes/s; Total 11 pkts, 0 MB, 260 flows; Errors 0 pkts; Traffic lost 0 pkts, 0 Kbytes, 0 flows. sock0:192.168.0.1:2504, sndbuf 18000000, filled 1, peak 2305; err: sndbuf reached 0, connect 0, cberr 0, other 0

all cores at 100% (9mpps):

ipt_NETFLOW 2.6-17-g0eb2092, srcversion 9D57791DE7AEF9589512DAB; llist Protocol version 10 (ipfix), refresh-rate 20, timeout-rate 30, (templates 5, active 5). Timeouts: active 120s, inactive 15s. Maxflows 4000000 Flows: active 20319 (peak 20323 reached 0d0h0m ago), mem 7977K, worker delay 25/250 [1..25] (56 ms, 0 us, 254:6 0 [cpu44]). Hash: size 655360 (mem 5120K), metric 0.18 [0.26, 0.79, 0.90]. InHash: 269500368 pkt, 12106498 K, InPDU 61, 7951. Rate: 3289403676 bits/sec, 8938561 packets/sec; Avg 1 min: 2012599261 bps, 5468980 pps; 5 min: 548642659 bps, 1490859 pps cpu# pps; <search found new [metric], trunc frag alloc maxflows>, traffic: <pkt, bytes>, drop: <pkt, bytes> Total 8938538; 4027126 269480683 20699 [1.01], 0 0 0 0, traffic: 269501382, 11822 MB, drop: 0, 0 K cpu0 158311; 55357 4776612 392 [1.00], 0 0 0 0, traffic: 4777004, 209 MB, drop: 0, 0 K cpu1 159422; 24130 4810066 401 [1.00], 0 0 0 0, traffic: 4810467, 211 MB, drop: 0, 0 K cpu2 159911; 92243 4818798 390 [1.00], 0 0 0 0, traffic: 4819188, 211 MB, drop: 0, 0 K cpu3 158693; 51096 4785777 402 [1.00], 0 0 0 0, traffic: 4786179, 209 MB, drop: 0, 0 K cpu4 160143; 50033 4824918 397 [1.00], 0 0 0 0, traffic: 4825315, 211 MB, drop: 0, 0 K cpu5 159357; 85772 4798810 412 [1.00], 0 0 0 0, traffic: 4799222, 210 MB, drop: 0, 0 K cpu6 161133; 73131 4861913 393 [1.00], 0 0 0 0, traffic: 4862306, 213 MB, drop: 0, 0 K cpu7 161567; 46939 4872439 424 [1.00], 0 0 0 0, traffic: 4872863, 213 MB, drop: 0, 0 K cpu8 158546; 52489 4783646 381 [1.00], 0 0 0 0, traffic: 4784027, 209 MB, drop: 0, 0 K cpu9 159129; 41593 4796358 362 [1.00], 0 0 0 0, traffic: 4796720, 210 MB, drop: 0, 0 K cpu10 148530; 27822 4477820 337 [1.00], 0 0 0 0, traffic: 4478157, 196 MB, drop: 0, 0 K cpu11 157678; 26149 4755980 372 [1.00], 0 0 0 0, traffic: 4756352, 208 MB, drop: 0, 0 K cpu12 160062; 142098 4822924 366 [1.00], 0 0 0 0, traffic: 4823290, 211 MB, drop: 0, 0 K cpu13 158402; 41649 4779861 357 [1.00], 0 0 0 0, traffic: 4780218, 209 MB, drop: 0, 0 K cpu14 158222; 83446 4769818 361 [1.00], 0 0 0 0, traffic: 4770179, 209 MB, drop: 0, 0 K cpu15 157891; 97057 4760158 358 [1.00], 0 0 0 0, traffic: 4760516, 208 MB, drop: 0, 0 K cpu16 159183; 41629 4802687 360 [1.00], 0 0 0 0, traffic: 4803047, 210 MB, drop: 0, 0 K cpu17 159528; 86386 4806900 372 [1.00], 0 0 0 0, traffic: 4807272, 210 MB, drop: 0, 0 K cpu18 157786; 88184 4756313 384 [1.00], 0 0 0 0, traffic: 4756697, 208 MB, drop: 0, 0 K cpu19 156979; 18005 4732470 375 [1.00], 0 0 0 0, traffic: 4732845, 207 MB, drop: 0, 0 K cpu20 161075; 83558 4857165 360 [1.00], 0 0 0 0, traffic: 4857525, 213 MB, drop: 0, 0 K cpu21 159146; 138691 4800093 358 [1.00], 0 0 0 0, traffic: 4800451, 210 MB, drop: 0, 0 K cpu22 158911; 39273 4788411 372 [1.00], 0 0 0 0, traffic: 4788783, 210 MB, drop: 0, 0 K cpu23 158706; 66495 4783067 380 [1.00], 0 0 0 0, traffic: 4783447, 209 MB, drop: 0, 0 K cpu24 158442; 80318 4778635 380 [1.00], 0 0 0 0, traffic: 4779015, 209 MB, drop: 0, 0 K cpu25 157566; 79460 4752664 370 [1.00], 0 0 0 0, traffic: 4753034, 208 MB, drop: 0, 0 K cpu26 159017; 38147 4794939 401 [1.00], 0 0 0 0, traffic: 4795340, 210 MB, drop: 0, 0 K cpu27 158421; 80944 4777996 370 [1.00], 0 0 0 0, traffic: 4778366, 209 MB, drop: 0, 0 K cpu28 166978; 90820 5033990 370 [1.00], 0 0 0 0, traffic: 5034360, 220 MB, drop: 0, 0 K cpu29 173897; 83437 5242570 388 [1.00], 0 0 0 0, traffic: 5242958, 230 MB, drop: 0, 0 K cpu30 178505; 75226 5381625 400 [1.00], 0 0 0 0, traffic: 5382025, 236 MB, drop: 0, 0 K cpu31 149915; 55606 4519509 340 [1.00], 0 0 0 0, traffic: 4519849, 198 MB, drop: 0, 0 K cpu32 160061; 69543 4825400 362 [1.00], 0 0 0 0, traffic: 4825762, 211 MB, drop: 0, 0 K cpu33 155454; 69544 4686280 346 [1.00], 0 0 0 0, traffic: 4686626, 205 MB, drop: 0, 0 K cpu34 156829; 139062 4728017 353 [1.00], 0 0 0 0, traffic: 4728370, 207 MB, drop: 0, 0 K cpu35 146689; 97344 4422255 333 [1.00], 0 0 0 0, traffic: 4422588, 194 MB, drop: 0, 0 K cpu36 156830; 125162 4728128 351 [1.00], 0 0 0 0, traffic: 4728479, 207 MB, drop: 0, 0 K cpu37 148527; 83437 4477666 329 [1.00], 0 0 0 0, traffic: 4477995, 196 MB, drop: 0, 0 K cpu38 161931; 111264 4881009 358 [1.00], 0 0 0 0, traffic: 4881367, 214 MB, drop: 0, 0 K cpu39 158681; 55633 4783668 355 [1.00], 0 0 0 0, traffic: 4784023, 209 MB, drop: 0, 0 K cpu40 154071; 55605 4644761 352 [1.00], 0 0 0 0, traffic: 4645113, 203 MB, drop: 0, 0 K cpu41 168823; 76292 5089567 378 [1.00], 0 0 0 0, traffic: 5089945, 223 MB, drop: 0, 0 K cpu42 159147; 69541 4797631 358 [1.00], 0 0 0 0, traffic: 4797989, 210 MB, drop: 0, 0 K cpu43 160065; 69526 4825398 358 [1.00], 0 0 0 0, traffic: 4825756, 211 MB, drop: 0, 0 K cpu44 162830; 55637 4908893 367 [1.00], 0 0 0 0, traffic: 4909260, 215 MB, drop: 0, 0 K cpu45 152687; 69886 4603045 344 [1.00], 0 0 0 0, traffic: 4603389, 201 MB, drop: 0, 0 K cpu46 146222; 83927 4408257 337 [1.00], 0 0 0 0, traffic: 4408594, 193 MB, drop: 0, 0 K cpu47 163296; 41716 4922843 368 [1.00], 0 0 0 0, traffic: 4923211, 215 MB, drop: 0, 0 K cpu48 162823; 69542 4908684 365 [1.00], 0 0 0 0, traffic: 4909049, 215 MB, drop: 0, 0 K cpu49 160986; 55636 4853265 360 [1.00], 0 0 0 0, traffic: 4853625, 212 MB, drop: 0, 0 K cpu50 167904; 158880 5061826 375 [1.00], 0 0 0 0, traffic: 5062201, 222 MB, drop: 0, 0 K cpu51 162819; 97360 4908709 365 [1.00], 0 0 0 0, traffic: 4909074, 215 MB, drop: 0, 0 K cpu52 155907; 83422 4700162 349 [1.00], 0 0 0 0, traffic: 4700511, 206 MB, drop: 0, 0 K cpu53 174825; 69512 5270406 398 [1.00], 0 0 0 0, traffic: 5270804, 231 MB, drop: 0, 0 K cpu54 179705; 26844 5406791 417 [1.00], 0 0 0 0, traffic: 5407208, 237 MB, drop: 0, 0 K cpu55 150374; 55634 4533310 336 [1.00], 0 0 0 0, traffic: 4533646, 198 MB, drop: 0, 0 K Export: Rate 695 bytes/s; Total 15 pkts, 0 MB, 358 flows; Errors 0 pkts; Traffic lost 0 pkts, 0 Kbytes, 0 flows. sock0: 192.168.0.1:2504, sndbuf 18000000, filled 1, peak 2305; err: sndbuf reached 0, connect 0, cberr 0, other 0`

Aug 22 '23 14:08 ozschmoz

it would be great to have at least output of perf top for the situation with overloaded CPU cores. otherwise it's only guessing

Nov 12 '23 04:11 vvfedorenko