Packet loss due to rx_out_of_buffer during route updates and stats polling (VPP 21.06 & 24.06)

[avadhootd]

Hello,

We are observing consistent packet loss caused by rx_out_of_buffer drops during routing table updates (addition and deletion of routes for routes other than the ones being used in traffic test) via VAPI and during polling of stats via vppctl such as show hardware-interfaces.

This issue occurs in both VPP 21.06 and 24.06, and appears more pronounced when we are sending traffic above 1.8 Mpps and either one of the below two in progress:

• Routing table updates are performed rapidly in batches using VAPI

• Stats are polled via vppctl

We currently have below configuration,

dpdk {

dev default {

num-rx-queues 8

num-rx-desc 4096

num-tx-desc 4096

}

socket-mem 2048

}

buffers {

buffers-per-numa 256000

}

Our observations as below,

• Packet loss coincides precisely with the timing of either routing updates or stat polling

• Drops disappear when routing and stats polling are paused. Even at 4Mpps or higher

• Increasing desc or socket-mem or buffers-per-numa not helping with the same.

Is there a known issue where routing or stats operations cause packet drops?

Is there a recommended configuration for queues at such high pps traffic while performing frequent route updates?

Regards,

Avadhoot