[VPP-2090] MRR < PDR: dpdk plugin with mlx5 driver does not read full queue

[vvalderrv]

Description

This is affecting 3-node performance, usually causing MRR values to be significantly lower than PDR. Affects both 3n-icx and 3n-alt. Example trending with 2 workers [0], but happens also with only one worker [1]. Not seen yet on other NICs [2], so this issue is probably specific to mlx5_core driver used by dpdk plugin for cx6dx NIC.

Looking at show runtime [3], dpdk-input Vectors/Call stays below 256 (good MRR tests achieve nearly 512) while still producing (almost) two frames per call.

In MRR tests the queue from TG should always be full, so it is not clear what makes dpdk-input not read 256 from it.

[0] https://csit.fd.io/trending/#eNrdVMsKwjAQ_Jp6kYW-e_Jg7X9Imq420EdIorR-vWkpbAuKKCLUQ17MZiY7DNGmVXjUWO2cKHWS1PETUdjJCfZbu1ylhKABVhnwXPeMvvR4Fxcd1FUXgZAhxGEOHgc05XCyI2caoVZqIPEPA0lxMQtGQmTZE_JUh-qZQkYXrBhBBvVMZfkcqjopVqMWN6RS2wDh3LpBkMeX_KaXM3TqMcnGiq9ZJ4v_t27q8YV1gnc_Sd1jnVWn7m3rPkzdqqyj1EXZpmlVPf55fjJsWTVpRNkdGLFe9A

[1] https://csit.fd.io/trending/#eNrdUksOgyAUPI3dNC8R_LDqoq33aFBfqwkoAdpoT18wJk8X3XXVBb_MMMNMcH60eHOoTklxScQl4aJvw5Rk52NYXsZANoBUHliaPpAb1kxlO4FWUwG9yaHMa2ANoO_iKYxaOgRtbRTh1yjSPv1OkRDTzYR89SG-tCjpQjAjyKPbuOyfQ6y7lRpd_0aihgCEN6ENgliz1_ez2aBrRlEtjJ9VZ9r_r27NGKorqsMwWr38Pi7iVqrVo6g-gey10Q

[2] https://csit.fd.io/trending/#eNrlUssOgjAQ_Bq8mE0oD_HiQeQ_TGlXbRRY22rEr7cQkoWDN28e2qaZ2ZnOpM53Fo8Ob7soL6OijJLC6LBF6X4djicRpC0Y9QIRx2dMSOBWxOoOmvQVDGWwyWoQCtBfhltYtXQIjbWDSnIYVPTDLyQZoUvPyHcjHpAWJU8EN4Y8upnN8j3MOlnZoDNvZGpIwLgKfTAk1FLf9zRDp5BFNTJ-Vx7pPyhvChnKy6tV29lm_IF59QEyybD4

[3] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3n-icx/383/log.html.gz#s1-s1-s1-s2-s1-t1-k2-k9-k9-k10-k1-k1-k1-k12

Assignee

Vratko Polak

Reporter

Vratko Polak

Comments

• vrpolak (Wed, 20 Nov 2024 12:46:24 +0000): No permanent changes in trending, rls2410 still shows [8] Vectors/Call around 200.

[8] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-spr/343/log.html.gz#s1-s1-s1-s2-s31-t1-k2-k9-k9-k10-k1-k1-k1-k12

• vrpolak (Mon, 12 Aug 2024 13:28:57 +0000): Symptom still present in rls2406. On 2n-spr [7], Vectors/Call is around 200.

[7] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-spr/284/log.html.gz#s1-s1-s1-s2-s31-t1-k2-k9-k9-k10-k1-k1-k1-k12

• vrpolak (Wed, 6 Dec 2023 13:37:19 +0000):

  The symptom is also affecting cx7. On 2-node testbeds the read still has less than 256 reads [6] in MRR tests, but in that case it does not lead to big difference in MRR vs PDR.

It seems on 3-node testbeds, the issue is affecting both queue equally, in sense similar percentage of available packets is left unread also on the dut-dut queue, causing additional loss.

[6] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-icx/402/log.html.gz#s1-s1-s1-s2-s29-t1-k2-k9-k9-k10-k1-k1-k1-k12

• vrpolak (Tue, 5 Dec 2023 16:54:00 +0000): I run two experiments. One [4] which simply iterates queues in reverse order, and one [5] which reads from at most one queue per dpdk-input call. Neither improved MRR performance, at least not above natural variance, far from approaching PDR.

[4] https://gerrit.fd.io/r/c/vpp/+/40062

[5] https://gerrit.fd.io/r/c/vpp/+/40061

Original issue: https://jira.fd.io/browse/VPP-2090