What is the reason for several tests to fail in a make-all run?

[vzhirnov]

Hi there,

Context

I'd like to implement vpp tests into our CI pipeline.

My environment

• WSL2 OS:

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.4 LTS"

Issue

• I run sanitation tests on my work machine with WSL2, they pass successfully:

make test

==============================================================================
TEST RESULTS:
    Scheduled tests: 2226
     Executed tests: 2226
       Passed tests: 1737
      Skipped tests: 489
==============================================================================

Test run was successful

• However, in the extended run there are several failed tests, and several that completed with an error:

make test-all

==============================================================================
TEST RESULTS:
    Scheduled tests: 2226
     Executed tests: 2226
       Passed tests: 1818
      Skipped tests: 392
           Failures: 6
             Errors: 10
FAILURES AND ERRORS IN TESTS:
  Testcase name: NAT64 Test Cases
    ERROR: IPFIX logging maximum session and BIB entries exceeded [test_nat64.TestNAT64.test_ipfix_max_bibs_sessions]
  Testcase name: Auto SDL Baasic Test Case
    ERROR: Auto SDL test [test_auto_sdl.TestAutoSDL.test_auto_sdl]
    ERROR: Auto SDL with appns test -- ip4 [test_auto_sdl.TestAutoSDL.test_auto_sdl_appns_ip4]
    ERROR: Auto SDL with appns test -- ip6 [test_auto_sdl.TestAutoSDL.test_auto_sdl_appns_ip6]
  Testcase name: IPv4 VRRP Test Case
    ERROR: IPv4 Backup VR becomes master if no advertisements received [test_vrrp.TestVRRP4.test_vrrp4_election]
    ERROR: IPv4 Master VR adjusts priority based on tracked interface [test_vrrp.TestVRRP4.test_vrrp4_intf_tracking]
    ERROR: IPv4 Master VR advertises (unicast) [test_vrrp.TestVRRP4.test_vrrp4_master_adv_unicast]
  Testcase name: IPv6 VRRP Test Case
    ERROR: IPv6 Master VR adjusts priority based on tracked interface [test_vrrp.TestVRRP6.test_vrrp6_intf_tracking]
  Testcase name: Deterministic NAT Test Cases
    FAILURE: Deterministic NAT session timeouts [test_det44.TestDET44.test_session_timeout]
  Testcase name: Disable Flowprobe feature
    ERROR: disable flowprobe feature and expect no leftover flows [test_flowprobe.DisableFP.test_no_leftover_flows_after_disabling]
  Testcase name: Container integration extended testcases
    FAILURE: IPv4 local-spoof connectivity test [test_container.ContainerIntegrationTestCase.test_0110_basic_conn_test]
    FAILURE: IPv6 local-spoof connectivity test [test_container.ContainerIntegrationTestCase.test_0111_basic_conn_test]
    FAILURE: IPv4 local-spoof after unconfig test [test_container.ContainerIntegrationTestCase.test_0410_spoof_test]
    FAILURE: IPv6 local-spoof after unconfig test [test_container.ContainerIntegrationTestCase.test_0411_spoof_test]
    ERROR: Create loopbacks overlapping with remote addresses [test_container.ContainerIntegrationTestCase.test_0050_loopback_prepare_test]
  Testcase name: MAP Test Case
    FAILURE: test_api_map_domains_get (test_map.TestMAP) [test_map.TestMAP.test_api_map_domains_get]
==============================================================================

0 attempt(s) left.
Killing possible remaining process IDs:  10513 10515
Not compressing files in temporary directories from failed test runs.
make[1]: *** [Makefile:320: test] Error 1
make[1]: Leaving directory '/home/vzhirnov/work/projects/vpp/test'
make: *** [Makefile:627: test-all] Error 2

Question

• What could be the probable cause of test failures and errors?
• I tried running tests inside a docker container, but still found several errors and fails in the tests. I don't understand why this is so. Maybe these are expected failures, or nuances of my working environment?

---

• I am ready to provide any information about my existing work environment, test process etc.