sonic-mgmt icon indicating copy to clipboard operation
sonic-mgmt copied to clipboard

Published 20 hours ago • verified publisher icon sonic-net

[Snappi] Added multi dut pfcwd burst test case

Open vikumarks opened this issue 1 year ago • 2 comments

Description of PR

  • Test PFC watchdog behavior under bursty PFC storms.

Summary: Fixes # (issue)

Type of change

  • [ ] Bug fix
  • [ ] Testbed and Framework(new/improvement)
  • [x ] Test case(new/improvement)

Back port request

  • [ ] 201911
  • [ ] 202012
  • [ ] 202205

Approach

What is the motivation for this PR?

To verify the pfc watchdog behavior in multi-dut/multi-asic scenario.

How did you do it?

The existing RDMA cases for single dut has been modified to accommodate T2 topology, where separate fixtures has been added that takes into account the chassis type and asic choice as user input, so that the script supports both T1 and T2 topology

How did you verify/test it?

Tested against voq chassis with SONiC installed in it

Any platform specific information?

Tested against J2C+

Supported testbed topology if it's a new test case?

T2, T1 & T0

Documentation

https://github.com/sonic-net/sonic-mgmt/tree/master/docs/testplan/pfcwd

Output

`AzDevOps@6105354dad99:~/sonic-mgmt/tests$ py.test --inventory ../ansible/snappi-sonic --host-pattern sonic-s6100-dut --testbed vms-snappi-sonic --testbed_file ../ansible/testbed.csv --show-capture=stdout --log-cli-level info --showlocals -ra --allow_recover --skip_sanity --disable_loganalyzer snappi_tests/dinesh_pfcwd/test_multidut_pfcwd_burst_storm_with_snappi.py /usr/local/lib/python2.7/dist-packages/ansible/parsing/vault/init.py:44: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release. from cryptography.exceptions import InvalidSignature ============================================================================================================ test session starts ============================================================================================================= platform linux2 -- Python 2.7.18, pytest-4.6.11, py-1.11.0, pluggy-0.13.1 ansible: 2.8.20 rootdir: /var/AzDevOps/sonic-mgmt/tests, inifile: pytest.ini plugins: allure-pytest-2.8.22, celery-4.4.7, ansible-2.2.4, forked-1.3.0, html-1.22.1, metadata-1.11.0, repeat-0.9.1, xdist-1.28.0 collecting 1 item
------------------------------------------------------------------------------------------------------------ live log collection ------------------------------------------------------------------------------------------------------------- 16:48:40 init.pytest_collection_modifyitems L0582 INFO | Available basic facts that can be used in conditional skip: { "topo_type": "ptf", "testbed": "vms-snappi-sonic", "topo_name": "ptf64" }

--------------------------------------------------------------------------------------------------------------- live log setup --------------------------------------------------------------------------------------------------------------- 16:48:40 init.set_default L0054 INFO | Completeness level not set during test execution. Setting to default level: CompletenessLevel.basic 16:48:40 init.check_test_completeness L0152 INFO | Test has no defined levels. Continue without test completeness checks 16:48:47 ptfhost_utils.run_icmp_responder_session L0239 INFO | Skip running icmp_responder at session level, it is only for dualtor testbed with active-active mux ports. 16:48:47 conftest.creds_on_dut L0704 INFO | dut sonic-s6100-dut belongs to groups [u'snappi-sonic', u'sonic', u'sonic_dell64_40', 'fanout'] 16:48:47 conftest.creds_on_dut L0728 INFO | skip empty var file ../ansible/group_vars/all/corefile_uploader.yml 16:48:47 conftest.creds_on_dut L0728 INFO | skip empty var file ../ansible/group_vars/all/env.yml 16:48:47 conftest.nbrhosts L0521 INFO | No VMs exist for this topology: ptf64 16:48:47 conftest.core_dump_and_config_check L2032 INFO | Collecting core dumps before test on sonic-s6100-dut 16:48:48 conftest.core_dump_and_config_check L2041 INFO | Collecting running config before test on sonic-s6100-dut 16:48:52 init.sanity_check L0125 INFO | Skip sanity check according to command line argument 16:48:52 conftest.generate_params_dut_hostname L1108 INFO | Using DUTs ['sonic-s6100-dut'] in testbed 'vms-snappi-sonic' 16:48:52 conftest.rand_one_dut_hostname L0378 INFO | Randomly select dut sonic-s6100-dut for testing 16:48:52 init._fixture_generator_decorator L0081 INFO | -------------------- fixture enable_packet_aging_after_test setup starts -------------------- 16:48:52 init._fixture_generator_decorator L0085 INFO | -------------------- fixture enable_packet_aging_after_test setup ends -------------------- 16:48:52 init._fixture_generator_decorator L0081 INFO | -------------------- fixture rand_lossless_prio setup starts -------------------- 16:48:52 init._fixture_generator_decorator L0085 INFO | -------------------- fixture rand_lossless_prio setup ends -------------------- 16:48:52 init._fixture_generator_decorator L0081 INFO | -------------------- fixture rand_lossy_prio setup starts -------------------- 16:48:52 init._fixture_generator_decorator L0085 INFO | -------------------- fixture rand_lossy_prio setup ends -------------------- 16:48:52 init._fixture_generator_decorator L0081 INFO | -------------------- fixture start_pfcwd_after_test setup starts -------------------- 16:48:52 init._fixture_generator_decorator L0085 INFO | -------------------- fixture start_pfcwd_after_test setup ends -------------------- 16:48:52 init._fixture_func_decorator L0069 INFO | -------------------- fixture snappi_api_serv_ip setup starts -------------------- 16:48:52 init._fixture_func_decorator L0076 INFO | -------------------- fixture snappi_api_serv_ip setup ends -------------------- 16:48:52 init._fixture_func_decorator L0069 INFO | -------------------- fixture snappi_api_serv_port setup starts -------------------- 16:48:52 init._fixture_func_decorator L0076 INFO | -------------------- fixture snappi_api_serv_port setup ends -------------------- 16:48:52 init._fixture_generator_decorator L0081 INFO | -------------------- fixture snappi_api setup starts -------------------- 16:48:52 init._fixture_generator_decorator L0085 INFO | -------------------- fixture snappi_api setup ends -------------------- 16:48:53 init._fixture_func_decorator L0069 INFO | -------------------- fixture prio_dscp_map setup starts -------------------- 16:48:54 init._fixture_func_decorator L0076 INFO | -------------------- fixture prio_dscp_map setup ends -------------------- 16:48:54 init._fixture_func_decorator L0069 INFO | -------------------- fixture all_prio_list setup starts -------------------- 16:48:54 init._fixture_func_decorator L0076 INFO | -------------------- fixture all_prio_list setup ends -------------------- 16:48:54 init._fixture_func_decorator L0069 INFO | -------------------- fixture lossless_prio_list setup starts -------------------- 16:48:55 init._fixture_func_decorator L0076 INFO | -------------------- fixture lossless_prio_list setup ends -------------------- 16:48:55 init._fixture_func_decorator L0069 INFO | -------------------- fixture lossy_prio_list setup starts -------------------- 16:48:55 init._fixture_func_decorator L0076 INFO | -------------------- fixture lossy_prio_list setup ends -------------------- 16:48:55 init.loganalyzer L0045 INFO | Log analyzer is disabled 16:48:55 init._fixture_func_decorator L0069 INFO | -------------------- fixture get_multidut_snappi_ports setup starts -------------------- 16:48:55 init._fixture_func_decorator L0076 INFO | -------------------- fixture get_multidut_snappi_ports setup ends -------------------- --------------------------------------------------------------------------------------------------------------- live log call ---------------------------------------------------------------------------------------------------------------- 16:48:55 snappi_fixtures.__intf_config_multidut L0789 INFO | Configuring Dut: sonic-s6100-dut with port Ethernet4 with IP 20.1.1.1/8 16:48:57 snappi_fixtures.__intf_config_multidut L0789 INFO | Configuring Dut: sonic-s6100-dut with port Ethernet12 with IP 21.1.1.1/8 16:49:17 connection._warn L0246 WARNING| Verification of certificates is disabled 16:49:17 connection._info L0243 INFO | Determining the platform and rest_port using the 10.36.78.135 address... 16:49:17 connection._warn L0246 WARNING| Unable to connect to http://10.36.78.135:443. 16:49:18 connection._info L0243 INFO | Connection established to https://10.36.78.135:443 on linux 16:49:31 connection._info L0243 INFO | Using IxNetwork api server version 9.30.2212.7 16:49:31 connection._info L0243 INFO | User info IxNetwork/ixnetworkweb/admin-127-4039 16:49:32 snappi_api.info L1132 INFO | snappi-0.9.1 16:49:32 snappi_api.info L1132 INFO | snappi_ixnetwork-0.9.1 16:49:32 snappi_api.info L1132 INFO | ixnetwork_restpy-1.0.64 16:49:33 snappi_api.info L1132 INFO | Config validation 0.030s 16:49:35 snappi_api.info L1132 INFO | Ports configuration 1.744s 16:49:35 snappi_api.info L1132 INFO | Captures configuration 0.177s 16:49:38 snappi_api.info L1132 INFO | Add location hosts [10.36.78.53] 2.405s 16:49:42 snappi_api.info L1132 INFO | Location hosts ready [10.36.78.53] 4.182s 16:49:43 snappi_api.info L1132 INFO | Speed conversion is not require for (port.name, speed) : [('Port 0', u'novusHundredGigNonFanOut'), ('Port 1', u'novusHundredGigNonFanOut')] 16:49:43 snappi_api.info L1132 INFO | Aggregation mode speed change 0.581s 16:49:47 snappi_api.info L1132 INFO | Location preemption [10.36.78.53;6;4, 10.36.78.53;6;2] 0.115s 16:50:09 snappi_api.info L1132 INFO | Location connect [Port 0, Port 1] 21.928s 16:50:09 snappi_api.warning L1138 WARNING| Port 0 connectedLinkDown 16:50:09 snappi_api.warning L1138 WARNING| Port 1 connectedLinkDown 16:50:09 snappi_api.info L1132 INFO | Location state check [Port 0, Port 1] 0.272s 16:50:09 snappi_api.info L1132 INFO | Location configuration 34.300s 16:50:23 snappi_api.info L1132 INFO | Layer1 configuration 13.634s 16:50:23 snappi_api.info L1132 INFO | Lag Configuration 0.087s 16:50:23 snappi_api.info L1132 INFO | Convert device config : 0.250s 16:50:23 snappi_api.info L1132 INFO | Create IxNetwork device config : 0.000s 16:50:24 snappi_api.info L1132 INFO | Push IxNetwork device config : 0.607s 16:50:24 snappi_api.info L1132 INFO | Devices configuration 0.941s 16:50:32 snappi_api.info L1132 INFO | Flows configuration 8.300s 16:50:39 snappi_api.info L1132 INFO | Start interfaces 6.742s 16:50:40 snappi_api.info L1132 INFO | IxNet - The Traffic Item was modified. Please perform a Traffic Generate to update the associated traffic Flow Groups 16:50:40 pfcwd_multidut_burst_storm_helper._run L0268 INFO | Wait for Arp to Resolve ... 16:50:41 pfcwd_multidut_burst_storm_helper._run L0271 INFO | Starting transmit on all flows ... 16:50:45 snappi_api.info L1132 INFO | Flows generate/apply 3.816s 16:50:57 snappi_api.info L1132 INFO | Flows clear statistics 11.806s 16:50:57 snappi_api.info L1132 INFO | Captures start 0.000s 16:51:00 snappi_api.info L1132 INFO | Flows start 2.720s 16:51:01 snappi_api.info L1132 INFO | IxNet - The total Traffic Rate of the Flow Groups transmitting from this port are exceeding the port capacity 16:51:22 pfcwd_multidut_burst_storm_helper._run L0303 INFO | Stop transmit on all flows ... 16:51:22 pfcwd_multidut_burst_storm_helper.__veri L0327 INFO | Flow Name : Pause Storm 0 , Tx Frames : 596, Rx Frames : 0 16:51:22 pfcwd_multidut_burst_storm_helper.__veri L0327 INFO | Flow Name : Pause Storm 1 , Tx Frames : 596, Rx Frames : 0 16:51:22 pfcwd_multidut_burst_storm_helper.__veri L0327 INFO | Flow Name : Pause Storm 2 , Tx Frames : 596, Rx Frames : 0 16:51:22 pfcwd_multidut_burst_storm_helper.__veri L0327 INFO | Flow Name : Pause Storm 3 , Tx Frames : 596, Rx Frames : 0 16:51:22 pfcwd_multidut_burst_storm_helper.__veri L0327 INFO | Flow Name : Pause Storm 4 , Tx Frames : 596, Rx Frames : 0 16:51:22 pfcwd_multidut_burst_storm_helper.__veri L0327 INFO | Flow Name : Pause Storm 5 , Tx Frames : 596, Rx Frames : 0 16:51:22 pfcwd_multidut_burst_storm_helper.__veri L0327 INFO | Flow Name : Data Flow Prio 4 , Tx Frames : 131704980, Rx Frames : 131704980 16:51:22 pfcwd_multidut_burst_storm_helper.__veri L0327 INFO | Flow Name : Pause Storm 7 , Tx Frames : 596, Rx Frames : 0 16:51:22 pfcwd_multidut_burst_storm_helper.__veri L0327 INFO | Flow Name : Pause Storm 8 , Tx Frames : 596, Rx Frames : 0 16:51:22 pfcwd_multidut_burst_storm_helper.__veri L0327 INFO | Flow Name : Pause Storm 9 , Tx Frames : 596, Rx Frames : 0 16:51:22 pfcwd_multidut_burst_storm_helper.__veri L0327 INFO | Flow Name : Warm Up Traffic Prio 4 , Tx Frames : 11973180, Rx Frames : 11973180 16:51:22 pfcwd_multidut_burst_storm_helper.__veri L0327 INFO | Flow Name : Pause Storm 6 , Tx Frames : 596, Rx Frames : 0 16:51:22 pfcwd_multidut_burst_storm_helper.__veri L0327 INFO | Flow Name : Pause Storm 12 , Tx Frames : 596, Rx Frames : 0 16:51:22 pfcwd_multidut_burst_storm_helper.__veri L0327 INFO | Flow Name : Pause Storm 13 , Tx Frames : 596, Rx Frames : 0 16:51:22 pfcwd_multidut_burst_storm_helper.__veri L0327 INFO | Flow Name : Pause Storm 10 , Tx Frames : 596, Rx Frames : 0 16:51:22 pfcwd_multidut_burst_storm_helper.__veri L0327 INFO | Flow Name : Pause Storm 11 , Tx Frames : 596, Rx Frames : 0 16:51:22 pfcwd_multidut_burst_storm_helper.__veri L0327 INFO | Flow Name : Pause Storm 14 , Tx Frames : 596, Rx Frames : 0 16:51:22 snappi_fixtures.cleanup_config L0942 INFO | Removing Configuration on Dut: sonic-s6100-dut with port Ethernet4 with ip :20.1.1.1/8 16:51:24 snappi_fixtures.cleanup_config L0942 INFO | Removing Configuration on Dut: sonic-s6100-dut with port Ethernet12 with ip :21.1.1.1/8 PASSED [100%] ------------------------------------------------------------------------------------------------------------- live log teardown -------------------------------------------------------------------------------------------------------------- 16:51:26 init._fixture_generator_decorator L0093 INFO | -------------------- fixture snappi_api teardown starts -------------------- 16:51:35 init._fixture_generator_decorator L0102 INFO | -------------------- fixture snappi_api teardown ends -------------------- 16:51:35 init._fixture_generator_decorator L0093 INFO | -------------------- fixture start_pfcwd_after_test teardown starts -------------------- 16:51:36 init._fixture_generator_decorator L0102 INFO | -------------------- fixture start_pfcwd_after_test teardown ends -------------------- 16:51:36 init._fixture_generator_decorator L0093 INFO | -------------------- fixture rand_lossy_prio teardown starts -------------------- 16:51:36 init._fixture_generator_decorator L0102 INFO | -------------------- fixture rand_lossy_prio teardown ends -------------------- 16:51:36 init._fixture_generator_decorator L0093 INFO | -------------------- fixture rand_lossless_prio teardown starts -------------------- 16:51:36 init._fixture_generator_decorator L0102 INFO | -------------------- fixture rand_lossless_prio teardown ends -------------------- 16:51:36 init._fixture_generator_decorator L0093 INFO | -------------------- fixture enable_packet_aging_after_test teardown starts -------------------- 16:51:38 init._fixture_generator_decorator L0102 INFO | -------------------- fixture enable_packet_aging_after_test teardown ends -------------------- 16:51:38 conftest.core_dump_and_config_check L2069 INFO | Collecting core dumps after test on sonic-s6100-dut 16:51:39 conftest.core_dump_and_config_check L2086 INFO | Collecting running config after test on sonic-s6100-dut 16:51:40 conftest.core_dump_and_config_check L2211 WARNING| Core dump or config check failed for test_multidut_pfcwd_burst_storm_with_snappi.py, results: {"core_dump_check": {"new_core_dumps": {"sonic-s6100-dut": []}, "pass": true}, "config_db_check": {"cur_only_config": {"sonic-s6100-dut": {"null": {"INTERFACE": {"Ethernet12|20.1.1.1/8": {}, "Ethernet8": {}, "Ethernet0|2000:1::1/16": {}, "Ethernet0": {}, "Ethernet4": {}, "Ethernet0|21.1.1.1/8": {}, "Ethernet12|23.1.1.1/8": {}, "Ethernet12|2003:1::1/16": {}, "Ethernet8|2002:1::1/16": {}, "Ethernet4|2001:1::1/16": {}, "Ethernet8|22.1.1.1/8": {}, "Ethernet4|21.1.1.1/8": {}, "Ethernet0|20.1.1.1/8": {}, "Ethernet12": {}}, "PFC_WD": {"Ethernet0": {"action": "drop", "detection_time": "200", "restoration_time": "200"}, "GLOBAL": {"POLL_INTERVAL": "200"}, "Ethernet4": {"action": "drop", "detection_time": "200", "restoration_time": "200"}, "Ethernet12": {"action": "drop", "detection_time": "200", "restoration_time": "200"}, "Ethernet8": {"action": "drop", "detection_time": "200", "restoration_time": "200"}}}}}, "inconsistent_config": {"sonic-s6100-dut": {"null": {}}}, "pre_only_config": {"sonic-s6100-dut": {"null": {"VLAN_MEMBER": {"Vlan2|Ethernet8": {"tagging_mode": "untagged"}, "Vlan2|Ethernet0": {"tagging_mode": "untagged"}, "Vlan2|Ethernet12": {"tagging_mode": "untagged"}, "Vlan2|Ethernet4": {"tagging_mode": "untagged"}}}}}, "pass": false}}

============================================================================================================== warnings summary ============================================================================================================== /usr/local/lib/python2.7/dist-packages/_pytest/config/init.py:545 /usr/local/lib/python2.7/dist-packages/_pytest/config/init.py:545: PytestAssertRewriteWarning: Module already imported so cannot be rewritten: tests.common.plugins.loganalyzer self.import_plugin(import_spec)

/usr/local/lib/python2.7/dist-packages/_pytest/config/init.py:545 /usr/local/lib/python2.7/dist-packages/_pytest/config/init.py:545: PytestAssertRewriteWarning: Module already imported so cannot be rewritten: tests.common.plugins.sanity_check self.import_plugin(import_spec)

/usr/local/lib/python2.7/dist-packages/_pytest/config/init.py:545 /usr/local/lib/python2.7/dist-packages/_pytest/config/init.py:545: PytestAssertRewriteWarning: Module already imported so cannot be rewritten: tests.common.dualtor self.import_plugin(import_spec)

snappi_tests/dinesh_pfcwd/test_multidut_pfcwd_burst_storm_with_snappi.py::test_pfcwd_burst_storm_single_lossless_prio[linecard_configuration_set0-non_chassis_single_line_card] /usr/local/lib/python2.7/dist-packages/pytest_ansible/module_dispatcher/v28.py:81: UserWarning: provided hosts list is empty, only localhost is available warnings.warn("provided hosts list is empty, only localhost is available")

-- Docs: https://docs.pytest.org/en/latest/warnings.html ----------------------------------------------------------------------------------------------------------- live log sessionfinish ----------------------------------------------------------------------------------------------------------- 16:51:40 init.pytest_terminal_summary L0064 INFO | Can not get Allure report URL. Please check logs ==================== 1 passed, 4 warnings in 180.21 seconds ==================== INFO:root:Can not get Allure report URL. Please check logs`

vikumarks avatar Mar 30 '23 00:03 vikumarks

@developfast Can you please review

dks0692 avatar Oct 06 '23 16:10 dks0692

@yxieca @wangxin please help merge.

selldinesh avatar Apr 08 '24 22:04 selldinesh