sonic-mgmt
sonic-mgmt copied to clipboard
sonic-net
[Snappi] Added multi dut pfcwd m2o test case
Description of PR
- Test PFC watchdog's test in a multimode topology.
Summary: Fixes # (issue)
Type of change
- [ ] Bug fix
- [ ] Testbed and Framework(new/improvement)
- [x ] Test case(new/improvement)
Back port request
- [ ] 201911
- [ ] 202012
- [ ] 202205
Approach
What is the motivation for this PR?
To test PFC lossless behavior in multi-dut/multi-asic scenario.
How did you do it?
The existing RDMA cases for single dut has been modified to accommodate T2 topology, where separate fixtures has been added that takes into account the chassis type and asic choice as user input, so that the script supports both T1 and T2 topology
How did you verify/test it?
Tested against voq chassis with SONiC installed in it
Any platform specific information?
Tested against J2C+
Supported testbed topology if it's a new test case?
T2, T1 & T0
Documentation
https://github.com/sonic-net/sonic-mgmt/tree/master/docs/testplan/pfcwd
Output
`AzDevOps@6105354dad99:~/sonic-mgmt/tests$ py.test --inventory ../ansible/snappi-sonic --host-pattern sonic-s6100-dut --testbed vms-snappi-sonic --testbed_file ../ansible/testbed.csv --show-capture=stdout --log-cli-level info --showlocals -ra --allow_recover --skip_sanity --disable_loganalyzer snappi_tests/dinesh_pfcwd/test_multidut_pfcwd_m2o_with_snappi.py
/usr/local/lib/python2.7/dist-packages/ansible/parsing/vault/init.py:44: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release.
from cryptography.exceptions import InvalidSignature
============================================================================================ test session starts ============================================================================================
platform linux2 -- Python 2.7.18, pytest-4.6.11, py-1.11.0, pluggy-0.13.1
ansible: 2.8.20
rootdir: /var/AzDevOps/sonic-mgmt/tests, inifile: pytest.ini
plugins: allure-pytest-2.8.22, celery-4.4.7, ansible-2.2.4, forked-1.3.0, html-1.22.1, metadata-1.11.0, repeat-0.9.1, xdist-1.28.0
collecting 1 item
-------------------------------------------------------------------------------------------- live log collection --------------------------------------------------------------------------------------------
21:43:31 init.pytest_collection_modifyitems L0582 INFO | Available basic facts that can be used in conditional skip:
{
"topo_type": "ptf",
"testbed": "vms-snappi-sonic",
"topo_name": "ptf64"
}
---------------------------------------------------------------------------------------------- live log setup -----------------------------------------------------------------------------------------------
21:43:31 init.set_default L0054 INFO | Completeness level not set during test execution. Setting to default level: CompletenessLevel.basic
21:43:31 init.check_test_completeness L0152 INFO | Test has no defined levels. Continue without test completeness checks
21:43:38 ptfhost_utils.run_icmp_responder_session L0239 INFO | Skip running icmp_responder at session level, it is only for dualtor testbed with active-active mux ports.
21:43:38 conftest.creds_on_dut L0704 INFO | dut sonic-s6100-dut belongs to groups [u'snappi-sonic', u'sonic', u'sonic_dell64_40', 'fanout']
21:43:38 conftest.creds_on_dut L0728 INFO | skip empty var file ../ansible/group_vars/all/corefile_uploader.yml
21:43:38 conftest.creds_on_dut L0728 INFO | skip empty var file ../ansible/group_vars/all/env.yml
21:43:38 conftest.nbrhosts L0521 INFO | No VMs exist for this topology: ptf64
21:43:38 conftest.core_dump_and_config_check L2032 INFO | Collecting core dumps before test on sonic-s6100-dut
21:43:39 conftest.core_dump_and_config_check L2041 INFO | Collecting running config before test on sonic-s6100-dut
21:43:44 init.sanity_check L0125 INFO | Skip sanity check according to command line argument
21:43:44 conftest.generate_params_dut_hostname L1108 INFO | Using DUTs ['sonic-s6100-dut'] in testbed 'vms-snappi-sonic'
21:43:44 conftest.rand_one_dut_hostname L0378 INFO | Randomly select dut sonic-s6100-dut for testing
21:43:44 init._fixture_generator_decorator L0081 INFO | -------------------- fixture enable_packet_aging_after_test setup starts --------------------
21:43:44 init._fixture_generator_decorator L0085 INFO | -------------------- fixture enable_packet_aging_after_test setup ends --------------------
21:43:44 init._fixture_generator_decorator L0081 INFO | -------------------- fixture rand_lossless_prio setup starts --------------------
21:43:44 init._fixture_generator_decorator L0085 INFO | -------------------- fixture rand_lossless_prio setup ends --------------------
21:43:44 init._fixture_generator_decorator L0081 INFO | -------------------- fixture rand_lossy_prio setup starts --------------------
21:43:44 init._fixture_generator_decorator L0085 INFO | -------------------- fixture rand_lossy_prio setup ends --------------------
21:43:44 init._fixture_generator_decorator L0081 INFO | -------------------- fixture start_pfcwd_after_test setup starts --------------------
21:43:44 init._fixture_generator_decorator L0085 INFO | -------------------- fixture start_pfcwd_after_test setup ends --------------------
21:43:44 init._fixture_func_decorator L0069 INFO | -------------------- fixture snappi_api_serv_ip setup starts --------------------
21:43:44 init._fixture_func_decorator L0076 INFO | -------------------- fixture snappi_api_serv_ip setup ends --------------------
21:43:44 init._fixture_func_decorator L0069 INFO | -------------------- fixture snappi_api_serv_port setup starts --------------------
21:43:44 init._fixture_func_decorator L0076 INFO | -------------------- fixture snappi_api_serv_port setup ends --------------------
21:43:44 init._fixture_generator_decorator L0081 INFO | -------------------- fixture snappi_api setup starts --------------------
21:43:44 init._fixture_generator_decorator L0085 INFO | -------------------- fixture snappi_api setup ends --------------------
21:43:44 init._fixture_func_decorator L0069 INFO | -------------------- fixture prio_dscp_map setup starts --------------------
21:43:45 init._fixture_func_decorator L0076 INFO | -------------------- fixture prio_dscp_map setup ends --------------------
21:43:45 init._fixture_func_decorator L0069 INFO | -------------------- fixture all_prio_list setup starts --------------------
21:43:45 init._fixture_func_decorator L0076 INFO | -------------------- fixture all_prio_list setup ends --------------------
21:43:45 init._fixture_func_decorator L0069 INFO | -------------------- fixture lossless_prio_list setup starts --------------------
21:43:46 init._fixture_func_decorator L0076 INFO | -------------------- fixture lossless_prio_list setup ends --------------------
21:43:46 init._fixture_func_decorator L0069 INFO | -------------------- fixture lossy_prio_list setup starts --------------------
21:43:46 init._fixture_func_decorator L0076 INFO | -------------------- fixture lossy_prio_list setup ends --------------------
21:43:46 init.loganalyzer L0045 INFO | Log analyzer is disabled
21:43:46 init._fixture_func_decorator L0069 INFO | -------------------- fixture get_multidut_snappi_ports setup starts --------------------
21:43:46 init._fixture_func_decorator L0076 INFO | -------------------- fixture get_multidut_snappi_ports setup ends --------------------
----------------------------------------------------------------------------------------------- live log call -----------------------------------------------------------------------------------------------
21:43:47 snappi_fixtures.__intf_config_multidut L0789 INFO | Configuring Dut: sonic-s6100-dut with port Ethernet8 with IP 20.1.1.1/8
21:43:48 snappi_fixtures.__intf_config_multidut L0789 INFO | Configuring Dut: sonic-s6100-dut with port Ethernet4 with IP 21.1.1.1/8
21:43:49 snappi_fixtures.__intf_config_multidut L0789 INFO | Configuring Dut: sonic-s6100-dut with port Ethernet12 with IP 22.1.1.1/8
21:43:59 connection._warn L0246 WARNING| Verification of certificates is disabled
21:43:59 connection._info L0243 INFO | Determining the platform and rest_port using the 10.36.78.135 address...
21:43:59 connection._warn L0246 WARNING| Unable to connect to http://10.36.78.135:443.
21:43:59 connection._info L0243 INFO | Connection established to https://10.36.78.135:443 on linux
21:44:12 connection._info L0243 INFO | Using IxNetwork api server version 9.30.2212.7
21:44:12 connection._info L0243 INFO | User info IxNetwork/ixnetworkweb/admin-124-9127
21:44:13 snappi_api.info L1132 INFO | snappi-0.9.1
21:44:13 snappi_api.info L1132 INFO | snappi_ixnetwork-0.9.1
21:44:13 snappi_api.info L1132 INFO | ixnetwork_restpy-1.0.64
21:44:14 snappi_api.info L1132 INFO | Config validation 0.029s
21:44:16 snappi_api.info L1132 INFO | Ports configuration 1.731s
21:44:16 snappi_api.info L1132 INFO | Captures configuration 0.176s
21:44:19 snappi_api.info L1132 INFO | Add location hosts [10.36.78.53] 2.315s
21:44:23 snappi_api.info L1132 INFO | Location hosts ready [10.36.78.53] 4.184s
21:44:23 snappi_api.info L1132 INFO | Speed conversion is not require for (port.name, speed) : [('Port 0', u'novusHundredGigNonFanOut'), ('Port 1', u'novusHundredGigNonFanOut'), ('Port 2', u'novusHundredGigNonFanOut')]
21:44:23 snappi_api.info L1132 INFO | Aggregation mode speed change 0.442s
21:44:30 snappi_api.info L1132 INFO | Location preemption [10.36.78.53;6;4, 10.36.78.53;6;2, 10.36.78.53;6;3] 0.119s
21:44:57 snappi_api.info L1132 INFO | Location connect [Port 0, Port 1, Port 2] 27.319s
21:44:58 snappi_api.warning L1138 WARNING| Port 0 connectedLinkDown
21:44:58 snappi_api.warning L1138 WARNING| Port 1 connectedLinkDown
21:44:58 snappi_api.warning L1138 WARNING| Port 2 connectedLinkDown
21:44:58 snappi_api.info L1132 INFO | Location state check [Port 0, Port 1, Port 2] 0.277s
21:44:58 snappi_api.info L1132 INFO | Location configuration 41.729s
21:45:20 snappi_api.info L1132 INFO | Layer1 configuration 22.132s
21:45:20 snappi_api.info L1132 INFO | Lag Configuration 0.090s
21:45:20 snappi_api.info L1132 INFO | Convert device config : 0.251s
21:45:20 snappi_api.info L1132 INFO | Create IxNetwork device config : 0.000s
21:45:21 snappi_api.info L1132 INFO | Push IxNetwork device config : 0.662s
21:45:21 snappi_api.info L1132 INFO | Devices configuration 0.993s
21:45:33 snappi_api.info L1132 INFO | Flows configuration 11.765s
21:45:41 snappi_api.info L1132 INFO | Start interfaces 7.832s
21:45:41 snappi_api.info L1132 INFO | IxNet - The Traffic Item was modified. Please perform a Traffic Generate to update the associated traffic Flow Groups
21:45:41 pfcwd_multidut_multi_node_helper.__run_t L0508 INFO | Wait for Arp to Resolve ...
21:45:43 pfcwd_multidut_multi_node_helper.__run_t L0511 INFO | Starting transmit on all flows ...
21:45:47 snappi_api.info L1132 INFO | Flows generate/apply 3.268s
21:45:59 snappi_api.info L1132 INFO | Flows clear statistics 12.220s
21:45:59 snappi_api.info L1132 INFO | Captures start 0.000s
21:46:03 snappi_api.info L1132 INFO | Flows start 4.075s
21:46:13 pfcwd_multidut_multi_node_helper.__run_t L0544 INFO | Stop transmit on all flows ...
21:46:14 pfcwd_multidut_multi_node_helper.__verif L0589 INFO | Flow Name : Background Flow 1 -> 0 Prio 2 , Tx Frames : 957854, Rx Frames : 957854
21:46:14 pfcwd_multidut_multi_node_helper.__verif L0599 INFO | Background flows expected not to have any dropped packets
21:46:14 pfcwd_multidut_multi_node_helper.__verif L0589 INFO | Flow Name : Warm Up Traffic 2 -> 0 Prio 4 , Tx Frames : 2634099, Rx Frames : 2634099
21:46:14 pfcwd_multidut_multi_node_helper.__verif L0589 INFO | Flow Name : Warm Up Traffic 1 -> 0 Prio 4 , Tx Frames : 2634099, Rx Frames : 2634099
21:46:14 pfcwd_multidut_multi_node_helper.__verif L0589 INFO | Flow Name : Background Flow 1 -> 0 Prio 5 , Tx Frames : 957854, Rx Frames : 957854
21:46:14 pfcwd_multidut_multi_node_helper.__verif L0599 INFO | Background flows expected not to have any dropped packets
21:46:14 pfcwd_multidut_multi_node_helper.__verif L0589 INFO | Flow Name : Background Flow 1 -> 0 Prio 6 , Tx Frames : 957854, Rx Frames : 957854
21:46:14 pfcwd_multidut_multi_node_helper.__verif L0599 INFO | Background flows expected not to have any dropped packets
21:46:14 pfcwd_multidut_multi_node_helper.__verif L0589 INFO | Flow Name : Test Flow 2 -> 0 Prio 4 , Tx Frames : 5268199, Rx Frames : 4207505
21:46:14 pfcwd_multidut_multi_node_helper.__verif L0621 INFO | Once PFC watchdog is triggered, it will impact bi-directional traffic
21:46:14 pfcwd_multidut_multi_node_helper.__verif L0622 INFO | Tx and Rx should have dropped packets
21:46:14 pfcwd_multidut_multi_node_helper.__verif L0589 INFO | Flow Name : Background Flow 1 -> 0 Prio 1 , Tx Frames : 957854, Rx Frames : 957854
21:46:14 pfcwd_multidut_multi_node_helper.__verif L0599 INFO | Background flows expected not to have any dropped packets
21:46:14 pfcwd_multidut_multi_node_helper.__verif L0589 INFO | Flow Name : Pause Storm , Tx Frames : 2384, Rx Frames : 0
21:46:14 pfcwd_multidut_multi_node_helper.__verif L0593 INFO | PFC pause storm expected to be dropped
21:46:14 pfcwd_multidut_multi_node_helper.__verif L0589 INFO | Flow Name : Test Flow 1 -> 0 Prio 4 , Tx Frames : 5268199, Rx Frames : 4207504
21:46:14 pfcwd_multidut_multi_node_helper.__verif L0621 INFO | Once PFC watchdog is triggered, it will impact bi-directional traffic
21:46:14 pfcwd_multidut_multi_node_helper.__verif L0622 INFO | Tx and Rx should have dropped packets
21:46:14 pfcwd_multidut_multi_node_helper.__verif L0589 INFO | Flow Name : Background Flow 2 -> 0 Prio 6 , Tx Frames : 957854, Rx Frames : 957854
21:46:14 pfcwd_multidut_multi_node_helper.__verif L0599 INFO | Background flows expected not to have any dropped packets
21:46:14 pfcwd_multidut_multi_node_helper.__verif L0589 INFO | Flow Name : Background Flow 1 -> 0 Prio 0 , Tx Frames : 957854, Rx Frames : 957854
21:46:14 pfcwd_multidut_multi_node_helper.__verif L0599 INFO | Background flows expected not to have any dropped packets
21:46:14 pfcwd_multidut_multi_node_helper.__verif L0589 INFO | Flow Name : Background Flow 2 -> 0 Prio 5 , Tx Frames : 957854, Rx Frames : 957854
21:46:14 pfcwd_multidut_multi_node_helper.__verif L0599 INFO | Background flows expected not to have any dropped packets
21:46:14 pfcwd_multidut_multi_node_helper.__verif L0589 INFO | Flow Name : Background Flow 2 -> 0 Prio 2 , Tx Frames : 957854, Rx Frames : 957854
21:46:14 pfcwd_multidut_multi_node_helper.__verif L0599 INFO | Background flows expected not to have any dropped packets
21:46:14 pfcwd_multidut_multi_node_helper.__verif L0589 INFO | Flow Name : Background Flow 2 -> 0 Prio 0 , Tx Frames : 957854, Rx Frames : 957854
21:46:14 pfcwd_multidut_multi_node_helper.__verif L0599 INFO | Background flows expected not to have any dropped packets
21:46:14 pfcwd_multidut_multi_node_helper.__verif L0589 INFO | Flow Name : Background Flow 2 -> 0 Prio 1 , Tx Frames : 957854, Rx Frames : 957854
21:46:14 pfcwd_multidut_multi_node_helper.__verif L0599 INFO | Background flows expected not to have any dropped packets
21:46:14 snappi_fixtures.cleanup_config L0942 INFO | Removing Configuration on Dut: sonic-s6100-dut with port Ethernet8 with ip :20.1.1.1/8
21:46:16 snappi_fixtures.cleanup_config L0942 INFO | Removing Configuration on Dut: sonic-s6100-dut with port Ethernet4 with ip :21.1.1.1/8
21:46:17 snappi_fixtures.cleanup_config L0942 INFO | Removing Configuration on Dut: sonic-s6100-dut with port Ethernet12 with ip :22.1.1.1/8
PASSED [100%]
--------------------------------------------------------------------------------------------- live log teardown ---------------------------------------------------------------------------------------------
21:46:18 init._fixture_generator_decorator L0093 INFO | -------------------- fixture snappi_api teardown starts --------------------
21:46:52 init._fixture_generator_decorator L0102 INFO | -------------------- fixture snappi_api teardown ends --------------------
21:46:52 init._fixture_generator_decorator L0093 INFO | -------------------- fixture start_pfcwd_after_test teardown starts --------------------
21:46:53 init._fixture_generator_decorator L0102 INFO | -------------------- fixture start_pfcwd_after_test teardown ends --------------------
21:46:53 init._fixture_generator_decorator L0093 INFO | -------------------- fixture rand_lossy_prio teardown starts --------------------
21:46:53 init._fixture_generator_decorator L0102 INFO | -------------------- fixture rand_lossy_prio teardown ends --------------------
21:46:53 init._fixture_generator_decorator L0093 INFO | -------------------- fixture rand_lossless_prio teardown starts --------------------
21:46:53 init._fixture_generator_decorator L0102 INFO | -------------------- fixture rand_lossless_prio teardown ends --------------------
21:46:53 init._fixture_generator_decorator L0093 INFO | -------------------- fixture enable_packet_aging_after_test teardown starts --------------------
21:46:54 init._fixture_generator_decorator L0102 INFO | -------------------- fixture enable_packet_aging_after_test teardown ends --------------------
21:46:56 conftest.core_dump_and_config_check L2069 INFO | Collecting core dumps after test on sonic-s6100-dut
21:46:56 conftest.core_dump_and_config_check L2086 INFO | Collecting running config after test on sonic-s6100-dut
21:46:58 conftest.core_dump_and_config_check L2211 WARNING| Core dump or config check failed for test_multidut_pfcwd_m2o_with_snappi.py, results: {"core_dump_check": {"new_core_dumps": {"sonic-s6100-dut": []}, "pass": true}, "config_db_check": {"cur_only_config": {"sonic-s6100-dut": {"null": {"INTERFACE": {"Ethernet12|20.1.1.1/8": {}, "Ethernet8": {}, "Ethernet0|2000:1::1/16": {}, "Ethernet8|21.1.1.1/8": {}, "Ethernet0": {}, "Ethernet4": {}, "Ethernet4|22.1.1.1/8": {}, "Ethernet0|21.1.1.1/8": {}, "Ethernet12|23.1.1.1/8": {}, "Ethernet12|2003:1::1/16": {}, "Ethernet8|2002:1::1/16": {}, "Ethernet4|2001:1::1/16": {}, "Ethernet8|22.1.1.1/8": {}, "Ethernet0|20.1.1.1/8": {}, "Ethernet12": {}}, "PFC_WD": {"Ethernet0": {"action": "drop", "detection_time": "200", "restoration_time": "200"}, "GLOBAL": {"POLL_INTERVAL": "200"}, "Ethernet4": {"action": "drop", "detection_time": "200", "restoration_time": "200"}, "Ethernet12": {"action": "drop", "detection_time": "200", "restoration_time": "200"}, "Ethernet8": {"action": "drop", "detection_time": "200", "restoration_time": "200"}}}}}, "inconsistent_config": {"sonic-s6100-dut": {"null": {}}}, "pre_only_config": {"sonic-s6100-dut": {"null": {"VLAN_MEMBER": {"Vlan2|Ethernet8": {"tagging_mode": "untagged"}, "Vlan2|Ethernet0": {"tagging_mode": "untagged"}, "Vlan2|Ethernet12": {"tagging_mode": "untagged"}, "Vlan2|Ethernet4": {"tagging_mode": "untagged"}}}}}, "pass": false}}
============================================================================================= warnings summary ============================================================================================== /usr/local/lib/python2.7/dist-packages/_pytest/config/init.py:545 /usr/local/lib/python2.7/dist-packages/_pytest/config/init.py:545: PytestAssertRewriteWarning: Module already imported so cannot be rewritten: tests.common.plugins.loganalyzer self.import_plugin(import_spec)
/usr/local/lib/python2.7/dist-packages/_pytest/config/init.py:545 /usr/local/lib/python2.7/dist-packages/_pytest/config/init.py:545: PytestAssertRewriteWarning: Module already imported so cannot be rewritten: tests.common.plugins.sanity_check self.import_plugin(import_spec)
/usr/local/lib/python2.7/dist-packages/_pytest/config/init.py:545 /usr/local/lib/python2.7/dist-packages/_pytest/config/init.py:545: PytestAssertRewriteWarning: Module already imported so cannot be rewritten: tests.common.dualtor self.import_plugin(import_spec)
snappi_tests/dinesh_pfcwd/test_multidut_pfcwd_m2o_with_snappi.py::test_pfcwd_many_to_one[linecard_configuration_set0-non_chassis_single_line_card-True] /usr/local/lib/python2.7/dist-packages/pytest_ansible/module_dispatcher/v28.py:81: UserWarning: provided hosts list is empty, only localhost is available warnings.warn("provided hosts list is empty, only localhost is available")
-- Docs: https://docs.pytest.org/en/latest/warnings.html ------------------------------------------------------------------------------------------ live log sessionfinish ------------------------------------------------------------------------------------------- 21:46:58 init.pytest_terminal_summary L0064 INFO | Can not get Allure report URL. Please check logs ==================== 1 passed, 4 warnings in 207.30 seconds ==================== INFO:root:Can not get Allure report URL. Please check logs`
@vikumarks it is failing in test. please check.
@yxieca @developfast @wangxin please help merge.
@vikumarks please fix failures here. @selldinesh
@vmittal-msft the failure is because of dependency on PR https://github.com/sonic-net/sonic-mgmt/pull/7916/ for pfcwd_multidut_multi_node_helper.py
