Adding modifications for pfcwd_function to handle new cisco-8000 behaviour

[rraghav-cisco]

Description of PR

Recently cisco-8000 was updated to not trigger pfcwd even when there is a lot of PFC-Xoff, when there is no traffic. Earlier behaviour is to trigger watchdog with empty traffic when there is a storm. This PR addresses the new behaviour of cisco-8000, and the modifications necessary to handle this. Particularly:

1. LogAnalyzer should flag an error if pfcwd is triggered before any traffic is sent. There should be no watchdog message in syslog after pfc storm is started, but before traffic is started.
2. LogAnalyzer should expect the watchdog messages after traffic is started.

Summary: Fixes failure of test_pfcwd_function.py for new cisco-8000.

Type of change

• [X] Bug fix
• [ ] Testbed and Framework(new/improvement)
• [ ] Test case(new/improvement)

Back port request

• [ ] 201911
• [ ] 202012
• [ ] 202205
• [X] 202305
• [X] 202311

Approach

What is the motivation for this PR?

New cisco-8000 pfcwd behaviour.

How did you do it?

Updated the test_pfcwd_function.py to handle the new behaviour.

How did you verify/test it?

Ran it on T0 testbed:

=============================================================================================== PASSES ===============================================================================================
____________________________________________________________________________ TestPfcwdFunc.test_pfcwd_actions[mth-t0-64] _____________________________________________________________________________
___________________________________________________________________________ TestPfcwdFunc.test_pfcwd_multi_port[mth-t0-64] ___________________________________________________________________________
___________________________________________________________________________ TestPfcwdFunc.test_pfcwd_mmu_change[mth-t0-64] ___________________________________________________________________________
__________________________________________________________________________ TestPfcwdFunc.test_pfcwd_port_toggle[mth-t0-64] ___________________________________________________________________________
------------------------------------------------ generated xml file: /run_logs/2024-03-22-00-31-33/pfcwd/test_pfcwd_function_2024-03-22-00-31-33.xml -------------------------------------------------
INFO:root:Can not get Allure report URL. Please check logs
--------------------------------------------------------------------------------------- live log sessionfinish ---------------------------------------------------------------------------------------
01:12:36 __init__.pytest_terminal_summary         L0064 INFO   | Can not get Allure report URL. Please check logs
====================================================================================== short test summary info =======================================================================================
PASSED pfcwd/test_pfcwd_function.py::TestPfcwdFunc::test_pfcwd_actions[mth-t0-64]
PASSED pfcwd/test_pfcwd_function.py::TestPfcwdFunc::test_pfcwd_multi_port[mth-t0-64]
PASSED pfcwd/test_pfcwd_function.py::TestPfcwdFunc::test_pfcwd_mmu_change[mth-t0-64]
PASSED pfcwd/test_pfcwd_function.py::TestPfcwdFunc::test_pfcwd_port_toggle[mth-t0-64]
============================================================================= 4 passed, 1 warning in 2461.26s (0:41:01) ==============================================================================
sonic@3852cf4ada95:/data/tests$ 

Any platform specific information?

This is specific to cisco-8000.

[rraghav-cisco]

@kevinskwang : Randall has approved the PR. Pls review.

[rraghav-cisco]

@kevinskwang , I have updated this PR to include only the traffic-pfcwd-triggered test. I will generate another PR with a different test which verifies that the pfcwd is not triggered without traffic.

[rraghav-cisco]

@lipxu , I have run both with and without the -e --fake-storm=True option:=========================================================================================================== PASSES =========================================================================================================== ________________________________________________________________________________________ TestPfcwdFunc.test_pfcwd_actions[mth-t0-64] _________________________________________________________________________________________ _______________________________________________________________________________________ TestPfcwdFunc.test_pfcwd_multi_port[mth-t0-64] _______________________________________________________________________________________ _______________________________________________________________________________________ TestPfcwdFunc.test_pfcwd_mmu_change[mth-t0-64] _______________________________________________________________________________________ ______________________________________________________________________________________ TestPfcwdFunc.test_pfcwd_port_toggle[mth-t0-64] _______________________________________________________________________________________ ---------------------------------------------------------------------------- generated xml file: /run_logs/logs/pfcwd/tr_2024-04-26-22-11-55.xml ----------------------------------------------------------------------------- INFO:root:Can not get Allure report URL. Please check logs --------------------------------------------------------------------------------------------------- live log sessionfinish --------------------------------------------------------------------------------------------------- 22:46:51 init.pytest_terminal_summary L0067 INFO | Can not get Allure report URL. Please check logs ================================================================================================== short test summary info =================================================================================================== PASSED pfcwd/test_pfcwd_function.py::TestPfcwdFunc::test_pfcwd_actions[mth-t0-64] PASSED pfcwd/test_pfcwd_function.py::TestPfcwdFunc::test_pfcwd_multi_port[mth-t0-64] PASSED pfcwd/test_pfcwd_function.py::TestPfcwdFunc::test_pfcwd_mmu_change[mth-t0-64] PASSED pfcwd/test_pfcwd_function.py::TestPfcwdFunc::test_pfcwd_port_toggle[mth-t0-64] ========================================================================================= 4 passed, 1 warning in 2094.13s (0:34:54) ========================================================================================== sonic@5e2e7ccf2d63:/data/tests$

[rraghav-cisco]

@kevinskwang , @lipxu : Pls review.

[kevinskwang]

@kevinskwang , I have updated this PR to include only the traffic-pfcwd-triggered test. I will generate another PR with a different test which verifies that the pfcwd is not triggered without traffic.

I didn't see you send background traffic in your new PR

[rraghav-cisco]

@kevinskwang , I have updated this PR to include only the traffic-pfcwd-triggered test. I will generate another PR with a different test which verifies that the pfcwd is not triggered without traffic.

I didn't see you send background traffic in your new PR

@kevinskwang : Yes, no background traffic is present for pfcwd_function. WD is triggered as soon as we send the first packet - using the traffic_inst.verify_tx_egress(self.tx_action).

[mssonicbld]

Cherry-pick PR to 202311: https://github.com/sonic-net/sonic-mgmt/pull/12757

[mssonicbld]

Cherry-pick PR to 202305: https://github.com/sonic-net/sonic-mgmt/pull/12758