sonic-mgmt icon indicating copy to clipboard operation
sonic-mgmt copied to clipboard

[QoS]Increasing LACP timer for lag ports for broadcom-dnx neighbor EOS host

Open ansrajpu-git opened this issue 1 year ago • 1 comments

Description of PR

Intermittently testQosSaiLossyQueue tests fails due to Port-channel flap on broadcom-dnx T2 Voq chassis. The reason the port-channel goes down is because this test requires disabling TX on the egress port (which is a member of a port-channel) With the huge buffer-size, it takes a longer time to send packets . This will result in the TX LACP packets to stop egressing, so after 3 LACP packets are missed (~90s) on the server side the LAG is torn down.

Issue # https://github.com/sonic-net/sonic-mgmt/issues/11682 Summary: Fixes # (issue)

Type of change

  • [ ] Bug fix
  • [ ] Testbed and Framework(new/improvement)
  • [x] Test case(new/improvement)

Back port request

  • [ ] 202012
  • [x] 202205
  • [ ] 202305
  • [ ] 202311
  • [x] 202405

Approach

What is the motivation for this PR?

Intermittently testQosSaiLossyQueue tests fails due to Port-channel flap

How did you do it?

The lacp timer multiplier on the EOS host is configurable. By default, timeout is 30 secs with a failure tolerance of 3. We changed the multiplier to an increased value to hold the connectivity for some time until all packets are sent. And revert the changes after test case execution.

How did you verify/test it?

Executed qos test cases and verfiy the results.

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

ansrajpu-git avatar Sep 07 '24 00:09 ansrajpu-git

The pre-commit check detected issues in the files touched by this pull request. The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results: trim trailing whitespace.................................................Passed
fix end of files.........................................................Failed
- hook id: end-of-file-fixer
- exit code: 1
- files were modified by this hook

Fixing tests/common/devices/eos.py
Fixing tests/qos/qos_sai_base.py

check yaml...........................................(no files to check)Skipped
check for added large files..............................................Passed
check python ast.........................................................Passed
flake8...................................................................Failed
- hook id: flake8
- exit code: 1

tests/common/devices/eos.py:553:26: E712 comparison to True should be 'if cond is True:' or 'if cond:'
tests/common/devices/eos.py:553:52: E712 comparison to False should be 'if cond is False:' or 'if not cond:'

flake8...............................................(no files to check)Skipped
...
[truncated extra lines, please run pre-commit locally to view full check results]

To run the pre-commit checks locally, you can follow below steps:

  1. Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt docker container.
  2. Ensure that the pre-commit package is installed:
sudo pip install pre-commit
  1. Go to repository root folder
  2. Install the pre-commit hooks:
pre-commit install
  1. Use pre-commit to check staged file:
pre-commit
  1. Alternatively, you can check committed files using:
pre-commit run --from-ref <commit_id> --to-ref <commit_id>

mssonicbld avatar Sep 07 '24 00:09 mssonicbld

@vmittal-msft .Please review.

ansrajpu-git avatar Sep 10 '24 21:09 ansrajpu-git

The pre-commit check detected issues in the files touched by this pull request. The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results: trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed
check yaml...........................................(no files to check)Skipped
check for added large files..............................................Passed
check python ast.........................................................Passed
flake8...................................................................Failed
- hook id: flake8
- exit code: 1

tests/qos/qos_sai_base.py:2515:5: E303 too many blank lines (2)

flake8...............................................(no files to check)Skipped
check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

  1. Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt docker container.
  2. Ensure that the pre-commit package is installed:
sudo pip install pre-commit
  1. Go to repository root folder
  2. Install the pre-commit hooks:
pre-commit install
  1. Use pre-commit to check staged file:
pre-commit
  1. Alternatively, you can check committed files using:
pre-commit run --from-ref <commit_id> --to-ref <commit_id>

mssonicbld avatar Oct 08 '24 14:10 mssonicbld

can we run this test on dest being non portchanne ?

abdosi avatar Oct 30 '24 17:10 abdosi

@judyjoseph ,@arlakshm , in prior discussion we concluded not to skip portchannel. Please let me know your view.

ansrajpu-git avatar Nov 06 '24 17:11 ansrajpu-git

/Azp run Azure.sonic-mgmt

arlakshm avatar Nov 06 '24 20:11 arlakshm

Azure Pipelines successfully started running 1 pipeline(s).

azure-pipelines[bot] avatar Nov 06 '24 20:11 azure-pipelines[bot]

The pre-commit check detected issues in the files touched by this pull request. The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results: trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed
check yaml...........................................(no files to check)Skipped
check for added large files..............................................Passed
check python ast.........................................................Passed
flake8...................................................................Failed
- hook id: flake8
- exit code: 1

tests/qos/qos_sai_base.py:2551:5: E303 too many blank lines (2)

flake8...............................................(no files to check)Skipped
check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

  1. Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt docker container.
  2. Ensure that the pre-commit package is installed:
sudo pip install pre-commit
  1. Go to repository root folder
  2. Install the pre-commit hooks:
pre-commit install
  1. Use pre-commit to check staged file:
pre-commit
  1. Alternatively, you can check committed files using:
pre-commit run --from-ref <commit_id> --to-ref <commit_id>

mssonicbld avatar Nov 06 '24 20:11 mssonicbld

@ansrajpu-git, can you fix the pre-commit failures?

arlakshm avatar Nov 07 '24 00:11 arlakshm

@arlakshm, please review all checked passed now.

ansrajpu-git avatar Nov 12 '24 22:11 ansrajpu-git

@ansrajpu-git Looks like this change won't work if fanout is non EOS ?

vmittal-msft avatar Nov 19 '24 01:11 vmittal-msft

@ansrajpu-git Looks like this change won't work if fanout is non EOS ?

@vmittal-msft , setting Lacp timer is not supported for vsonic neighbors

ansrajpu-git avatar Nov 19 '24 19:11 ansrajpu-git

@ansrajpu-git how about any other HW vendor fanout switches?

vmittal-msft avatar Nov 20 '24 00:11 vmittal-msft

@ansrajpu-git how about any other HW vendor fanout switches?

@vmittal-msft , For EOS based VMs, these changes should work. For vsonic based VMs, we are skipping the test. These changes are only for broadcom-dnx platform.

ansrajpu-git avatar Nov 20 '24 18:11 ansrajpu-git

/Azp Azure.sonic-mgmt

arlakshm avatar Nov 25 '24 22:11 arlakshm

Command 'Azure.sonic-mgmt' is not supported by Azure Pipelines.

Supported commands

  • help:
    • Get descriptions, examples and documentation about supported commands
    • Example: help "command_name"
  • list:
    • List all pipelines for this repository using a comment.
    • Example: "list"
  • run:
    • Run all pipelines or specific pipelines for this repository using a comment. Use this command by itself to trigger all related pipelines, or specify specific pipelines to run.
    • Example: "run" or "run pipeline_name, pipeline_name, pipeline_name"
  • where:
    • Report back the Azure DevOps orgs that are related to this repository and org
    • Example: "where"

See additional documentation.

azure-pipelines[bot] avatar Nov 25 '24 22:11 azure-pipelines[bot]

@yejianquan can you please approve this change for 202405. This changes is for DNX platforms only

arlakshm avatar Nov 26 '24 03:11 arlakshm

Pending on #15778

bingwang-ms avatar Nov 30 '24 17:11 bingwang-ms

@arlakshm, please add tag for 202205 as well.

ansrajpu-git avatar Dec 02 '24 17:12 ansrajpu-git

@Javier-Tan this is the prerequisite for #16224, can you cherry pick this into 202405-msft? Thanks.

@rlhui @vmittal-msft @yejianquan for viz.

sdszhang avatar Jan 27 '25 03:01 sdszhang

@Javier-Tan this is the prerequisite for #16224, can you cherry pick this into 202405-msft? Thanks.

@rlhui @vmittal-msft @yejianquan for viz.

Ack

Javier-Tan avatar Jan 27 '25 04:01 Javier-Tan

only for chassis, only go to 202405-msft rather than 202405

yejianquan avatar Jan 29 '25 01:01 yejianquan