Macsec scale test on a t2 topo

[judyjoseph]

Summary: Fixes # 13244

Type of change

• [ ] Bug fix
• [ ] Testbed and Framework(new/improvement)
• [ ] Test case(new/improvement)

Back port request

• [ ] 202012
• [ ] 202205
• [ ] 202305
• [ ] 202311
• [ ] 202405

Approach

What is the motivation for this PR?

• Basic infra and fine tuning needed to run macsec sessions on a linecard in a chassis with 32 macsec sessions.
• Support more tests on dnx based platforms which has limitation now with portchannel
• Add config save at startup and cleanup of macsec sessions, so that macsec sessions remain present in config_db for a pipeline run of the entire sonic-mgmt testsuite
• Add a testcase to trigger link flap in 16+ interface and see macsec sessions comes back stable.

How did you do it?

Added logic and finetuning

How did you verify/test it?

Ran the macsec testsuite to get 100% pass on a testbed with LC having 32 macsec sessions.

jujoseph@sonic-mgmt-jujoseph_master:/var/src/sonic-mgmt-int/tests$ ./run_tests.sh -u -n vmsvc3-t2-xxxx-1 -c macsec/test_deployment.py::**TestDeployment::test_scale_rekey** -f ../ansible/testbed.yaml -i ../ansible/strsvc,../ansible/veos -t 't2,any' -e "--skip_sanity --disable_loganalyzer -x --enable_macsec --macsec_profile=MACSEC_PROFILE"                                                                                                                                      
=== Running tests in groups ===
Running: python2 -m pytest macsec/test_deployment.py::TestDeployment::test_scale_rekey --inventory ../ansible/strsvc,../ansible/veos --host-pattern svcstr-xxxx-lc1-1,svcstr-xxxx-lc2-1,svcstr-xxxx-lc3-1,svcstr-xxxx-sup-1 --testbed vmsvc3-t2-xxxx-1 --testbed_file ../ansible/testbed.yaml --log-cli-level warning --log-file-level debug --kube_master unset --showlocals --assert plain --show-capture no -rav --allow_recover --ignore=ptftests --ignore=acstests --ignore=saitests --ignore=scripts --ignore=k8s --ignore=sai_qualify --junit-xml=logs/tr.xml --log-file=logs/test.log --topology t2,any --skip_sanity --disable_loganalyzer -x --enable_macsec --macsec_profile=MACSEC_PROFILE
/usr/local/lib/python2.7/dist-packages/ansible/parsing/vault/__init__.py:44: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release.
  from cryptography.exceptions import InvalidSignature
=================================================================================== test session starts ====================================================================================
platform linux2 -- Python 2.7.18, pytest-4.6.11, py-1.11.0, pluggy-0.13.1
ansible: 2.8.20
rootdir: /var/src/sonic-mgmt-int/tests, inifile: pytest.ini
plugins: ansible-2.2.4, celery-4.4.7, html-1.22.1, repeat-0.9.1, forked-1.3.0, metadata-1.11.0, xdist-1.28.0, allure-pytest-2.8.22
collecting ... 
----------------------------------------------------------------------------------- live log collection ------------------------------------------------------------------------------------
00:28:17 testbed.get_testbed_type                 L0261 WARNING| Unsupported testbed type - dpu
00:28:17 testbed.get_testbed_type                 L0261 WARNING| Unsupported testbed type - dpu
00:28:17 testbed.get_testbed_type                 L0261 WARNING| Unsupported testbed type - dpu
00:28:28 testbed.get_testbed_type                 L0261 WARNING| Unsupported testbed type - ptp-256
00:28:28 testbed.get_testbed_type                 L0261 WARNING| Unsupported testbed type - ptp-256
00:29:03 testbed.get_testbed_type                 L0261 WARNING| Unsupported testbed type - dpu
00:29:03 testbed.get_testbed_type                 L0261 WARNING| Unsupported testbed type - dpu
00:29:03 testbed.get_testbed_type                 L0261 WARNING| Unsupported testbed type - dpu
00:29:14 testbed.get_testbed_type                 L0261 WARNING| Unsupported testbed type - ptp-256
00:29:14 testbed.get_testbed_type                 L0261 WARNING| Unsupported testbed type - ptp-256
             PASSED                                                                                                   [100%] 

-------------------------------------------------------------- generated xml file: /var/src/sonic-mgmt-int/tests/logs/tr.xml ---------------------------------------------------------------
=============================================================================== 1 passed in 4292.18 seconds ==============================================================================

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

[mssonicbld]

The pre-commit check detected issues in the files touched by this pull request. The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results: trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed
check yaml...............................................................Passed
check for added large files..............................................Passed
check python ast.........................................................Passed
flake8...................................................................Failed
- hook id: flake8
- exit code: 1

tests/macsec/macsec_helper.py:126:20: F821 undefined name 're'
tests/macsec/macsec_platform_helper.py:53:71: E271 multiple spaces after keyword
tests/macsec/macsec_platform_helper.py:53:121: E501 line too long (128 > 120 characters)
tests/macsec/test_fault_handling.py:99:11: E271 multiple spaces after keyword
tests/macsec/test_interop_protocol.py:10:1: F811 redefinition of unused 'find_portchannel_from_member' from line 8
tests/macsec/test_interop_protocol.py:57:13: E265 block comment should start with '# '
tests/macsec/test_interop_protocol.py:107:13: E265 block comment should start with '# '
tests/macsec/test_interop_protocol.py:125:13: E265 block comment should start with '# '

flake8...............................................(no files to check)Skipped
check conditional mark sort..............................................Passed

To run the pre-commit checks locally, you can follow below steps:

1. Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt docker container.
2. Ensure that the pre-commit package is installed:

sudo pip install pre-commit

3. Go to repository root folder
4. Install the pre-commit hooks:

pre-commit install

5. Use pre-commit to check staged file:

pre-commit

6. Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

[mssonicbld]

The pre-commit check detected issues in the files touched by this pull request. The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results: trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed
check yaml...............................................................Passed
check for added large files..............................................Passed
check python ast.........................................................Passed
flake8...................................................................Failed
- hook id: flake8
- exit code: 1

tests/macsec/macsec_helper.py:126:20: F821 undefined name 're'
tests/macsec/macsec_platform_helper.py:53:71: E271 multiple spaces after keyword
tests/macsec/macsec_platform_helper.py:53:121: E501 line too long (128 > 120 characters)
tests/macsec/test_fault_handling.py:99:11: E271 multiple spaces after keyword
tests/macsec/test_interop_protocol.py:10:1: F811 redefinition of unused 'find_portchannel_from_member' from line 8
tests/macsec/test_interop_protocol.py:57:13: E265 block comment should start with '# '
tests/macsec/test_interop_protocol.py:107:13: E265 block comment should start with '# '
tests/macsec/test_interop_protocol.py:125:13: E265 block comment should start with '# '

flake8...............................................(no files to check)Skipped
check conditional mark sort..............................................Passed

To run the pre-commit checks locally, you can follow below steps:

1. Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt docker container.
2. Ensure that the pre-commit package is installed:

sudo pip install pre-commit

3. Go to repository root folder
4. Install the pre-commit hooks:

pre-commit install

5. Use pre-commit to check staged file:

pre-commit

6. Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

[mssonicbld]

The pre-commit check detected issues in the files touched by this pull request. The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results: trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed
check yaml...............................................................Passed
check for added large files..............................................Passed
check python ast.........................................................Passed
flake8...................................................................Failed
- hook id: flake8
- exit code: 1

tests/macsec/macsec_helper.py:126:20: F821 undefined name 're'
tests/macsec/macsec_platform_helper.py:53:71: E271 multiple spaces after keyword
tests/macsec/macsec_platform_helper.py:53:121: E501 line too long (128 > 120 characters)
tests/macsec/test_deployment.py:30:5: E303 too many blank lines (2)
tests/macsec/test_deployment.py:41:93: E225 missing whitespace around operator
tests/macsec/test_deployment.py:41:95: F821 undefined name 'get_appl_db'
tests/macsec/test_deployment.py:46:9: F821 undefined name 'sleep'
tests/macsec/test_deployment.py:48:9: E265 block comment should start with '# '
tests/macsec/test_deployment.py:55:106: F821 undefined name 'get_appl_db'
tests/macsec/test_deployment.py:63:9: E303 too many blank lines (2)
tests/macsec/test_deployment.py:64:9: F821 undefined name 'sleep'
...
[truncated extra lines, please run pre-commit locally to view full check results]

To run the pre-commit checks locally, you can follow below steps:

1. Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt docker container.
2. Ensure that the pre-commit package is installed:

sudo pip install pre-commit

3. Go to repository root folder
4. Install the pre-commit hooks:

pre-commit install

5. Use pre-commit to check staged file:

pre-commit

6. Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

[StormLiangMS]

/azp run Azure.sonic-mgmt

[azure-pipelines[bot]]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

The pre-commit check detected issues in the files touched by this pull request. The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results: trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed
check yaml...............................................................Passed
check for added large files..............................................Passed
check python ast.........................................................Passed
flake8...................................................................Failed
- hook id: flake8
- exit code: 1

tests/macsec/macsec_helper.py:126:20: F821 undefined name 're'
tests/macsec/macsec_platform_helper.py:53:71: E271 multiple spaces after keyword
tests/macsec/macsec_platform_helper.py:53:121: E501 line too long (128 > 120 characters)
tests/macsec/test_deployment.py:30:5: E303 too many blank lines (2)
tests/macsec/test_deployment.py:41:93: E225 missing whitespace around operator
tests/macsec/test_deployment.py:41:95: F821 undefined name 'get_appl_db'
tests/macsec/test_deployment.py:46:9: F821 undefined name 'sleep'
tests/macsec/test_deployment.py:48:9: E265 block comment should start with '# '
tests/macsec/test_deployment.py:55:106: F821 undefined name 'get_appl_db'
tests/macsec/test_deployment.py:63:9: E303 too many blank lines (2)
tests/macsec/test_deployment.py:64:9: F821 undefined name 'sleep'
...
[truncated extra lines, please run pre-commit locally to view full check results]

To run the pre-commit checks locally, you can follow below steps:

1. Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt docker container.
2. Ensure that the pre-commit package is installed:

sudo pip install pre-commit

3. Go to repository root folder
4. Install the pre-commit hooks:

pre-commit install

5. Use pre-commit to check staged file:

pre-commit

6. Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

[mssonicbld]

The pre-commit check detected issues in the files touched by this pull request. The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results: trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed
check yaml...........................................(no files to check)Skipped
check for added large files..............................................Passed
check python ast.........................................................Passed
flake8...................................................................Failed
- hook id: flake8
- exit code: 1

tests/macsec/test_deployment.py:38:93: E225 missing whitespace around operator
tests/macsec/test_deployment.py:38:95: F821 undefined name 'get_appl_db'
tests/macsec/test_deployment.py:43:9: F821 undefined name 'sleep'
tests/macsec/test_deployment.py:45:9: E265 block comment should start with '# '
tests/macsec/test_deployment.py:52:106: F821 undefined name 'get_appl_db'
tests/macsec/test_deployment.py:60:9: E303 too many blank lines (2)
tests/macsec/test_deployment.py:61:9: F821 undefined name 'sleep'
tests/macsec/test_deployment.py:64:91: E225 missing whitespace around operator
tests/macsec/test_deployment.py:64:93: F821 undefined name 'get_appl_db'

flake8...............................................(no files to check)Skipped
...
[truncated extra lines, please run pre-commit locally to view full check results]

To run the pre-commit checks locally, you can follow below steps:

1. Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt docker container.
2. Ensure that the pre-commit package is installed:

sudo pip install pre-commit

3. Go to repository root folder
4. Install the pre-commit hooks:

pre-commit install

5. Use pre-commit to check staged file:

pre-commit

6. Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

[judyjoseph]

/azp run Azure.sonic-mgmt

[azure-pipelines[bot]]

Azure Pipelines successfully started running 1 pipeline(s).

[judyjoseph]

/azp run

[azure-pipelines[bot]]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

The pre-commit check detected issues in the files touched by this pull request. The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results: trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed
check yaml...........................................(no files to check)Skipped
check for added large files..............................................Passed
check python ast.........................................................Passed
flake8...................................................................Failed
- hook id: flake8
- exit code: 1

tests/macsec/test_deployment.py:41:96: F821 undefined name 'get_appl_db'
tests/macsec/test_deployment.py:55:106: F821 undefined name 'get_appl_db'
tests/macsec/test_deployment.py:67:98: F821 undefined name 'get_appl_db'

flake8...............................................(no files to check)Skipped
check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

1. Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt docker container.
2. Ensure that the pre-commit package is installed:

sudo pip install pre-commit

3. Go to repository root folder
4. Install the pre-commit hooks:

pre-commit install

5. Use pre-commit to check staged file:

pre-commit

6. Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

[judyjoseph]

/azp run

[azure-pipelines[bot]]

Azure Pipelines successfully started running 1 pipeline(s).

[judyjoseph]

@StormLiangMS could you please help review and merge this PR

[judyjoseph]

/azp run

[azure-pipelines[bot]]

Azure Pipelines successfully started running 1 pipeline(s).

[judyjoseph]

@arlakshm please review again