[logrotate]: Relax APPL_DB check tolerance

[theasianpianist]

Description of PR

Summary: Fixes #12612

Type of change

• [x] Bug fix
• [ ] Testbed and Framework(new/improvement)
• [ ] Test case(new/improvement)

Back port request

• [ ] 201911
• [ ] 202012
• [ ] 202205
• [ ] 202305
• [ ] 202311

Approach

What is the motivation for this PR?

The orchagent log rotate notification test can fail if orchagent is busy executing many operations (such as during system init after a config reload). This leads to a delay in processing APPL_DB notifications. If the test case checks APPL_DB during the delay, the test will fail even if orchagent later does process the notification correctly.

How did you do it?

Add 5 seconds of tolerance to the test case

How did you verify/test it?

Run the entire test module. The test case which runs before test_orchagent_logrotate calls config_reload during teardown. Prior to the fix, this would result in test_orchagent_logrotate failing. After the fix, all tests pass consistently.

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

[mssonicbld]

The pre-commit check detected issues in the files touched by this pull request. The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results: trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed
check yaml...........................................(no files to check)Skipped
check for added large files..............................................Passed
check python ast.........................................................Passed
flake8...................................................................Failed
- hook id: flake8
- exit code: 1

tests/syslog/test_logrotate.py:312:5: F841 local variable 'pending_entries' is assigned to but never used
tests/syslog/test_logrotate.py:314:50: F821 undefined name 'rand_selected_dut'

flake8...............................................(no files to check)Skipped
check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

1. Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt docker container.
2. Ensure that the pre-commit package is installed:

sudo pip install pre-commit

3. Go to repository root folder
4. Install the pre-commit hooks:

pre-commit install

5. Use pre-commit to check staged file:

pre-commit

6. Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

[congh-nvidia]

@wangxin could you please cherry-pick this to 202405?

[echuawu]

We also are facing the failure of test_orchagent_logrotate[xxx-None-1-5], even we have this PR. Add the wait_until of no_pending_entries could not fix that. The hung and recovered orchagent process would not handle the pending entries. The root cause of the failure was after the config reload in fixture simulate_small_var_log_partition, the switch does not fully recovered, here I created a PR https://github.com/sonic-net/sonic-mgmt/pull/14478 to handle it and the fix was validated PASS at our setup.

[mssonicbld]

Cherry-pick PR to 202405: https://github.com/sonic-net/sonic-mgmt/pull/14532