Handle max_num_warnings properly

[zyaoj]

What does this PR do? Please describe:

The max_num_warnings tests have been skipped and it does not function as expected.

Does your PR introduce any breaking changes? If yes, please list them:

N/A

Check list:

• [ ] Was the content of this PR discussed and approved via a GitHub issue? (no need for typos or documentation improvements)
• [x] Did you read the contributor guideline?
• [x] Did you make sure that your PR does only one thing instead of bundling different changes together?
• [ ] Did you make sure to update the documentation with your changes? (if necessary)
• [x] Did you write any new necessary tests?
• [x] Did you verify new and existing tests pass locally with your changes?
• [ ] Did you update the CHANGELOG? (no need for typos, documentation, or minor internal changes)