Add tests for --max-warnings

[fvictorio]

This can be a unit test (assuming this is something unit-testable, otherwise it might require a refactor) or a e2e (which we don't have right now, but it would be nice to add).