More control over exit code

[kbasgall]

In Azure Pipelines, the exit code of a command determines the success or failure of a task. If any task in the pipeline fails, the entire pipeline run shows as a failure. We would like our Bandit task (and therefore the build) to fail if high severity and high confidence warnings are found, however we want the task (and the build) to succeed if there are low or medium severity warnings.

We would like a command line option or entry in the config file to allow users to choose exactly how Bandit will exit. This will allow users to much more easily integrate Bandit into their CI pipelines without the need for extra scripts. Since by default all warnings will cause Bandit to exit 1, we would like a command line option to specify which levels to exit 0 on. For example, --exit-zero-severity l would cause low severity warnings to exit zero, and --exit-zero-confidence mh would cause medium and high confidence warnings to exit zero.

We've considered only reporting high severity warnings using -lll, however we still want to see the low and medium warnings in the artifact and this flag would not display them. We could also force Bandit to always return 0, but then we lose the visibility of having high severity items fail in the pipeline.

[maciejstromich]

As the author of --exit-zero PR I can implement this enhancement fairly easy. I've few questions though.

1. should this maybe be implemented in a way that choosing medium level is also choosing lower level results implicitly (--exit-zero-severity m would mean that both low and medium level results are considered)? or should this be explicit (to choose low and medium severity results --exit-zero-severity lm would have to be provided)?

2. should using --exit-zero-severity require --exit-zero-confidence to be provided as well as a required field?

3. should using --exit-zero-confidence be allowed as a standalone option?

4. and last but not least should the values be lmh or be treated as severity_string and confidence_string (more verbose all, low, medium, high)

[kbasgall]

Hello, thanks for the reply! To answer your questions:

1. I really could go either way here. Choosing lower level results implicitly would be perfect for our use case. However, the extra level of customization might not be a bad thing.
2. No. I want to be able to enter just --exit-zero-severity l and use default confidence exit behavior. I think --exit-zero-severity would be more used than --exit-zero-confidence so I'd rather not have to enter that option every time.
3. Not for my use case specifically, but yes.
4. I prefer lmh, because it makes it easier to select 2 levels, i.e. lm. How would that look for the severity_string option? lowmedium?

[maciejstromich]

https://github.com/PyCQA/bandit/pull/715

[falnyr]

Not sure if this is still an active issue. My current solution to reporting only on high severity using jq:

bandit -r . -f json | jq -e '.metrics._totals.["SEVERITY.HIGH"] == 0'