logdata-anomaly-miner icon indicating copy to clipboard operation
logdata-anomaly-miner copied to clipboard

Replace Deepsource with open source components

Open ernstleierzopf opened this issue 1 year ago • 10 comments

Following modules should be included in the CI-Pipeline:

  • Bandit(Security-Checks)
  • mypy (already included)
  • prospector(wrapper for: pylint, pep8, pep257, pyflakes, mccabe, dodgy, pyroma, vulture) -> flake8 > pyflakes, mccabe includes flake8,
  • trivy (Security-Checks for the Repository)
  • flake8

Whitelisting comments must be replaced with the ones from the new components.

ernstleierzopf avatar Mar 18 '24 18:03 ernstleierzopf

not sure if using wrappers like prospector is a good idea for us. Keeping it modular for the moment.

ernstleierzopf avatar Mar 25 '24 12:03 ernstleierzopf

not sure if using wrappers like prospector is a good idea for us. Keeping it modular for the moment.

what are your doubts?

whotwagner avatar Mar 25 '24 12:03 whotwagner

not sure if using wrappers like prospector is a good idea for us. Keeping it modular for the moment.

what are your doubts?

I got scared after running it locally within a pre-commit hook. Pylint seems to be too pedantic about issues where none exist. If we use prospector (or other wrapper tools), it is not easily possible to just turn off Pylint. However, it is very easy to include the wanted tool separately.

ernstleierzopf avatar Mar 25 '24 13:03 ernstleierzopf

not sure how to run trivy: https://github.com/aquasecurity/trivy There is no apt package for it right? Also no pre-commit hook

ernstleierzopf avatar Mar 25 '24 13:03 ernstleierzopf

also probably overkill to run with every test..

ernstleierzopf avatar Mar 25 '24 13:03 ernstleierzopf

I just want to remind you that this task is assigned to me. As far as I know you can disable pylint with:

pylint:
  run: false

And each tool can be configured separately. see https://prospector.landscape.io/en/latest/profiles.html

whotwagner avatar Mar 25 '24 13:03 whotwagner

But you doubts let me thing about the advantages of using prospector other than having one single config-file? Maybe we can really spare it.

whotwagner avatar Mar 25 '24 13:03 whotwagner

Deepsource seems to be running now, but I would still progress in replacing it with open-source tools.

ernstleierzopf avatar Mar 25 '24 13:03 ernstleierzopf

I just want to remind you that this task is assigned to me. As far as I know you can disable pylint with:

pylint:
  run: false

And each tool can be configured separately. see https://prospector.landscape.io/en/latest/profiles.html

sorry, I thought I can do it as it requires removing all of the deepsource comments and creation of stages in the jenkinsfile. I am already almost finished with it..

Please review the changes when I open the PR.

ernstleierzopf avatar Mar 25 '24 14:03 ernstleierzopf

got mypy, bandit and flake8 tests in the Jenkins CI and also extended the pre-commit hook, notably with PEP257 checks with docformatter. There are also quite some changes to the format of docstrings..

ernstleierzopf avatar Mar 25 '24 16:03 ernstleierzopf