Base implementation of a Vulnerability models #94

[tdruez]

This PR is the first step in the CRAVEX implementation. See https://github.com/nexB/dejacode/issues/94

• It introduces a new Vulnerability model and all the code logic to fetch and create Vulnerability records and assign those to Package/Component through ManyToMany relationships.
• A new fetchvulnerabilities management command is available to fetch all the relevant data from VulnerableCode for a given Dataspace.
• A scheduler was added to run the vulnerability data update daily (we can discuss and adjust this to the most suitable value, depending on how often VCIO is updated for example).
• The latest vulnerability data refresh date is displayed in the Admin dashboard in a new "Data updates" section in the botton right corner.
• The Package/Component views that display vulnerability information (icon or tab) are now using the data from the Vulnerability model in place of calling the VulnerableCode API on each request. This result into much better performances as we do not depend on the VulnerableCode performances to render the DejaCode view anymore. Also, this will make Vulnerability data available in the Reporting system.
• A filter is available next to the "Identifier" column header in the Package list view, and Product tabs.
• The vulnerability icon is displayed next to the Package/Component identifier in the Product views: "Inventory", "Hierarchy", "Dependencies" tabs.
• The vulnerability data is available in Reporting either through the is_vulnerable property on Package/Component column template or going through the full affected_by_vulnerabilities m2m field. This is available in both Query and ColumnTemplate. Query example: Package > affected_by_vulnerabilities > IS_NULL = False

Scheduler:

• Add a scheduler service to run the vulnerability updates on a daily basis: https://github.com/aboutcode-org/dejacode/pull/148/commits/5978612140443057e5aa8df2a19b3fb3be019831 https://github.com/rq/rq-scheduler?tab=readme-ov-file#cron-jobs https://github.com/rq/django-rq?tab=readme-ov-file#support-for-scheduled-jobs

---

TODO:

• [x] Update the ProductTabInventoryView and all Product related views in general to the new system
• [x] Display the vulnerability icon in Herarchy/Dependency tabs
• [x] https://github.com/nexB/dejacode/issues/2
• [x] Add a scheduler to run the vulnerability updates on a daily basis using RQScheduler.
• [x] Fix and add tests
• [ ] On Package/Component addition (Add package, Import, etc...) lookup and create the vulnerabilities
• [ ] Complete the management command

[DennisClark]

Hi @tdruez I checked everything you mentioned in Staging, and it all works very nicely.

My one suggestion at this point is to add a little bug icon next to the filter button in the Identifier column header on the Packages list view (also on Components) similar to what you did on the Inventory tab of the Product view.

[tdruez]

@DennisClark Thanks for the review and suggestions.

Added:

• Icon for all dropdown filter related to vulnerabilities
• Vulnerability data are now fetch on Package creation and modification: "Add package", "Add package form", "Admin form".

For example, click on "Add package" and provide one or more vulnerable purls: pkg:pypi/[email protected]. All vulnerability data are now fetched during the data collection.

[DennisClark]

@tdruez the new icon placement looks good, thanks. The vulnerability filter is working fine on the Packages list, but when I click on it on any Product Inventory, it seems to go into an endless loop (it was working fine yesterday). Perhaps a problem on the VulnerableCode side?

[tdruez]

@DennisClark just a bug, I've pushed a fix for this one.

[DennisClark]

@tdruez I encountered something very unusual while testing the enhanced Add Package feature. When I use that in the nexB dataspace, I can create a new Package for pkg:pypi/[email protected] and it creates it just fine and finds the vulnerabilities. However, when I try the same thing in the Starship dataspace I get Error: error in the Add Package form. Both dataspaces use the same Integrations and show a good status. I also tried the following values in Starship and continue to get the same Error: error result:

https://repo1.maven.org/maven2/org/apache/commons/commons-compress/1.9/commons-compress-1.9.jar

pkg:maven/org.apache.commons/[email protected]

That package is already defined in the nexB dataspace but that should not be a problem in Starship. Please investigate when you get a chance.

[tdruez]

@DennisClark I've deployed the latest improvements for you to review. The set of features is now complete.

Changes:

• The issue reporting in https://github.com/aboutcode-org/dejacode/pull/148#issuecomment-2289236632 is fixed
• The scheduler service is in place (vulnerability data update run daily)
• Display the latest data update on the integration status page
• The vulnerabilities are fetched each time a Package is created/modified (note that a purl is required on the package for the lookup), either through:
  • Add package (providing a purl)
  • Add package form
  • Import package
  • Add package Admin
• Also, the whole Packages of a Product are updated with latest vulnerabilities from VCIO following importing data in Product using:
  • Import data from Scan
  • Load Packages from SBOMs
  • Import Packages from manifests
  • Pull ScanCode.io Project data

[DennisClark]

Fix to the problem creating a new package in a non-reference dataspace confirmed in Staging.

[DennisClark]

@tdruez everything looks good to me, no problems found!