Dependency Track deduplication not working

[8L4ckc0FF33]

Bug description I integrated DefectDojo per project into Dependency Track as described here: https://docs.dependencytrack.org/integrations/defectdojo/ By this a report is sent from DependencyTrack to DefectDojo every hour. Looking at the Engagement-view (https://defectdojo.example.com/engagement/1?page=1) (Screenshot) the findings are marked as duplicates in the 6th column. But when trying to filter and deduplicate the findings (Filter: "Duplicate - No") in the Findings-view (https://defectdojo.example.com/product/1/finding/open) (Screenshot) nothing happens. Filter "Duplicate - Either" doesn't change anything either, and by selecting "Yes" no results are shown.

Steps to reproduce Integrate DefectDojo per project into DependecyTrack as described here: https://docs.dependencytrack.org/integrations/defectdojo/. Upload a few reports and watch the findings in the Findings-view. Then use the filter "Duplicate - No".

Expected behavior When using the filter "Duplicate - No" no duplicates should be shown in the view.

Deployment method (select with an X)

• [x] Kubernetes
• [ ] Docker
• [ ] setup.bash / legacy-setup.bash

Environment information DefectDojo v. 2.11.0-dev Dependency Track v4.2.2

Screenshots

[damiencarol]

@8L4ckc0FF33 did you activated the de-duplication in the system settings?

[8L4ckc0FF33]

@damiencarol Yes, but only in the system settings. I also read something about an environmental variable, bu I don't konw if this is still necessary.

[StefanFl]

@8L4ckc0FF33 Your first screenshot with the Tests shows the first 4 tests in the list have duplicates, did you change something in the settings before these 4 tests?

Another thing I have noticed: The Open Findings list has a preselection for active findings, the filter for duplicates doesn't find anything, because duplicate findings are not active by default. Please try it again with the All Findings list.

[damiencarol]

I think @StefanFl is right, your data are here, you just need to use the good filter in the good view

[8L4ckc0FF33]

@StefanFl That's the weird thing. I didn't change any settings since i set up the tool.

When switching to "all findings"-view, filtering by "Duplicate: No" reduces the list from 2407 to 657, "Duplicate: Yes" from 2407 to 1750 and "Duplicate: Either" does nothing. But even if the number of vulnerabilities drop, there are still a lot of duplicates, since Dependency Track sends the same vulnerability report every hour. I'd expect to see about 10 findings after deduplication.

@damiencarol Could you specify what a "good view" and a "good filter" is for deduplication? Are there further configurations that have to be made? My naive opinion was, that vulnerabilities with the same CVE-number will be deduplicated without any further specifications.

[dzmitry-savitski]

@8L4ckc0FF33 , I had the same issue and resolved it by modifying the DEDUPLICATION_ALGORITHM_PER_PARSER in local_settings. I have the following line: 'Dependency Track Finding Packaging Format (FPF) Export': DEDUPE_ALGO_UNIQUE_ID_FROM_TOOL_OR_HASH_CODE,

[damiencarol]

My naive opinion was, that vulnerabilities with the same CVE-number will be deduplicated without any further specifications.

Sadly this is not how the data model of DefectDojo is designed. CVE are not unique by product+test. Some parser also generate more than one findings for one CVE (because in some case, the same CVE impact few files+line number and you want to track them all)

[8L4ckc0FF33]

@dzmitry-savitski Thank you, I will give it a try. I just restarted everything to have a clean start. btw: How do I switch the algorithm in k8s?

@damiencarol Ok, yes I could reproduce that and it makes sense. But still I haven't figured out the deduplication completely: As I said I restarted everything and I have one product, with two engagements and 1+2 tests. Every test is the same OWASP ZAP report. The report consists of 9 findings. When looking at all findings I got 27 but without any duplication? Shouldn't there be only 9 findings?

[8L4ckc0FF33]

Thanks to @dzmitry-savitski i figured something out: The deduplication happens on parsing but not inside the view of Defectdojo? So when having duplicate reports inside Defectdojo without having deduplication working during parsing, there will not be any deduplication afterwards?

[dzmitry-savitski]

@8L4ckc0FF33 I believe it works only during parsing inside a celery/worker pod, but I might be wrong. For k8s, it's a bit tricky. I have a config map with local settings I want to change, and then I mount it to django/uwsgi and celery deployments via the following helm configs:

---
celery:
  extraVolumes:
    - type: configMap
      name: local-settings
      path: /app/dojo/settings/local_settings.py
      subPath: local_settings.py
---
django:
  extraVolumes:
    - type: configMap
      name: local-settings
      path: /app/dojo/settings/local_settings.py
      container: uwsgi
      subPath: local_settings.py

Here's how I create a configmap:

kubectl create configmap local-settings --from-file=./defectdojo/settings/local_settings.py --dry-run=client -o yaml > ./defectdojo/k8s/local_settings_map.yaml