Merge does not deduplicate components with differing bom-ref

[wkoot]

Given the following two test files, their merged output does not deduplicate components as mentioned in https://github.com/CycloneDX/cyclonedx-python-lib/issues/540#issuecomment-2343195560

File test/in1.json:

{
  "$schema": "http://cyclonedx.org/schema/bom-1.5.schema.json",
  "bomFormat": "CycloneDX",
  "specVersion": "1.5",
  "metadata": {"component": {"bom-ref": "test11", "type": "container", "name": "alpine"}},
  "components": [{"bom-ref": "test12", "type": "operating-system", "name": "alpine"}],
  "dependencies": [{"ref": "test11", "dependsOn": ["test12"]}]
}

and test/in2.json:

{
  "$schema": "http://cyclonedx.org/schema/bom-1.5.schema.json",
  "bomFormat": "CycloneDX",
  "specVersion": "1.5",
  "metadata": {"component": {"bom-ref": "test21", "type": "container", "name": "alpine"}},
  "components": [{"bom-ref": "test22", "type": "operating-system", "name": "alpine"}],
  "dependencies": [{"ref": "test21", "dependsOn": ["test22"]}]
}

The following merge command:

docker run --rm -v $(pwd)/test:/test cyclonedx/cyclonedx-cli:0.26.0 merge --input-files /test/in1.json /test/in2.json --input-format json --output-format json --output-file /test/out.json --name test

Produces the following output file test/out.json, which contains duplicate components:

{
  "bomFormat": "CycloneDX",
  "specVersion": "1.5",
  "serialNumber": "urn:uuid:66fa5692-2e9d-45c5-830a-ec8ccaf7dcc9",
  "version": 1,
  "metadata": {
    "component": {
      "type": "application",
      "name": "test"
    }
  },
  "components": [
    {
      "type": "operating-system",
      "bom-ref": "test12",
      "name": "alpine"
    },
    {
      "type": "container",
      "bom-ref": "test11",
      "name": "alpine"
    },
    {
      "type": "operating-system",
      "bom-ref": "test22",
      "name": "alpine"
    },
    {
      "type": "container",
      "bom-ref": "test21",
      "name": "alpine"
    }
  ],
  "dependencies": [
    {
      "ref": "test11",
      "dependsOn": [
        "test12"
      ]
    },
    {
      "ref": "test21",
      "dependsOn": [
        "test22"
      ]
    }
  ]
}

[wkoot]

Since the file validation changes introduced to Dependency-Track in version 4.11, files produced by cyclonedx-cli merge in this manner also produce errors in DT:

ERROR [BomUploadProcessingTask] Error while processing bom
java.lang.IllegalStateException: Duplicate key Identity[group=aquasecurity, name=trivy:FilePath, value=home/frontend/node_modules/body-parser/node_modules/debug/package.json] (attempted merging values ComponentProperty{id=97037, component=pkg:npm/[email protected], groupName=aquasecurity, propertyName=trivy:FilePath, propertyValue=home/frontend/node_modules/body-parser/node_modules/debug/package.json, propertyType=STRING, uuid=c7528c5d-c315-4aa8-b259-6010af83c96c} and ComponentProperty{id=101526, component=pkg:npm/[email protected], groupName=aquasecurity, propertyName=trivy:FilePath, propertyValue=home/frontend/node_modules/body-parser/node_modules/debug/package.json, propertyType=STRING, uuid=0a33962a-62c7-4952-a108-f77edfc143d9})
	at java.base/java.util.stream.Collectors.duplicateKeyException(Unknown Source)
	at java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Unknown Source)
	at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(Unknown Source)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source)
[...]