nexB
Exception when creating cyclonedx output
There is an exception when generating cyclonedx output and external references are merged. The error is: seen = set(x) TypeError: unhashable type: 'CycloneDxExternalRef'
It is not reproduceable using sample files but I've managed to slightly modify json output from scanning the samples and got the same error. Json file is attached, please try it with --cyclonedx and --from-json options. I've run it on Windows 10 using python 3.8.10.
Thank you, Janusz
Thanks for the report, I could reproduce from this JSON file you attached, and looking into this.
I have the same problem, when I try to generate a cyclonedx or cyclonedx-xml I get the following error message:
ERROR: failed to run output plugin: cyclonedx:
Traceback (most recent call last): File "scancode-toolkit-v31.1.0\venv\lib\site-packages\scancode\cli.py", line 1077, in run_codebase_plugins plugin.process_codebase(codebase, **kwargs) File "scancode-toolkit-v31.1.0\venv\lib\site-packages\formattedcode\output_cyclonedx.py", line 794, in process_codebase bom = CycloneDxBom.from_codebase(codebase) File "scancode-toolkit-v31.1.0\venv\lib\site-packages\formattedcode\output_cyclonedx.py", line 681, in from_codebase components = list(CycloneDxComponent.from_packages(packages)) File "scancode-toolkit-v31.1.0\venv\lib\site-packages\formattedcode\output_cyclonedx.py", line 339, in from_packages base_component.merge(other_component) File "scancode-toolkit-v31.1.0\venv\lib\site-packages\formattedcode\output_cyclonedx.py", line 375, in merge merge_lists(self.externalReferences, other.externalReferences) File "scancode-toolkit-v31.1.0\venv\lib\site-packages\formattedcode\output_cyclonedx.py", line 431, in merge_lists seen = set(x) TypeError: unhashable type: 'CycloneDxExternalRef'
Using scancode scancode-toolkit-v31.1.1 and phyton 3.8.10
Hi there,
got a similar issue with the cyclonedx-output when scanning a very large package...
Used version:
ScanCode version: 31.0.1
ScanCode Output Format version: 2.0.0
SPDX License list version: 3.17
Download, prepare package and start scanning:
> wget https://download.qt.io/official_releases/qt/5.15/5.15.7/submodules/qtwebengine-everywhere-opensource-src-5.15.7.zip
> unzip qtwebengine-everywhere-opensource-src-5.15.7.zip
> mv qtwebengine-everywhere-src-5.15.7/ qtwebengine-5.15.7/
> scancode -n 2 -clip qtwebengine-5.15.7/ --json-pp qtwebengine-5.15.7.json --cyclonedx qtwebengine-5.15.7_cdxSBOM.json
Gives the following error message:
:
:
ERROR: failed to run output plugin: cyclonedx:
Traceback (most recent call last):
File "/home/scanner/venv3.8-scancode/lib/python3.8/site-packages/scancode/cli.py", line 1077, in run_codebase_plugins
plugin.process_codebase(codebase, **kwargs)
File "/home/scanner/venv3.8-scancode/lib/python3.8/site-packages/formattedcode/output_cyclonedx.py", line 794, in process_codebas
e
bom = CycloneDxBom.from_codebase(codebase)
File "/home/scanner/venv3.8-scancode/lib/python3.8/site-packages/formattedcode/output_cyclonedx.py", line 681, in from_codebase
components = list(CycloneDxComponent.from_packages(packages))
File "/home/scanner/venv3.8-scancode/lib/python3.8/site-packages/formattedcode/output_cyclonedx.py", line 339, in from_packages
base_component.merge(other_component)
File "/home/scanner/venv3.8-scancode/lib/python3.8/site-packages/formattedcode/output_cyclonedx.py", line 370, in merge
merge_lists(self.licenses, other.licenses)
File "/home/scanner/venv3.8-scancode/lib/python3.8/site-packages/formattedcode/output_cyclonedx.py", line 431, in merge_lists
seen = set(x)
TypeError: unhashable type: 'CycloneDxLicenseExpression'
Scanning done.
Some files failed to scan properly:
Path: qtwebengine-5.15.7/src/3rdparty/chromium/third_party/icu/common/icudtb.dat
Path: qtwebengine-5.15.7/src/3rdparty/chromium/third_party/icu/common/icudtl.dat
Path: qtwebengine-5.15.7/src/3rdparty/chromium/third_party/icu/ios/icudtl.dat
Summary: info, licenses, copyrights, packages with 2 process(es)
Errors count: 3
Scan Speed: 2.45 files/sec. 27.69 KB/sec.
Initial counts: 183592 resource(s): 167376 file(s) and 16216 directorie(s)
Final counts: 183592 resource(s): 167376 file(s) and 16216 directorie(s) for 1.80 GB
Timings:
scan_start: 2022-11-21T121128.360738
scan_end: 2022-11-22T072026.947766
setup_scan:licenses: 2.22s
setup: 2.22s
inventory: 185.09s
scan:packages: 180.89s
scan: 68326.72s
output:json-pp: 123.34s
output: 123.37s
total: 69087.68s
Removing temporary files...done.
Maybe this helps...
Regards, Thorsten
@tgodau Thanks for the report. I am planning this for the next milestone
Oops closed by mistake!
@tgodau @jkwasny @MochonDeVuelta could you test the fix? this is in latest develop (and will be released soon!).
@tgodau @jkwasny @MochonDeVuelta Gentle ping, did you get to try this out?
@AyanSinhaMahapatra if you have a test that verifies that this is fixed, let's close this as fixed.
@AyanSinhaMahapatra Maybe you could use the four lines from my description to download, prepare and scan the specific mentioned Qt package where this error appeared...
@tgodau we have added this failure as a test in scancode already, see this commit: 553e624.
Apart from that I've also run the full scan for you, which ran successfully. See attached file below: (The JSON was too big ~500mb to attach)
scancode -n 12 -clip qtwebengine-5.15.7/ --json-pp qtwebengine-5.15.7.json --cyclonedx qtwebengine-5.15.7_cdxSBOM.json --classify --summary --license-text
Setup plugins...
Collect file inventory...
Scan files for: info, packages, licenses, copyrights with 12 process(es)...
[####################] 334752
/home/ayansinha/nexB/write_access/scancode-toolkit/venv/lib/python3.8/site-packages/fingerprints/cleanup.py:54: ICUWarning: Install 'pyicu' for better text transliteration.
text = ascii_text(text)
Scanning done.
Summary: info, packages, licenses, copyrights with 12 process(es)
Errors count: 0
Scan Speed: 7.98 files/sec. 90.23 KB/sec.
Initial counts: 183592 resource(s): 167376 file(s) and 16216 directorie(s)
Final counts: 183592 resource(s): 167376 file(s) and 16216 directorie(s) for 1.80 GB
Timings:
scan_start: 2023-01-18T131259.695160
scan_end: 2023-01-18T191854.259676
setup_scan:licenses: 1.40s
setup: 1.40s
inventory: 55.32s
pre-scan:classify: 76.43s
pre-scan: 76.43s
scan:packages: 77.72s
scan:licenses: 8586.14s
scan: 20964.43s
post-scan:summary: 714.78s
post-scan:license-references: 26.71s
post-scan: 741.50s
output:json-pp: 56.46s
output: 56.47s
total: 22022.89s
qtwebengine-5.15.7_cdxSBOM.json.txt