Manual dependencies using `remote-dependencies` for Node.js not working

[thernstig]

The following in fossa-deps.yaml:

remote-dependencies:
  - name: Node.js
    version: "16.14.2"
    url: https://github.com/nodejs/node/archive/refs/tags/v16.14.2.tar.gz

gives the following error in the UI after it has been scanned when clicking the "FAILED: View Logs" for the failed dependency.

=== STEP 0. INITIALIZING & DOWNLOADING SOURCE CODE === BUILD ID: 583736 CODE LOCATOR: url-private+1/https://github.com/nodejs/node/archive/refs/tags/v16.14.2.tar.gz$16.14.2 USING THEMIS ROOT DIRECTORY: /tmp/tmp-101592-t5Bc7ShAIu9w ====================================================== ANALYSIS DIRECTORY: /tmp/tmp-101592-t5Bc7ShAIu9w ENRICHMENT DIRECTORY: /tmp/tmp-101592-t5Bc7ShAIu9w === STEP 1. BEGINNING CODE ANALYSIS === 2022/08/24 11:11:02 Failed to detect git/hg repository root dir for "."; continuing. Failed to detect git/hg repository root dir for "."; continuing. Running: [/home/fossa/.srclib/srclib-php/.bin/srclib-php scan --repo --subdir .] --> with input {} Running: [/home/fossa/.srclib/srclib-iOS/.bin/srclib-iOS scan --repo --subdir .] --> with input {} Running: [/home/fossa/.srclib/srclib-javascript/.bin/srclib-javascript scan-commonjs --repo --subdir .] --> with input {} Running: [/home/fossa/.srclib/srclib-nomos/.bin/srclib-nomos scan-licenses --repo --subdir .] --> with input {} Running: [/home/fossa/.srclib/srclib-dotnet/.bin/srclib-dotnet scan --repo --subdir .] --> with input {} Running: [/home/fossa/.srclib/srclib-go/.bin/srclib-go scan --repo --subdir .] --> with input {} Running: [/home/fossa/.srclib/srclib-perl/.bin/srclib-perl scan --repo --subdir .] --> with input {} Running: [/home/fossa/.srclib/srclib-java/.bin/srclib-java scan --repo --subdir .] --> with input {} ## No bundled JRE found at /home/fossa/.srclib/srclib-java/.bin/../bundled-jdk/jdk1.8.0_45.ARCH-amd64_OS-linux; using system JRE. Running: [/home/fossa/.srclib/srclib-pip/.bin/srclib-pip scan --repo --subdir .] --> with input {} /home/fossa/.gem/gems/rogdl-0.2.3/lib/fixnum.rb:1: warning: constant ::Fixnum is deprecated Running: [/home/fossa/.srclib/srclib-python/.bin/srclib-python scan --repo --subdir .] --> with input {} 2022-08-24 11:11:05,582 [main] INFO Main - srclib-java version development 2022-08-24 11:11:05,985 [main] INFO ScanCommand - Collecting Maven source units Running: [/home/fossa/.srclib/srclib-ruby/.bin/srclib-ruby scan --repo --subdir .] --> with input {} 2022-08-24 11:11:09,686 [main] INFO ScanCommand - Collecting Gradle source units Running: [/home/fossa/.srclib/srclib-rust/.bin/srclib-rust scan --repo --subdir .] --> with input {} Running: [/home/fossa/.srclib/srclib-themis/.bin/srclib-themis scan-licenses --repo --subdir .] --> with input {} 2022/08/24 11:11:10 maxprocs: Updating GOMAXPROCS=1: determined from CPU quota 2022/08/24 11:11:10 Loading index from /home/fossa/.srclib/srclib-themis/.bin/../themis/... No Godeps file found. No govendor file found. 2022-08-24 11:11:11,480 [main] INFO ScanCommand - Collecting Ant source units Retrying analysis === STEP 0. INITIALIZING & DOWNLOADING SOURCE CODE === ROOT DIRECTORY: /tmp/tmp-38855-isKKHC6dGK8q BUILD ID: 583736 CODE LOCATOR: url-private+1/https://github.com/nodejs/node/archive/refs/tags/v16.14.2.tar.gz$16.14.2 USING THEMIS ====================================================== ANALYSIS DIRECTORY: /tmp/tmp-38855-isKKHC6dGK8q ENRICHMENT DIRECTORY: /tmp/tmp-38855-isKKHC6dGK8q === STEP 1. BEGINNING CODE ANALYSIS === 2022/08/24 11:12:57 Failed to detect git/hg repository root dir for "."; continuing. Failed to detect git/hg repository root dir for "."; continuing. Running: [/home/fossa/.srclib/srclib-php/.bin/srclib-php scan --repo --subdir .] Running: [/home/fossa/.srclib/srclib-iOS/.bin/srclib-iOS scan --repo --subdir .] Running: [/home/fossa/.srclib/srclib-javascript/.bin/srclib-javascript scan-commonjs --repo --subdir .] --> with input {} Running: [/home/fossa/.srclib/srclib-nomos/.bin/srclib-nomos scan-licenses --repo --subdir .] Running: [/home/fossa/.srclib/srclib-java/.bin/srclib-java scan --repo --subdir .] Running: [/home/fossa/.srclib/srclib-perl/.bin/srclib-perl scan --repo --subdir .] Running: [/home/fossa/.srclib/srclib-dotnet/.bin/srclib-dotnet scan --repo --subdir .] Running: [/home/fossa/.srclib/srclib-go/.bin/srclib-go scan --repo --subdir .] --> with input {} --> with input {} --> with input {} --> with input {} --> with input {} --> with input {} --> with input {} ## No bundled JRE found at /home/fossa/.srclib/srclib-java/.bin/../bundled-jdk/jdk1.8.0_45.ARCH-amd64_OS-linux; using system JRE. Running: [/home/fossa/.srclib/srclib-pip/.bin/srclib-pip scan --repo --subdir .] --> with input {} /home/fossa/.gem/gems/rogdl-0.2.3/lib/fixnum.rb:1: warning: constant ::Fixnum is deprecated Running: [/home/fossa/.srclib/srclib-python/.bin/srclib-python scan --repo --subdir .] --> with input {} 2022-08-24 11:13:01,028 [main] INFO Main - srclib-java version development 2022-08-24 11:13:01,423 [main] INFO ScanCommand - Collecting Maven source units Running: [/home/fossa/.srclib/srclib-ruby/.bin/srclib-ruby scan --repo --subdir .] --> with input {} 2022-08-24 11:13:05,123 [main] INFO ScanCommand - Collecting Gradle source units Running: [/home/fossa/.srclib/srclib-rust/.bin/srclib-rust scan --repo --subdir .] --> with input {} Running: [/home/fossa/.srclib/srclib-themis/.bin/srclib-themis scan-licenses --repo --subdir .] --> with input {} No Godeps file found. No govendor file found. 2022/08/24 11:13:06 maxprocs: Updating GOMAXPROCS=1: determined from CPU quota 2022/08/24 11:13:06 Loading index from /home/fossa/.srclib/srclib-themis/.bin/../themis/... 2022-08-24 11:13:06,726 [main] INFO ScanCommand - Collecting Ant source units 2022-08-24 11:13:08,322 [main] INFO ScanCommand - Collecting Sbt source units failed due to error: could not get requirements due to error setup.py does not exist Retrying analysis BUILD ID: 583736 CODE LOCATOR: url-private+1/https://github.com/nodejs/node/archive/refs/tags/v16.14.2.tar.gz$16.14.2 === STEP 0. INITIALIZING & DOWNLOADING SOURCE CODE === USING THEMIS ROOT DIRECTORY: /tmp/tmp-83036-euTuLryyGn8w ====================================================== ANALYSIS DIRECTORY: /tmp/tmp-83036-euTuLryyGn8w ENRICHMENT DIRECTORY: /tmp/tmp-83036-euTuLryyGn8w === STEP 1. BEGINNING CODE ANALYSIS === 2022/08/24 11:15:36 Failed to detect git/hg repository root dir for "."; continuing. Failed to detect git/hg repository root dir for "."; continuing. Running: [/home/fossa/.srclib/srclib-php/.bin/srclib-php scan --repo --subdir .] --> with input {} Running: [/home/fossa/.srclib/srclib-javascript/.bin/srclib-javascript scan-commonjs --repo --subdir .] --> with input {} Running: [/home/fossa/.srclib/srclib-iOS/.bin/srclib-iOS scan --repo --subdir .] --> with input {} Running: [/home/fossa/.srclib/srclib-dotnet/.bin/srclib-dotnet scan --repo --subdir .] --> with input {} Running: [/home/fossa/.srclib/srclib-nomos/.bin/srclib-nomos scan-licenses --repo --subdir .] --> with input {} Running: [/home/fossa/.srclib/srclib-java/.bin/srclib-java scan --repo --subdir .] --> with input {} Running: [/home/fossa/.srclib/srclib-perl/.bin/srclib-perl scan --repo --subdir .] --> with input {} Running: [/home/fossa/.srclib/srclib-go/.bin/srclib-go scan --repo --subdir .] --> with input {} ## No bundled JRE found at /home/fossa/.srclib/srclib-java/.bin/../bundled-jdk/jdk1.8.0_45.ARCH-amd64_OS-linux; using system JRE. Running: [/home/fossa/.srclib/srclib-pip/.bin/srclib-pip scan --repo --subdir .] --> with input {} /home/fossa/.gem/gems/rogdl-0.2.3/lib/fixnum.rb:1: warning: constant ::Fixnum is deprecated Running: [/home/fossa/.srclib/srclib-python/.bin/srclib-python scan --repo --subdir .] --> with input {} 2022-08-24 11:15:41,254 [main] INFO Main - srclib-java version development 2022-08-24 11:15:41,855 [main] INFO ScanCommand - Collecting Maven source units Running: [/home/fossa/.srclib/srclib-ruby/.bin/srclib-ruby scan --repo --subdir .] --> with input {} 2022-08-24 11:15:47,458 [main] INFO ScanCommand - Collecting Gradle source units No Godeps file found. No govendor file found. Running: [/home/fossa/.srclib/srclib-rust/.bin/srclib-rust scan --repo --subdir .] --> with input {} Running: [/home/fossa/.srclib/srclib-themis/.bin/srclib-themis scan-licenses --repo --subdir .] --> with input {} 2022/08/24 11:15:50 maxprocs: Updating GOMAXPROCS=1: determined from CPU quota 2022-08-24 11:15:50,454 [main] INFO ScanCommand - Collecting Ant source units 2022/08/24 11:15:50 Loading index from /home/fossa/.srclib/srclib-themis/.bin/../themis/... 2022-08-24 11:15:52,662 [main] INFO ScanCommand - Collecting Sbt source units failed due to error: could not get requirements due to error setup.py does not exist {"component":{"id":""},"environment":"development","hostname":"fossa-prd-fossa-core-worker-primary-78f6788bd4-qz8rq","level":"info","msg":"Index loaded from cache","service":{"name":""},"serviceName":"default","span":{"id":""},"task":{"id":"","queue":"","type":""},"time":"2022-08-24T11:16:02Z","trace":{"id":""}} 2022/08/24 11:16:02 Index successfully loaded! 2022/08/24 11:16:02 Scanning for licenses in .

[spatten]

Hi Tobias,

Can you send the debug logs for this command by adding a --debug flag to the fossa-cli command

fossa analyze --debug

and then sending the resulting fossa.debug.json.gz file?

Thank you, Scott

[thernstig]

@spatten it is a private project so I cannot do that unfortunately. Did you test this on your side to see if it works?

[ghost]

ما يلي في fossa-deps.yaml:

remote-dependencies:
  - name: Node.js
    version: "16.14.2"
    url: https://github.com/nodejs/node/archive/refs/tags/v16.14.2.tar.gz

يعطي الخطأ التالي في واجهة المستخدم بعد فحصه عند النقر فوق " فشل: عرض" استمرار "الفاشلة.

=== STEP 0. INITIALIZING & DOWNLOADING SOURCE CODE === BUILD ID: 583736 CODE LOCATOR: url-private+1/https://github.com/nodejs/node/archive/refs/tags/v16.14.2.tar.gz$16.14.2 USING THEMIS ROOT DIRECTORY: /tmp/tmp-101592-t5Bc7ShAIu9w ====================================================== ANALYSIS DIRECTORY: /tmp/tmp-101592-t5Bc7ShAIu9w ENRICHMENT DIRECTORY: /tmp/tmp-101592-t5Bc7ShAIu9w === STEP 1. BEGINNING CODE ANALYSIS === 2022/08/24 11:11:02 Failed to detect git/hg repository root dir for "."; continuing. Failed to detect git/hg repository root dir for "."; continuing. Running: [/home/fossa/.srclib/srclib-php/.bin/srclib-php scan --repo --subdir .] --> with input {} Running: [/home/fossa/.srclib/srclib-iOS/.bin/srclib-iOS scan --repo --subdir .] --> with input {} Running: [/home/fossa/.srclib/srclib-javascript/.bin/srclib-javascript scan-commonjs --repo --subdir .] --> with input {} Running: [/home/fossa/.srclib/srclib-nomos/.bin/srclib-nomos scan-licenses --repo --subdir .] --> with input {} Running: [/home/fossa/.srclib/srclib-dotnet/.bin/srclib-dotnet scan --repo --subdir .] --> with input {} Running: [/home/fossa/.srclib/srclib-go/.bin/srclib-go scan --repo --subdir .] --> with input {} Running: [/home/fossa/.srclib/srclib-perl/.bin/srclib-perl scan --repo --subdir .] --> with input {} Running: [/home/fossa/.srclib/srclib-java/.bin/srclib-java scan --repo --subdir .] --> with input {} ## No bundled JRE found at /home/fossa/.srclib/srclib-java/.bin/../bundled-jdk/jdk1.8.0_45.ARCH-amd64_OS-linux; using system JRE. Running: [/home/fossa/.srclib/srclib-pip/.bin/srclib-pip scan --repo --subdir .] --> with input {} /home/fossa/.gem/gems/rogdl-0.2.3/lib/fixnum.rb:1: warning: constant ::Fixnum is deprecated Running: [/home/fossa/.srclib/srclib-python/.bin/srclib-python scan --repo --subdir .] --> with input {} 2022-08-24 11:11:05,582 [main] INFO Main - srclib-java version development 2022-08-24 11:11:05,985 [main] INFO ScanCommand - Collecting Maven source units Running: [/home/fossa/.srclib/srclib-ruby/.bin/srclib-ruby scan --repo --subdir .] --> with input {} 2022-08-24 11:11:09,686 [main] INFO ScanCommand - Collecting Gradle source units Running: [/home/fossa/.srclib/srclib-rust/.bin/srclib-rust scan --repo --subdir .] --> with input {} Running: [/home/fossa/.srclib/srclib-themis/.bin/srclib-themis scan-licenses --repo --subdir .] --> with input {} 2022/08/24 11:11:10 maxprocs: Updating GOMAXPROCS=1: determined from CPU quota 2022/08/24 11:11:10 Loading index from /home/fossa/.srclib/srclib-themis/.bin/../themis/... No Godeps file found. No govendor file found. 2022-08-24 11:11:11,480 [main] INFO ScanCommand - Collecting Ant source units Retrying analysis === STEP 0. INITIALIZING & DOWNLOADING SOURCE CODE === ROOT DIRECTORY: /tmp/tmp-38855-isKKHC6dGK8q BUILD ID: 583736 CODE LOCATOR: url-private+1/https://github.com/nodejs/node/archive/refs/tags/v16.14.2.tar.gz$16.14.2 USING THEMIS ====================================================== ANALYSIS DIRECTORY: /tmp/tmp-38855-isKKHC6dGK8q ENRICHMENT DIRECTORY: /tmp/tmp-38855-isKKHC6dGK8q === STEP 1. BEGINNING CODE ANALYSIS === 2022/08/24 11:12:57 Failed to detect git/hg repository root dir for "."; continuing. Failed to detect git/hg repository root dir for "."; continuing. Running: [/home/fossa/.srclib/srclib-php/.bin/srclib-php scan --repo --subdir .] Running: [/home/fossa/.srclib/srclib-iOS/.bin/srclib-iOS scan --repo --subdir .] Running: [/home/fossa/.srclib/srclib-javascript/.bin/srclib-javascript scan-commonjs --repo --subdir .] --> with input {} Running: [/home/fossa/.srclib/srclib-nomos/.bin/srclib-nomos scan-licenses --repo --subdir .] Running: [/home/fossa/.srclib/srclib-java/.bin/srclib-java scan --repo --subdir .] Running: [/home/fossa/.srclib/srclib-perl/.bin/srclib-perl scan --repo --subdir .] Running: [/home/fossa/.srclib/srclib-dotnet/.bin/srclib-dotnet scan --repo --subdir .] Running: [/home/fossa/.srclib/srclib-go/.bin/srclib-go scan --repo --subdir .] --> with input {} --> with input {} --> with input {} --> with input {} --> with input {} --> with input {} --> with input {} ## No bundled JRE found at /home/fossa/.srclib/srclib-java/.bin/../bundled-jdk/jdk1.8.0_45.ARCH-amd64_OS-linux; using system JRE. Running: [/home/fossa/.srclib/srclib-pip/.bin/srclib-pip scan --repo --subdir .] --> with input {} /home/fossa/.gem/gems/rogdl-0.2.3/lib/fixnum.rb:1: warning: constant ::Fixnum is deprecated Running: [/home/fossa/.srclib/srclib-python/.bin/srclib-python scan --repo --subdir .] --> with input {} 2022-08-24 11:13:01,028 [main] INFO Main - srclib-java version development 2022-08-24 11:13:01,423 [main] INFO ScanCommand - Collecting Maven source units Running: [/home/fossa/.srclib/srclib-ruby/.bin/srclib-ruby scan --repo --subdir .] --> with input {} 2022-08-24 11:13:05,123 [main] INFO ScanCommand - Collecting Gradle source units Running: [/home/fossa/.srclib/srclib-rust/.bin/srclib-rust scan --repo --subdir .] --> with input {} Running: [/home/fossa/.srclib/srclib-themis/.bin/srclib-themis scan-licenses --repo --subdir .] --> with input {} No Godeps file found. No govendor file found. 2022/08/24 11:13:06 maxprocs: Updating GOMAXPROCS=1: determined from CPU quota 2022/08/24 11:13:06 Loading index from /home/fossa/.srclib/srclib-themis/.bin/../themis/... 2022-08-24 11:13:06,726 [main] INFO ScanCommand - Collecting Ant source units 2022-08-24 11:13:08,322 [main] INFO ScanCommand - Collecting Sbt source units failed due to error: could not get requirements due to error setup.py does not exist Retrying analysis BUILD ID: 583736 CODE LOCATOR: url-private+1/https://github.com/nodejs/node/archive/refs/tags/v16.14.2.tar.gz$16.14.2 === STEP 0. INITIALIZING & DOWNLOADING SOURCE CODE === USING THEMIS ROOT DIRECTORY: /tmp/tmp-83036-euTuLryyGn8w ====================================================== ANALYSIS DIRECTORY: /tmp/tmp-83036-euTuLryyGn8w ENRICHMENT DIRECTORY: /tmp/tmp-83036-euTuLryyGn8w === STEP 1. BEGINNING CODE ANALYSIS === 2022/08/24 11:15:36 Failed to detect git/hg repository root dir for "."; continuing. Failed to detect git/hg repository root dir for "."; continuing. Running: [/home/fossa/.srclib/srclib-php/.bin/srclib-php scan --repo --subdir .] --> with input {} Running: [/home/fossa/.srclib/srclib-javascript/.bin/srclib-javascript scan-commonjs --repo --subdir .] --> with input {} Running: [/home/fossa/.srclib/srclib-iOS/.bin/srclib-iOS scan --repo --subdir .] --> with input {} Running: [/home/fossa/.srclib/srclib-dotnet/.bin/srclib-dotnet scan --repo --subdir .] --> with input {} Running: [/home/fossa/.srclib/srclib-nomos/.bin/srclib-nomos scan-licenses --repo --subdir .] --> with input {} Running: [/home/fossa/.srclib/srclib-java/.bin/srclib-java scan --repo --subdir .] --> with input {} Running: [/home/fossa/.srclib/srclib-perl/.bin/srclib-perl scan --repo --subdir .] --> with input {} Running: [/home/fossa/.srclib/srclib-go/.bin/srclib-go scan --repo --subdir .] --> with input {} ## No bundled JRE found at /home/fossa/.srclib/srclib-java/.bin/../bundled-jdk/jdk1.8.0_45.ARCH-amd64_OS-linux; using system JRE. Running: [/home/fossa/.srclib/srclib-pip/.bin/srclib-pip scan --repo --subdir .] --> with input {} /home/fossa/.gem/gems/rogdl-0.2.3/lib/fixnum.rb:1: warning: constant ::Fixnum is deprecated Running: [/home/fossa/.srclib/srclib-python/.bin/srclib-python scan --repo --subdir .] --> with input {} 2022-08-24 11:15:41,254 [main] INFO Main - srclib-java version development 2022-08-24 11:15:41,855 [main] INFO ScanCommand - Collecting Maven source units Running: [/home/fossa/.srclib/srclib-ruby/.bin/srclib-ruby scan --repo --subdir .] --> with input {} 2022-08-24 11:15:47,458 [main] INFO ScanCommand - Collecting Gradle source units No Godeps file found. No govendor file found. Running: [/home/fossa/.srclib/srclib-rust/.bin/srclib-rust scan --repo --subdir .] --> with input {} Running: [/home/fossa/.srclib/srclib-themis/.bin/srclib-themis scan-licenses --repo --subdir .] --> with input {} 2022/08/24 11:15:50 maxprocs: Updating GOMAXPROCS=1: determined from CPU quota 2022-08-24 11:15:50,454 [main] INFO ScanCommand - Collecting Ant source units 2022/08/24 11:15:50 Loading index from /home/fossa/.srclib/srclib-themis/.bin/../themis/... 2022-08-24 11:15:52,662 [main] INFO ScanCommand - Collecting Sbt source units failed due to error: could not get requirements due to error setup.py does not exist {"component":{"id":""},"environment":"development","hostname":"fossa-prd-fossa-core-worker-primary-78f6788bd4-qz8rq","level":"info","msg":"Index loaded from cache","service":{"name":""},"serviceName":"default","span":{"id":""},"task":{"id":"","queue":"","type":""},"time":"2022-08-24T11:16:02Z","trace":{"id":""}} 2022/08/24 11:16:02 Index successfully loaded! 2022/08/24 11:16:02 Scanning for licenses in .

[thernstig]

@spatten I believe this can be closed, as fossa support has already handled this elsewhere. The problem seems to be the new license scanner requiring more resources, but scanning this manual dependency still takes minutes to do.

Is it intended in the new license scanner that manually specified dependencies like above are supposed to take minutes even with 4 cores and 4 GB of ram for the scanning?

[thernstig]

@spatten I will actually close this as it is handled elsewhere, but if you have a reply to the above question I'm still interested in it.