Scanner is taking much longer and fails to fetch packages

[bennati]

We noticed significant slowdowns in scans: the scanner hours to scan projects that previously took minutes. This can be reproduced with MoveTK, latest version.

Description: The scanner starts it's normal operations, once in a while output lines are printed which start with a timestamp. Then, for hours we get a series of similar errors, without any timestamp in between. Right before these errors we get:

06:49:17.829 [DefaultDispatcher-worker-62] WARN  org.ossreviewtoolkit.downloader.vcs.Git - Failed to fetch everything: InvalidRemoteException: Invalid remote: origin
Caused by: NoRemoteRepositoryException: ssh://[email protected]/substack/node-shell-quote.git: ERROR: Repository not found.
java.io.IOException: Could not resolve revision for package 'NPM::shell-quote:1.7.3' with VcsInfo(type=Git, url=ssh://[email protected]/substack/node-shell-quote.git, revision=6a8a899c62a58a30fb128a7079f02826ed4faee0, path=):
	Could not resolve revision candidate '6a8a899c62a58a30fb128a7079f02826ed4faee0': InvalidRemoteException: Invalid remote: origin

and then a long list of IOexceptions like

java.io.IOException: Could not resolve revision for package 'NPM:@tsconfig:node14:1.0.1' with VcsInfo(type=Git, url=https://github.com/tsconfig/bases.git, revision=, path=bases):
	Could not find any revision candidates for package 'NPM:@tsconfig:node14:1.0.1' with VcsInfo(type=Git, url=https://github.com/tsconfig/bases.git, revision=, path=bases).

...

java.io.IOException: Could not resolve revision for package 'NPM:@types:body-parser:1.19.1' with VcsInfo(type=Git, url=https://github.com/DefinitelyTyped/DefinitelyTyped.git, revision=, path=types/body-parser):
	Could not find any revision candidates for package 'NPM:@types:body-parser:1.19.1' with VcsInfo(type=Git, url=https://github.com/DefinitelyTyped/DefinitelyTyped.git, revision=, path=types/body-parser).

Here a full trace

java.io.IOException: Could not resolve revision for package 'NPM:@babel:plugin-syntax-nullish-coalescing-operator:7.8.3' with VcsInfo(type=Git, url=https://github.com/babel/babel.git, revision=master, path=packages/babel-plugin-syntax-nullish-coalescing-operator):
	Discarding revision 'master' because the requested VCS path 'packages/babel-plugin-syntax-nullish-coalescing-operator' does not exist.
	Discarding revision 'v7.8.3' because the requested VCS path 'packages/babel-plugin-syntax-nullish-coalescing-operator' does not exist.
	at org.ossreviewtoolkit.scanner.provenance.DefaultPackageProvenanceResolver$resolveVcs$6.invoke(PackageProvenanceResolver.kt:270)
	at org.ossreviewtoolkit.scanner.provenance.DefaultPackageProvenanceResolver$resolveVcs$6.invoke(PackageProvenanceResolver.kt:207)
	at org.ossreviewtoolkit.scanner.utils.DefaultWorkingTreeCache.use(WorkingTreeCache.kt:65)
	at org.ossreviewtoolkit.scanner.utils.DefaultWorkingTreeCache$use$1.invokeSuspend(WorkingTreeCache.kt)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
	at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:284)
	at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:85)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59)
	at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:38)
	at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
	at org.ossreviewtoolkit.scanner.provenance.DefaultPackageProvenanceResolver.resolveProvenance(PackageProvenanceResolver.kt:85)
	at org.ossreviewtoolkit.scanner.Scanner$resolvePackageProvenances$duration$1$1$1$1.invokeSuspend(Scanner.kt:175)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
	at kotlinx.coroutines.internal.LimitedDispatcher.run(LimitedDispatcher.kt:42)
	at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:95)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:570)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:677)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:664)

I haven't found any curation for plugin-syntax-nullish-coalescing-operator, and I don't think the teams have changed the build system, so I suspect it is some change in how the default source is identified.

[fviernau]

I've investigated the babel one and came up with this bug report: https://github.com/oss-review-toolkit/ort/issues/6028

[sschuberth]

@bennati, is this still reproducible and worth tracing separately from @fviernau's #6028, or can we close this?

[ChenZhaobin]

I got a bunch of similar errors , they all have the same pattern NPM:@types:****, but actually ,the repo paths as ( https://github.com/DefinitelyTyped/DefinitelyTyped/tree/master/types/resolve https://github.com/DefinitelyTyped/DefinitelyTyped/tree/master/types/express ) exist. Fyi, I am scanning the open source repo : https://github.com/GerritCodeReview/gerrit

 java.io.IOException: Could not resolve revision for package 'NPM:@types:resolve:0.0.8' with VcsInfo(type=Git, url=https://github.com/DefinitelyTyped/DefinitelyTyped.git.git, revision=, path=):
	Could not find any revision candidates for package 'NPM:@types:resolve:0.0.8' with VcsInfo(type=Git, url=https://github.com/DefinitelyTyped/DefinitelyTyped.git.git, revision=, path=).
	at org.ossreviewtoolkit.scanner.provenance.DefaultPackageProvenanceResolver$resolveVcs$6.invoke(PackageProvenanceResolver.kt:273)
	at org.ossreviewtoolkit.scanner.provenance.DefaultPackageProvenanceResolver$resolveVcs$6.invoke(PackageProvenanceResolver.kt:210)
	at org.ossreviewtoolkit.scanner.utils.DefaultWorkingTreeCache.use(WorkingTreeCache.kt:65)
	at org.ossreviewtoolkit.scanner.provenance.DefaultPackageProvenanceResolver.resolveVcs(PackageProvenanceResolver.kt:210)
	at org.ossreviewtoolkit.scanner.provenance.DefaultPackageProvenanceResolver.access$resolveVcs(PackageProvenanceResolver.kt:61)
	at org.ossreviewtoolkit.scanner.provenance.DefaultPackageProvenanceResolver$resolveProvenance$1$1$2.invokeSuspend(PackageProvenanceResolver.kt:87)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104)
	at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:277)
	at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:95)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:69)
	at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:48)
	at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
	at org.ossreviewtoolkit.scanner.provenance.DefaultPackageProvenanceResolver.resolveProvenance(PackageProvenanceResolver.kt:87)
	at org.ossreviewtoolkit.scanner.Scanner$resolvePackageProvenances$duration$1$1$1$1.invokeSuspend(Scanner.kt:247)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104)
	at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:111)
	at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:99)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:585)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:802)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:706)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:693)
java.io.IOException: Could not resolve revision for package 'NPM:@types:express:4.17.14' with VcsInfo(type=Git, url=https://github.com/DefinitelyTyped/DefinitelyTyped.git, revision=, path=types/express):
	Could not find any revision candidates for package 'NPM:@types:express:4.17.14' with VcsInfo(type=Git, url=https://github.com/DefinitelyTyped/DefinitelyTyped.git, revision=, path=types/express).
	at org.ossreviewtoolkit.scanner.provenance.DefaultPackageProvenanceResolver$resolveVcs$6.invoke(PackageProvenanceResolver.kt:273)
	at org.ossreviewtoolkit.scanner.provenance.DefaultPackageProvenanceResolver$resolveVcs$6.invoke(PackageProvenanceResolver.kt:210)
	at org.ossreviewtoolkit.scanner.utils.DefaultWorkingTreeCache.use(WorkingTreeCache.kt:65)
	at org.ossreviewtoolkit.scanner.provenance.DefaultPackageProvenanceResolver.resolveVcs(PackageProvenanceResolver.kt:210)
	at org.ossreviewtoolkit.scanner.provenance.DefaultPackageProvenanceResolver.access$resolveVcs(PackageProvenanceResolver.kt:61)
	at org.ossreviewtoolkit.scanner.provenance.DefaultPackageProvenanceResolver$resolveProvenance$1$1$2.invokeSuspend(PackageProvenanceResolver.kt:87)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104)
	at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:277)
	at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:95)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:69)
	at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:48)
	at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
	at org.ossreviewtoolkit.scanner.provenance.DefaultPackageProvenanceResolver.resolveProvenance(PackageProvenanceResolver.kt:87)
	at org.ossreviewtoolkit.scanner.Scanner$resolvePackageProvenances$duration$1$1$1$1.invokeSuspend(Scanner.kt:247)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104)
	at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:111)
	at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:99)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:585)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:802)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:706)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:693)
java.io.IOException: Could not resolve revision for package 'NPM:@types:doctrine:0.0.1' with VcsInfo(type=Git, url=https://github.com/DefinitelyTyped/DefinitelyTyped.git, revision=, path=):
	Could not find any revision candidates for package 'NPM:@types:doctrine:0.0.1' with VcsInfo(type=Git, url=https://github.com/DefinitelyTyped/DefinitelyTyped.git, revision=, path=).
	at org.ossreviewtoolkit.scanner.provenance.DefaultPackageProvenanceResolver$resolveVcs$6.invoke(PackageProvenanceResolver.kt:273)
	at org.ossreviewtoolkit.scanner.provenance.DefaultPackageProvenanceResolver$resolveVcs$6.invoke(PackageProvenanceResolver.kt:210)
	at org.ossreviewtoolkit.scanner.utils.DefaultWorkingTreeCache.use(WorkingTreeCache.kt:65)
	at org.ossreviewtoolkit.scanner.utils.DefaultWorkingTreeCache$use$1.invokeSuspend(WorkingTreeCache.kt)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104)
	at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:277)
	at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:95)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:69)
	at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:48)
	at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
	at org.ossreviewtoolkit.scanner.provenance.DefaultPackageProvenanceResolver.resolveProvenance(PackageProvenanceResolver.kt:87)
	at org.ossreviewtoolkit.scanner.Scanner$resolvePackageProvenances$duration$1$1$1$1.invokeSuspend(Scanner.kt:247)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104)
	at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:111)
	at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:99)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:585)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:802)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:706)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:693)
 
 

[sschuberth]

but actually ,the repo paths as [...] exists

Note that the error messages says it's about the revision, not the path. So ORT simply does not know which revision to checkout for scanning, as no revision could be determined that matches the package version. To solve this, probably package curations are required.

[sschuberth]

Closed as part of backlog grooming. Feel free to comment if you would like to contribute to this.