Bundler fails to parse dependency "NoSuchElementException: Key spoon is missing in the map."

[bennati]

One of our projects gets the following scan issue: Failed to parse dependency 'spoon' of project 'Bundler::Gemfile:' in '/builds/oss/oss-review-toolkit/ort-gitlab-ci/project': NoSuchElementException: Key spoon is missing in the map.

I was able to reproduce the issue with this minimal Gemfile:

# frozen_string_literal: true

source 'https://rubygems.org'

ruby '3.0.2'

source '<ARTIFACTORY>' do
  gem 'bootstrap-sass-2', '2.3.2.2.pre.5'
end

group :test, :development, :ui_test do
  gem 'pry-rails'
end

Interestingly, the error disappears when changing the Gemfile to the following:

# frozen_string_literal: true

source 'https://rubygems.org'

ruby '3.0.2'

gem 'bootstrap-sass', '2.3.2.2'

group :test, :development, :ui_test do
  gem 'pry-rails'
end

Here is the diff between bootstrap-sass-2:2.3.2.2.pre.5 and bootstrap-sass:2.3.2.2

[sschuberth]

Can you please share a stacktrace, or a real value for the <ARTIFACTORY> placeholder?

[bennati]

There is no stacktrace: the error is not fatal and the analyzer continues

 ______________________________                                                
/        \_______   \__    ___/ The OSS Review Toolkit, version fdd83a3.       
|    |   | |       _/ |    |                                                   
|    |   | |    |   \ |    |    Running 'analyze' as 'ort' under Java 17.0.8 on
\________/ |____|___/ |____|    with 4 CPUs and a maximum of 27648 MiB of memor
                                                                               
Environment variables:                                                        
ORT_CONFIG_DIR = ort-configuration                                            
ORT_DATA_DIR = /builds/oss/oss-review-toolkit/ort-gitlab-ci/.ort              
JAVA_HOME = /opt/java/openjdk                                                 
ANDROID_HOME = /opt/android-sdk                                               
                                                                              
Looking for ORT configuration in the following file:
        /builds/oss/oss-review-toolkit/ort-gitlab-ci/config.yml

Looking for analyzer-specific configuration in the following files and directories:
	/builds/oss/oss-review-toolkit/ort-gitlab-ci/project/.ort.yml (does not exist)
	/builds/oss/oss-review-toolkit/ort-gitlab-ci/ort-configuration/resolutions.yml
The following 25 package manager(s) are enabled:
	Bower, Bundler, Cargo, Carthage, CocoaPods, Composer, Conan, GoDep, GoMod, Gradle, Maven, NPM, NuGet, PIP, Pipenv, PNPM, Poetry, Pub, SBT, SpdxDocumentFile, SPM, Stack, Unmanaged, Yarn, Yarn2
The following 2 package curation provider(s) are enabled:
	RepositoryConfiguration, File
Analyzing project path:
	/builds/oss/oss-review-toolkit/ort-gitlab-ci/project
Found 1 Bundler definition file(s) at:
	Gemfile
Found 1 NPM definition file(s) at:
	package.json
Found in total 2 definition file(s) from the following 2 package manager(s):
	Bundler, NPM
WARNING:  You don't have /home/ort/.local/share/gem/jruby/3.1.0/bin in your PATH,
	  gem executables will not run.
Calling `DidYouMean::SPELL_CHECKERS.merge!(error_name => spell_checker)' has been deprecated. Please call `DidYouMean.correct_error(error_name, spell_checker)' instead.
09:54:40.509 [DefaultDispatcher-worker-1] ERROR org.ossreviewtoolkit.plugins.packagemanagers.bundler.Bundler - Failed to parse dependency 'spoon' of project 'Bundler::Gemfile:' in '/builds/oss/oss-review-toolkit/ort-gitlab-ci/project': NoSuchElementException: Key spoon is missing in the map.
09:54:45.304 [DefaultDispatcher-worker-1] ERROR org.ossreviewtoolkit.plugins.packagemanagers.bundler.Bundler - Failed to parse dependency 'spoon' of project 'Bundler::Gemfile:' in '/builds/oss/oss-review-toolkit/ort-gitlab-ci/project': NoSuchElementException: Key spoon is missing in the map.
09:54:45.305 [DefaultDispatcher-worker-1] ERROR org.ossreviewtoolkit.plugins.packagemanagers.bundler.Bundler - Failed to parse dependency 'spoon' of project 'Bundler::Gemfile:' in '/builds/oss/oss-review-toolkit/ort-gitlab-ci/project': NoSuchElementException: Key spoon is missing in the map.
09:54:52.030 [DefaultDispatcher-worker-1] ERROR org.ossreviewtoolkit.plugins.packagemanagers.bundler.Bundler - Failed to parse dependency 'spoon' of project 'Bundler::Gemfile:' in '/builds/oss/oss-review-toolkit/ort-gitlab-ci/project': NoSuchElementException: Key spoon is missing in the map.
09:54:52.036 [DefaultDispatcher-worker-1] ERROR org.ossreviewtoolkit.plugins.packagemanagers.bundler.Bundler - Failed to parse dependency 'spoon' of project 'Bundler::Gemfile:' in '/builds/oss/oss-review-toolkit/ort-gitlab-ci/project': NoSuchElementException: Key spoon is missing in the map.
Writing analyzer result to '/builds/oss/oss-review-toolkit/ort-gitlab-ci/ort-results/analyzer-result.json'.
The analysis took 5m 18.309024162s.

I shared the value of <ARTIFACTORY> with you on Slack

[sschuberth]

There is no stacktrace: the error is not fatal and the analyzer continues

Have you tried --stacktrace which also lists suppressed exceptions?

[bennati]

Here is the stacktrace:

13:27:19.337 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.plugins.packagemanagers.bundler.Bundler - Parsing dependency 'pry-rails'.
13:27:19.444 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.ort.OkHttpClientHelper - Downloaded from https://rubygems.org/api/v2/rubygems/pry-rails/versions/0.3.9.yaml via network.
13:27:19.444 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.plugins.packagemanagers.bundler.Bundler - Parsing dependency 'pry'.
13:27:19.445 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.ort.OkHttpClientHelper - Retrieved https://rubygems.org/api/v2/rubygems/pry/versions/0.13.1.yaml from local cache.
13:27:19.445 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.plugins.packagemanagers.bundler.Bundler - Parsing dependency 'coderay'.
13:27:19.445 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.ort.OkHttpClientHelper - Retrieved https://rubygems.org/api/v2/rubygems/coderay/versions/1.1.3.yaml from local cache.
13:27:19.445 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.plugins.packagemanagers.bundler.Bundler - Parsing dependency 'method_source'.
13:27:19.445 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.ort.OkHttpClientHelper - Retrieved https://rubygems.org/api/v2/rubygems/method_source/versions/1.0.0.yaml from local cache.
13:27:19.445 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.plugins.packagemanagers.bundler.Bundler - Parsing dependency 'spoon'.
java.util.NoSuchElementException: Key spoon is missing in the map.
	at kotlin.collections.MapsKt__MapWithDefaultKt.getOrImplicitDefaultNullable(MapWithDefault.kt:24)
	at kotlin.collections.MapsKt__MapsKt.getValue(Maps.kt:360)
	at org.ossreviewtoolkit.plugins.packagemanagers.bundler.Bundler.parseDependency(Bundler.kt:274)
	at org.ossreviewtoolkit.plugins.packagemanagers.bundler.Bundler.parseDependency(Bundler.kt:292)
	at org.ossreviewtoolkit.plugins.packagemanagers.bundler.Bundler.parseDependency(Bundler.kt:292)
	at org.ossreviewtoolkit.plugins.packagemanagers.bundler.Bundler.parseScope(Bundler.kt:257)
	at org.ossreviewtoolkit.plugins.packagemanagers.bundler.Bundler.resolveDependencies(Bundler.kt:215)
	at org.ossreviewtoolkit.analyzer.PackageManager.resolveDependencies(PackageManager.kt:319)
	at org.ossreviewtoolkit.analyzer.PackageManagerRunner$run$3.invokeSuspend(Analyzer.kt:323)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:108)
	at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:115)
	at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:103)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684)
13:27:19.448 [DefaultDispatcher-worker-1] ERROR org.ossreviewtoolkit.plugins.packagemanagers.bundler.Bundler - Failed to parse dependency 'spoon' of project 'Bundler::Gemfile:' in '/builds/oss/oss-review-toolkit/ort-gitlab-ci/project': NoSuchElementException: Key spoon is missing in the map.
13:27:19.448 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.plugins.packagemanagers.bundler.Bundler - Parsing dependency 'rspec-rails'.

[sschuberth]

Could be a very similar problem to #7523 or #7543.

[bennati]

I tested with 311ab74 but the problem persists

[sschuberth]

I tested with 311ab74 but the problem persists

That's expected. It's just that a (probably) similar issue was fixed for another package manager (CocoaPods).

[aames-cls]

I'm having a similar issue as well

WARNING:  You don't have /home/ort/.local/share/gem/jruby/3.1.0/bin in your PATH,
	  gem executables will not run.
java.lang.NullPointerException: Cannot invoke "com.fasterxml.jackson.databind.JsonNode.textValue()" because the return value of "com.fasterxml.jackson.databind.JsonNode.get(String)" is null
	at org.ossreviewtoolkit.plugins.packagemanagers.bundler.GemSpec$Companion.createFromMetadata(Bundler.kt:423)

[sschuberth]

@aames-cls, can you please share the project you're using for testing?

[lancedikson]

I see a similar issue as @aames-cls mentioned only when my Gemfile includes comments # some comment. Removing the comments helps to bypass the problem.

[sschuberth]

@lancedikson are you able to reproduce the problem by adding comments to any of the test projects in this directory? Because I tried, and I can't.

[sschuberth]

@lancedikson, your issue (with is different from the original issue reported in this thread) should be fixed by https://github.com/oss-review-toolkit/ort/pull/8644. Please give it a try.

[sschuberth]

@aames-cls I believe your issue (which also is different than the OP's) got also resolved with #8644.