ort icon indicating copy to clipboard operation
ort copied to clipboard

Published 20 hours ago • verified publisher icon oss-review-toolkit

Gemfile parsing for Bundler (Ruby) doesn't correctly take into account platforms (ruby, java etc.)

Open clemens opened this issue 1 year ago • 9 comments

Describe the bug

Disclaimer: Strictly speaking, this is probably a mixture of a bug and a feature request. But since it does cause unexpected error messages in the output, I'll qualify it as a bug in this ticket.

The gist is that in a Ruby project, ORT doesn't seem to take into account Bundler's concept of "platforms" (ruby, java etc.) and subsequently might look up wrong transitive dependencies and/or complain about missing dependencies.

To Reproduce

Steps to reproduce the behavior:

Note that running bundle lock --add-platform java also doesn't make a difference – while it includes the dependencies correctly in Gemfile.lock, the error persists

# GitLab CI configuration
include:
  - https://raw.githubusercontent.com/oss-review-toolkit/ort-ci-gitlab/main/templates/ort-scan.yml

ort-scan:
  stage: test
  extends: .ort-scan
  variables:
    ORT_LOG_LEVEL: debug

# Minimal Gemfile example

source 'https://rubygems.org'

gem 'psych', '5.1.2'

Expected behavior

  • I'd like to not see errors related to platforms that I don't care about (= haven't declared in my Gemfile.lock). In the example, I'd expect to not see an error that tells me that jar-dependencies couldn't be found.
  • I'd like dependencies to be handled correctly for all platforms that I specify – in other words, if the Gemfile.lock includes only psych (5.1.2), ORT should correctly handle it with its ruby platform dependencies; if it includes psych (5.1.2-java), ORT should correctly handle it with its java platform dependencies; and if it includes multiple platforms, ORT should handle both versions

A common use case for having multiple platforms is when engineers are using Macs for development (=> arm64-darwin or x86_64-darwin platform, depending on whether it's an older or newer Mac) but Linux for production (=> e.g. x86_64-linux). A gem like nokogiri would then be listed with multiple versions:

GEM
  remote: https://rubygems.org/
  specs:
    mini_portile2 (2.8.6)
    nokogiri (1.16.5)
      mini_portile2 (~> 2.8.2)
      racc (~> 1.4)
    nokogiri (1.16.5-arm64-darwin)
      racc (~> 1.4)
    nokogiri (1.16.5-java)
      racc (~> 1.4)
    nokogiri (1.16.5-x86_64-linux)
      racc (~> 1.4)
    racc (1.8.0)
    racc (1.8.0-java)

PLATFORMS
  arm64-darwin-23
  java
  ruby
  x86_64-linux

DEPENDENCIES
  nokogiri

RUBY VERSION
   ruby 3.3.0p0

BUNDLED WITH
   2.5.6

Notice again the differences between the platforms, e.g. whether or not mini_portile2 is a dependency and the fact that there is a specific version of racc for the java platform but not e.g. Macs.

Console / log output

Snippet from GitLab CI output:

09:34:34.502 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.plugins.packagemanagers.bundler.Bundler - Parsing dependency 'psych'.
09:34:34.508 [DefaultDispatcher-worker-1] DEBUG okhttp3.OkHttpClient - Retrieved https://rubygems.org/api/v2/rubygems/psych/versions/5.1.2.yaml from local cache.
09:34:34.508 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.plugins.packagemanagers.bundler.Bundler - Parsing dependency 'jar-dependencies'.
java.util.NoSuchElementException: Key jar-dependencies is missing in the map.
	at kotlin.collections.MapsKt__MapWithDefaultKt.getOrImplicitDefaultNullable(MapWithDefault.kt:24)
	at kotlin.collections.MapsKt__MapsKt.getValue(Maps.kt:369)
	at org.ossreviewtoolkit.plugins.packagemanagers.bundler.Bundler.parseDependency(Bundler.kt:274)
	at org.ossreviewtoolkit.plugins.packagemanagers.bundler.Bundler.parseDependency(Bundler.kt:292)
	at org.ossreviewtoolkit.plugins.packagemanagers.bundler.Bundler.parseDependency(Bundler.kt:292)
	at org.ossreviewtoolkit.plugins.packagemanagers.bundler.Bundler.parseDependency(Bundler.kt:292)
	at org.ossreviewtoolkit.plugins.packagemanagers.bundler.Bundler.parseDependency(Bundler.kt:292)
	at org.ossreviewtoolkit.plugins.packagemanagers.bundler.Bundler.parseDependency(Bundler.kt:292)
	at org.ossreviewtoolkit.plugins.packagemanagers.bundler.Bundler.parseScope(Bundler.kt:257)
	at org.ossreviewtoolkit.plugins.packagemanagers.bundler.Bundler.resolveDependencies(Bundler.kt:215)
	at org.ossreviewtoolkit.analyzer.PackageManager.resolveDependencies(PackageManager.kt:293)
	at org.ossreviewtoolkit.analyzer.PackageManagerRunner$run$3.invokeSuspend(Analyzer.kt:320)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104)
	at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:111)
	at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:99)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:811)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:715)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:702)
09:34:34.508 [DefaultDispatcher-worker-1] ERROR org.ossreviewtoolkit.plugins.packagemanagers.bundler.Bundler - Failed to parse dependency 'jar-dependencies' of project 'Bundler::Gemfile:' in '/builds/clemensk/playground': NoSuchElementException: Key jar-dependencies is missing in the map.

When doing curl 'https://rubygems.org/api/v2/rubygems/psych/versions/5.1.2.yaml' (thus effectively reproducing what ORT does in https://github.com/oss-review-toolkit/ort/blob/740b7d6fc0ee36acded3bc5626f0f67a7f5441ab/plugins/package-managers/bundler/src/main/kotlin/Bundler.kt#L360), I'm getting the java version of it:

---
name: psych
downloads: 59909647
version: 5.1.2
version_created_at: '2023-12-19T02:07:03.237Z'
version_downloads: 205707
platform: java
authors: Aaron Patterson, SHIBATA Hiroshi, Charles Oliver Nutter
info: |
  Psych is a YAML parser and emitter. Psych leverages libyaml[https://pyyaml.org/wiki/LibYAML]
  for its YAML parsing and emitting capabilities. In addition to wrapping libyaml,
  Psych also knows how to serialize and de-serialize most Ruby objects to and from the YAML format.
licenses:
- MIT
metadata:
  msys2_mingw_dependencies: libyaml
yanked: false
sha: 1dd68dc609eddbc884e6892e11da942e16f7256bd30ebde9d35449d43043a6fe
project_uri: https://rubygems.org/gems/psych
gem_uri: https://rubygems.org/gems/psych-5.1.2-java.gem
homepage_uri: https://github.com/ruby/psych
wiki_uri:
documentation_uri: https://www.rubydoc.info/gems/psych/5.1.2
mailing_list_uri:
source_code_uri:
bug_tracker_uri:
changelog_uri:
funding_uri:
dependencies:
  development: []
  runtime:
  - name: jar-dependencies
    requirements: ">= 0.1.7"
built_at: '2023-12-19T00:00:00.000Z'
created_at: '2023-12-19T02:07:03.237Z'
description: |
  Psych is a YAML parser and emitter. Psych leverages libyaml[https://pyyaml.org/wiki/LibYAML]
  for its YAML parsing and emitting capabilities. In addition to wrapping libyaml,
  Psych also knows how to serialize and de-serialize most Ruby objects to and from the YAML format.
downloads_count: 205707
number: 5.1.2
summary: Psych is a YAML parser and emitter
rubygems_version: ">= 0"
ruby_version: ">= 2.5.0"
prerelease: false
requirements:
- jar org.snakeyaml:snakeyaml-engine, 2.7
spec_sha: 3b9fd76cca5746f22d4acd77c731f07302df94ea2dbee9a401c9bac79f0af056

As specified in http://guides.rubygems.org/rubygems-org-api-v2/, a platform parameter can be given (curl 'https://rubygems.org/api/v2/rubygems/psych/versions/5.1.2.yaml?platform=ruby'), which then returns the correct version:

---
name: psych
downloads: 59909666
version: 5.1.2
version_created_at: '2023-12-19T02:06:51.830Z'
version_downloads: 6035609
platform: ruby
authors: Aaron Patterson, SHIBATA Hiroshi, Charles Oliver Nutter
info: |
  Psych is a YAML parser and emitter. Psych leverages libyaml[https://pyyaml.org/wiki/LibYAML]
  for its YAML parsing and emitting capabilities. In addition to wrapping libyaml,
  Psych also knows how to serialize and de-serialize most Ruby objects to and from the YAML format.
licenses:
- MIT
metadata:
  msys2_mingw_dependencies: libyaml
yanked: false
sha: 337322f58fc2bf24827d2b9bd5ab595f6a72971867d151bb39980060ea40a368
project_uri: https://rubygems.org/gems/psych
gem_uri: https://rubygems.org/gems/psych-5.1.2.gem
homepage_uri: https://github.com/ruby/psych
wiki_uri:
documentation_uri: https://www.rubydoc.info/gems/psych/5.1.2
mailing_list_uri:
source_code_uri:
bug_tracker_uri:
changelog_uri:
funding_uri:
dependencies:
  development: []
  runtime:
  - name: stringio
    requirements: ">= 0"
built_at: '2023-12-19T00:00:00.000Z'
created_at: '2023-12-19T02:06:51.830Z'
description: |
  Psych is a YAML parser and emitter. Psych leverages libyaml[https://pyyaml.org/wiki/LibYAML]
  for its YAML parsing and emitting capabilities. In addition to wrapping libyaml,
  Psych also knows how to serialize and de-serialize most Ruby objects to and from the YAML format.
downloads_count: 6035609
number: 5.1.2
summary: Psych is a YAML parser and emitter
rubygems_version: ">= 0"
ruby_version: ">= 2.5.0"
prerelease: false
requirements: []
spec_sha: 4734febf6e6013fba539e12d7f2049d85549af0a0747e27cc3f28cf3eb196b02

Take note of the different dependencies – jar-dependencies for the java version, stringio for the ruby version.

Environment

Output of the ort requirements command:

10:45:25,236 |-INFO in ch.qos.logback.classic.LoggerContext[default] - This is logback-classic version 1.5.6
10:45:25,238 |-INFO in ch.qos.logback.classic.util.ContextInitializer@29d80d2b - No custom configurators were discovered as a service.
10:45:25,238 |-INFO in ch.qos.logback.classic.util.ContextInitializer@29d80d2b - Trying to configure with ch.qos.logback.classic.joran.SerializedModelConfigurator
10:45:25,238 |-INFO in ch.qos.logback.classic.util.ContextInitializer@29d80d2b - Constructed configurator of type class ch.qos.logback.classic.joran.SerializedModelConfigurator
10:45:25,242 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.scmo]
10:45:25,242 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.scmo]
10:45:25,246 |-INFO in ch.qos.logback.classic.util.ContextInitializer@29d80d2b - ch.qos.logback.classic.joran.SerializedModelConfigurator.configure() call lasted 4 milliseconds. ExecutionStatus=INVOKE_NEXT_IF_ANY
10:45:25,246 |-INFO in ch.qos.logback.classic.util.ContextInitializer@29d80d2b - Trying to configure with ch.qos.logback.classic.util.DefaultJoranConfigurator
10:45:25,247 |-INFO in ch.qos.logback.classic.util.ContextInitializer@29d80d2b - Constructed configurator of type class ch.qos.logback.classic.util.DefaultJoranConfigurator
10:45:25,247 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
10:45:25,248 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [jar:file:/opt/ort/lib/cli-22.5.0.jar!/logback.xml]
10:45:25,249 |-WARN in ch.qos.logback.classic.util.DefaultJoranConfigurator@58e1d9d - Resource [logback.xml] occurs multiple times on the classpath.
10:45:25,249 |-WARN in ch.qos.logback.classic.util.DefaultJoranConfigurator@58e1d9d - Resource [logback.xml] occurs at [jar:file:/opt/ort/lib/helper-cli-22.5.0.jar!/logback.xml]
10:45:25,249 |-WARN in ch.qos.logback.classic.util.DefaultJoranConfigurator@58e1d9d - Resource [logback.xml] occurs at [jar:file:/opt/ort/lib/cli-22.5.0.jar!/logback.xml]
10:45:25,253 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@446a1e84 - URL [jar:file:/opt/ort/lib/cli-22.5.0.jar!/logback.xml] is not of type file
10:45:25,309 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - Processing appender named [STDOUT]
10:45:25,309 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
10:45:25,313 |-INFO in ch.qos.logback.core.model.processor.ImplicitModelHandler - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
10:45:25,328 |-INFO in ch.qos.logback.classic.model.processor.RootLoggerModelHandler - Setting level of ROOT logger to WARN
10:45:25,328 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [STDOUT] to Logger[ROOT]
10:45:25,329 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.http.headers] to ERROR
10:45:25,329 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.http.wire] to ERROR
10:45:25,329 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.eclipse.jgit.internal.storage.file.FileSnapshot] to ERROR
10:45:25,329 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.analyzer.managers.Yarn2] to INFO
10:45:25,329 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.clients.fossid.FossIdRestService] to INFO
10:45:25,329 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.plugins.reporters.fossid.FossIdReporter] to INFO
10:45:25,330 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.plugins.reporters.fossid.FossIdSnippetReporter] to INFO
10:45:25,330 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.plugins.scanners.fossid.FossId] to INFO
10:45:25,330 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.plugins.scanners.fossid.FossIdConfig] to INFO
10:45:25,330 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.ossreviewtoolkit.plugins.scanners.fossid.FossIdUrlProvider] to INFO
10:45:25,330 |-INFO in ch.qos.logback.core.model.processor.DefaultProcessor@4f0f2942 - End of configuration.
10:45:25,330 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@2657d4dd - Registering current configuration as safe fallback point
10:45:25,330 |-INFO in ch.qos.logback.classic.util.ContextInitializer@29d80d2b - ch.qos.logback.classic.util.DefaultJoranConfigurator.configure() call lasted 83 milliseconds. ExecutionStatus=DO_NOT_INVOKE_NEXT_IF_ANY
 ______________________________                                                
/        \_______   \__    ___/ The OSS Review Toolkit, version 22.5.0,        
|    |   | |       _/ |    |    built with JDK 11.0.23+9, running under Java 17
|    |   | |    |   \ |    |    Executing 'requirements' as 'ort' on Linux     
\________/ |____|___/ |____|    with 32 CPUs and a maximum of 65536 MiB of memo
                                                                               
Environment variables:                                                        
ORT_CONFIG_DIR = /home/ort/.ort/config                                        
ORT_DATA_DIR = /home/ort/.ort                                                 
HOME = /home/ort                                                              
JAVA_HOME = /opt/java/openjdk                                                 
ANDROID_HOME = /opt/android-sdk                                               
                                                                              
Looking for ORT configuration in the following file:
        /home/ort/.ort/config/config.yml (does not exist)
AdviceProviderFactory plugins:
        * GitHubDefects
        * NexusIQ
        * OssIndex
        * OSV
        * VulnerableCode
OrtCommand plugins:
        * advise
        * analyze
        * compare
        * config
        * download
        * evaluate
        * migrate
        * notify
        * report
        * requirements
        * scan
        * upload-curations
        * upload-result-to-postgres
        * upload-result-to-sw360
PackageConfigurationProviderFactory plugins:
        * DefaultDir
        * Dir
        * OrtConfig
PackageCurationProviderFactory plugins:
        * ClearlyDefined
        * DefaultDir
        * DefaultFile
        * File
        * OrtConfig
        * SW360
PackageManagerFactory plugins:
        * Bazel
        * Bower
        * Bundler
        * Cargo
        * Carthage
        * CocoaPods
        * Composer
        * Conan
        * GoMod
        * Gradle
        * GradleInspector
        * Maven
        * NPM
        * NuGet
        * PIP
        * Pipenv
        * PNPM
        * Poetry
        * Pub
        * SBT
        * SpdxDocumentFile
        * Stack
        * SwiftPM
        * Unmanaged
        * Yarn
        * Yarn2
Reporter plugins:
        * CtrlXAutomation
        * CycloneDx
        * DocBookTemplate
        * EvaluatedModel
        * FossId
        * FossIdSnippet
        * GitLabLicenseModel
        * HtmlTemplate
        * ManPageTemplate
        * Opossum
        * PdfTemplate
        * PlainTextTemplate
        * SpdxDocument
        * StaticHtml
        * TrustSource
        * WebApp
ScannerWrapperFactory plugins:
        * Askalono
        * BoyterLc
        * FossId
        * Licensee
        * ScanCode
        * SCANOSS
VersionControlSystem plugins:
        * Git
        * GitRepo
        * Mercurial
        * Subversion
Scanners:
        - Askalono: Requires 'askalono' in no specific version. Tool not found.
        - BoyterLc: Requires 'lc' in no specific version. Tool not found.
        - Licensee: Requires 'licensee' in no specific version. Tool not found.
        * ScanCode: Requires 'scancode' in version >=3.0.0. Found version 32.1.0.
PackageManagers:
        * Bazel: Requires 'bazel' in version >=7.0.0. Found version 7.0.1.
        * Bower: Requires 'bower' in version >=1.8.8. Found version 1.8.14.
        * Cargo: Requires 'cargo' in no specific version. Found version 1.72.0.
        + CocoaPods: Requires 'pod' in version >=1.11.0. Could not determine the version.
        * Composer: Requires 'composer' in version >=1.5.0. Found version 2.2.23.
        * Conan: Requires 'conan' in version >=1.18.0. Found version 1.63.0.
        * GoMod: Requires 'go' in version >=1.21.1. Found version 1.22.2.
        * Npm: Requires 'npm' in version >=6.0.0 and <11.0.0. Found version 10.5.0.
        + NuGetInspector: Requires 'nuget-inspector' in no specific version. Could not determine the version.
        * Pipenv: Requires 'pipenv' in version >=2018.10.9. Found version 2023.12.1.
        * Pnpm: Requires 'pnpm' in version >=5.0.0 and <9.0.0. Found version 8.10.3.
        * Poetry: Requires 'poetry' in no specific version. Found version 1.7.0.
        * Pub: Requires 'dart' in version >=2.10.0. Found version 2.18.4.
        * PythonInspector: Requires 'python-inspector' in version >=0.9.2. Found version 0.10.0.
        * Sbt: Requires 'sbt' in version >=0.13.0. Found version 1.10.0.
        * Stack: Requires 'stack' in version >=2.1.1. Found version 2.15.7.
        * SwiftPm: Requires 'swift' in no specific version. Found version 5.9.2.
        * Yarn: Requires 'yarn' in version >=1.3.0 and <1.23.0. Found version 1.22.19.
VersionControlSystems:
        * GitCommand: Requires 'git' in version >=2.29.0. Found version 2.34.1.
        * GitRepo: Requires 'repo' in no specific version. Found version 2.45 (launcher).
        * MercurialCommand: Requires 'hg' in no specific version. Found version 6.7.3.
Prefix legend:
        - The tool was not found in the PATH environment.
        + The tool was found in the PATH environment, but not in the required version.
        * The tool was found in the PATH environment in the required version.
ScanCode license texts not found.
Not all tools requirements were satisfied:
        ! For some tools the version could not be determined.

And specify (relevant parts of) your ORT configuration (config.yml):

Nothing special, just using the default config that https://github.com/oss-review-toolkit/ort-ci-gitlab/blob/main/templates/ort-scan.yml provides/generates.

Additional context

n/a

clemens avatar May 24 '24 10:05 clemens

Thanks for the sophisticated report. In essence, this is a specific case of https://github.com/oss-review-toolkit/ort/issues/4013.

sschuberth avatar May 24 '24 12:05 sschuberth

I guess part of the problem is this line:

https://github.com/oss-review-toolkit/ort/blob/69f8c6e2c7b03d23cfe5373f193b132c71ec42b4/plugins/package-managers/bundler/src/main/resources/resolve_dependencies.rb#L31-L32

sschuberth avatar May 24 '24 12:05 sschuberth

I haven't traced the whole call chain and I'm by no means a Kotlin expert, but based on the stack trace, the problem is in this function: https://github.com/oss-review-toolkit/ort/blob/740b7d6fc0ee36acded3bc5626f0f67a7f5441ab/plugins/package-managers/bundler/src/main/kotlin/Bundler.kt#L360

It seems that independent of that Ruby script that uses Bundler's own facilities to resolve dependencies, the RubyGems API is still queried directly, using ORT's HTTP client (at least that's how I'm interpreting okHttpClient.downloadText(url)).

This is also evidenced by the fact that ORT seems to be trying to resolve a dependency for the java version of the psych gem, even though Bundler is set to force the generic (= non-java) version for resolving.

If ORT were to solely rely on Bundler's mechanisms instead of querying the API manually, I think it might generally be fine, but you might want to either remove the Bundler.settings.set_global(:force_ruby_platform, true) (which could cause wrong results for basically any package with native extensions) or at least offer some kind of configuration option to allow to opt out of this. Because Bundler actually behaves correctly by default, IMO – see https://github.com/ruby/ruby/blob/e49d68bf2732cf467a1532283de4a1716e238677/lib/bundler/definition.rb#L634.

clemens avatar May 24 '24 13:05 clemens

Not sure if it helps, but I toyed around with it a bit more and the minimal Gemfile with which I can trigger the problem seems to be this:

source 'https://rubygems.org'

gem 'rails'
gem 'rails-i18n'

rails depends on railties, which depends on irb, which depends on rdoc, which depends on psych, which has a dependency on jar-dependencies on the Java platform, but not others.

And rails-i18n has a dependency on rails and then consequently the same transitive dependencies.

When I remove one or the other, the problem doesn't appear. But when I have both in my Gemfile, the problem is there.

The same is true when I replace either of them with vite_rails (= another package that depends on rails).

But the problem doesn't appear when I put in 2 packages that both depend on rdoc, but otherwise don't share transitive dependencies; e.g. this is fine:

source 'https://rubygems.org'

gem 'irb'
gem 'minitest'

clemens avatar May 27 '24 18:05 clemens

Further digging: According to the source code of rubygems.org, it is expected behavior that when not explicitly specifying platform, it picks whatever was last created.

For psych, this seems to be the Java version:

curl 'https://rubygems.org/api/v2/rubygems/psych/versions/5.1.2.yaml?platform=ruby'
...
created_at: '2023-12-19T02:06:51.830Z'
...

curl 'https://rubygems.org/api/v2/rubygems/psych/versions/5.1.2.yaml'
...
created_at: '2023-12-19T02:07:03.237Z'
...

Whereas e.g. for nokogiri (another multi-platform gem), it's Ruby:

curl 'https://rubygems.org/api/v2/rubygems/nokogiri/versions/1.16.5.yaml'
...
created_at: '2024-05-13T14:01:29.505Z'
...

curl 'https://rubygems.org/api/v2/rubygems/nokogiri/versions/1.16.5.yaml?platform=java'
...
created_at: '2024-05-13T14:01:04.168Z'
...

This still doesn't explain why sometimes it seems to be fine and sometimes it isn't (see my previous comment), but at least indicates that if ORT wants to have deterministic behavior, the platform needs to always be explicitly specified. (And which platform it is can be derived from Gemfile.lock only.)

clemens avatar May 28 '24 11:05 clemens

Any news on this one, @sschuberth ? Can I help with further info?

Also, if the intention was to enforce ruby as a platform and it's just not working for that one part of the codebase, would it make sense to adapt this accordingly until a more elaborate fix (which takes into account all declared platforms) is in place?

clemens avatar Jun 10 '24 09:06 clemens

Any news on this one, @sschuberth ? Can I help with further info?

I believe your info is more than sufficient, thank you @clemens. It's just that to be best of my knowledge no one from the community is planning to work on this (for free) in the near future. If you're interested in a commercial ORT support subscription that could accelerate this (also see https://github.com/oss-review-toolkit/ort/issues/8381), feel free to contact any of the mentioned parties directly.

sschuberth avatar Jun 10 '24 10:06 sschuberth

That's understandable. :)

Would you accept an MR that appends ?platform=ruby to the problematic URL so that the problem is at least consistent and in line what you apparently intended with https://github.com/oss-review-toolkit/ort/blob/69f8c6e2c7b03d23cfe5373f193b132c71ec42b4/plugins/package-managers/bundler/src/main/resources/resolve_dependencies.rb#L31-L32? (And arguably, that could also solve the problem for a lot of people in a lot of scenarios, since the majority of Ruby installations are with regular CRuby and the majority of packages there doesn't have native extensions.)

clemens avatar Jun 11 '24 13:06 clemens

Would you accept an MR that appends ?platform=ruby to the problematic URL

Sounds reasonable. But for me it's the easiest to tell from the diff of a MR / PR, as that would need to include any necessary changes to the expected test results.

sschuberth avatar Jun 11 '24 14:06 sschuberth