ort
ort copied to clipboard
oss-review-toolkit
analyzer: Add support for Swift Package Manager
Swift Package Manager is still very young and no other open-source scanners support it. I thought it would benefit the community if ORT had SPM support. This commit adds a new analyzer that parses Package.resolved files and creates results that are enriched with as much data as possible.
Codecov Report
Merging #5092 (fbdb6d8) into main (ddfbf62) will decrease coverage by
0.08%. The diff coverage is56.32%.
:exclamation: Current head fbdb6d8 differs from pull request most recent head 1f7e11d. Consider uploading reports for the commit 1f7e11d to get more accurate results
@@ Coverage Diff @@
## main #5092 +/- ##
============================================
- Coverage 72.43% 72.34% -0.09%
- Complexity 1895 1900 +5
============================================
Files 251 252 +1
Lines 13450 13537 +87
Branches 1899 1920 +21
============================================
+ Hits 9742 9794 +52
- Misses 2721 2736 +15
- Partials 987 1007 +20
| Impacted Files | Coverage Δ | |
|---|---|---|
| analyzer/src/main/kotlin/managers/SPM.kt | 56.32% <56.32%> (ø) |
|
| downloader/src/main/kotlin/vcs/Subversion.kt | 68.53% <0.00%> (+0.69%) |
:arrow_up: |
| .../curation/ClearlyDefinedPackageCurationProvider.kt | 64.06% <0.00%> (+3.12%) |
:arrow_up: |
Continue to review full report at Codecov.
Legend - Click here to learn more
Δ = absolute <relative> (impact),ø = not affected,? = missing dataPowered by Codecov. Last update ddfbf62...1f7e11d. Read the comment docs.
![codecov[bot] avatar](https://avatars.githubusercontent.com/in/254?v=4 "Published by a pub.dev verified publisher"){kind=small}
Thanks a lot for your contribution @UgniusV! Unfortunately, I won't have time to look at this within the next days. So please be a bit patient, or maybe hope for a review by another maintainer 😉
@UgniusV Thank you for your contribution, just about to go on holiday so only had about over 1 hour to look into this PR (I am to my knowledge the only ORT maintainer running on MacOS).
My first struggle was very simple finding a test repository that actually has Package.resolved file, browsed the code of several Swift packages on https://swiftpackageregistry.com/ but most of the popular one don't seem to use a Package.resolved file. Seeing that so many projects don't use "Package.resolved", ".package.resolved" have you looked into other ways to determine the dependencies for a Swift package?
Could you maybe add execution of swift package show-dependencies --format json if Package.swift but not a Package.resolved or .package.resolved so ORT will support analyzing a more Swift packages? In ORT we normally prefer to execute the build tool itself over parsing lockfiles as not all info is lock file, dependencies in lockfile may differ from what actually used.
Note to self: https://github.com/vapor/vapor seems to be a good candidate for a test project to test ORT's support for Swift.
Hey @tsteenbe ! First of all, thank you for your time and for reviewing my PR. The idea of this PR is to parse dependencies from iOS/macOS app projects that are built with XCode (using Swift Package Manager).
In these cases, there are no Package.swift files and the package show-dependencies command fails:
❯ swift package show-dependencies --format json
error: root manifest not found
This is because Package.swift files are only present for libraries that are distributed using the SPM. Not for full iOS/macOS projects. These types of projects contain only Package.resolved which are automatically created & updated by XCode whenever a new dependency is added to the app project.
There are already tools that can generate SBOMs from Package.swift. For example:
https://github.com/opensbom-generator/spdx-sbom-generator.
But there were no tools that could generate SBOM from Package.resolved files, that's the reason behind this pull request.
I do agree with you that we should make ORT cover as many cases as possible. So I will add a fallback mechanism that parses the output of swift package show-dependencies --format json in case no Package.resolved files are found and we are dealing with a Swift Library, rather than an XCode app build.
@tsteenbe I just noticed that when running swift package show-dependencies --format json inside a directory with a Package.swift file present. Package.resolved file automatically gets created and it contains the whole dependency graph (transitive dependencies are included). Maybe it would be a good idea to override beforeResolution method inside SPM class and make it run swift package show-dependencies --format json so that the Package.resolved file would get created and then we could reuse the same parsing logic and avoid additional complexity? What do you think?
What do you think?
Please have a look how this is done in other package managers that support the concept of "lockfiles", like in NPM. By default ORT refuses to analyze projects without a lockfile as results would not be deterministic. Only if the allowDynamicVersions configuration option is set to true projects without a lockfile should be analyzed.
The idea to implement analysis of projects without a lockfile by creating a temporary lockfile and then reusing the code to parse the lockfile is a good one. We were actually thinking about implementing it like this for several of the existing package managers, too, instead of coming up with a completely separate code path for the analysis of such projects.
Awesome, so I will override beforeResolution method to perform swift package show-dependencies --format json if allowDynamicVersions configuration option is set to true. Fix the README and rebase my changes.
@tsteenbe @sschuberth I have added the conditional conversion of Package.swift to Package.resolved file inside mapDefinitionFiles override. This seemed like a better place to implement this rather than beforeResolution since we also need to provide a custom definition file mapping (Explained in javadoc). The parsing logic was left untouched.
I just noticed that when running
swift package show-dependencies --format jsoninside a directory with aPackage.swiftfile present.Package.resolvedfile automatically gets created and it contains the whole dependency graph (transitive dependencies are included). Maybe it would be a good idea to overridebeforeResolutionmethod inside SPM class and make it runswift package show-dependencies --format jsonso that thePackage.resolvedfile would get created and then we could reuse the same parsing logic and avoid additional complexity? What do you think?
The output of swift package show-dependencies --format json includes the transitive dependencies and associate them with their parents.
Is this association also visible in the lockfile? Or is it a flat list?
Edit: I just checked: The lockfile does not contain the association between the packages. The output of swift package does. So maybe a combination of parsing the output of swift package and parsing the lockfile is necessary if allowDynamicVersions is enabled.
Re @tsteenbe
I am to my knowledge the only ORT maintainer running on MacOS
MacOS is not required, as SPM is also available for Linux: https://github.com/oss-review-toolkit/ort/issues/723#issuecomment-698872983. So it could also be bootstrapped or included in the Docker image.
So it could also be bootstrapped or included in the Docker image.
Then I'd prefer to have it added to the Docker image as part of this PR (if the PR relies on executing SPM; I wonder how the funTests pass without it...).
I just noticed that when running
swift package show-dependencies --format jsoninside a directory with aPackage.swiftfile present.Package.resolvedfile automatically gets created and it contains the whole dependency graph (transitive dependencies are included). Maybe it would be a good idea to overridebeforeResolutionmethod inside SPM class and make it runswift package show-dependencies --format jsonso that thePackage.resolvedfile would get created and then we could reuse the same parsing logic and avoid additional complexity? What do you think?The output of
swift package show-dependencies --format jsonincludes the transitive dependencies and associate them with their parents. Is this association also visible in the lockfile? Or is it a flat list?Edit: I just checked: The lockfile does not contain the association between the packages. The output of
swift packagedoes. So maybe a combination of parsing the output ofswift packageand parsing the lockfile is necessary ifallowDynamicVersionsis enabled.
Re @MarcelBochtler That's a very good idea, I will implement dependency associations when parsing Package.swift
So it could also be bootstrapped or included in the Docker image.
Then I'd prefer to have it added to the Docker image as part of this PR (if the PR relies on executing SPM; I wonder how the funTests pass without it...).
@sschuberth And will also update the Dockerfile to have swift installed
@sschuberth @MarcelBochtler I have updated the Dockerfile and Licenses. Also, the SPM analyzer now associates parent-child dependencies whenever parsing Package.swift. I have also provided the required test coverage.
The Package.swift from Vapor is used a test case.
The current state of this PR fails to analyze the vapor project locally:
❯ swift --version
swift-driver version: 1.45.2 Apple Swift version 5.6 (swiftlang-5.6.0.323.62 clang-1316.0.20.8)
Target: x86_64-apple-macosx12.0
15:28:07.773 [DefaultDispatcher-worker-3] ERROR org.ossreviewtoolkit.analyzer.managers.SPM - Resolving SPM dependencies for 'Package.swift' failed with: JsonParseException: Unrecognized token 'Fetching': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
at [Source: (String)"Fetching https://github.com/vapor/websocket-kit.git from cache
Fetching https://github.com/apple/swift-log.git from cache
Fetching https://github.com/vapor/multipart-kit.git from cache
Fetched https://github.com/vapor/websocket-kit.git (0.71s)
Fetched https://github.com/vapor/multipart-kit.git (0.71s)
Fetched https://github.com/apple/swift-log.git (0.71s)
Fetching https://github.com/apple/swift-metrics.git from cache
Fetching https://github.com/apple/swift-nio-http2.git from cache
Fetching https"[truncated 16191 chars]; line: 1, column: 9]
I'm facing the same issue in Docker. However, it was working once, and fails now.
Another thing I noticed is, that if allowDynamicVersions is set to false in the ort.conf, no SPM project is detected.
ORT handles this as UnmanagedProject and the analysis succeeds with exit code 0.
This is handled differently, e.g. for NPM, which exists with exit code 2 and the message:
15:39:45.402 [DefaultDispatcher-worker-3] ERROR org.ossreviewtoolkit.analyzer.managers.Npm - Resolving NPM dependencies for 'package.json' failed with: IllegalArgumentException: No lockfile found in '.'. This potentially results in unstable versions of dependencies. To support this, enable the 'allowDynamicVersions' option in 'ort.conf'.
@MarcelBochtler I have updated the behavior when analyzing Package.swift without allowDynamicVersions set to true to throw IllegalArgumentException (just like NPM does). I have also written an additional test case for that.
The error message looks like this now:
15:08:55.864 [DefaultDispatcher-worker-3] ERROR org.ossreviewtoolkit.analyzer.managers.SPM - Resolving SPM dependencies for 'Package.swift' failed with: IllegalArgumentException: Resolving SPM dependencies from Package.swift might result in potentially unstable versions of dependencies. To support this, enable the 'allowDynamicVersions' option in 'ort.conf'.
About "failing to analyze vapor project locally" - How are you spinning up your container? The following works fine for me.
docker run -v /opt/vapor:/vapor ort -P ort.analyzer.allowDynamicVersions=true --info analyze -i /vapor -o /vapor/ort-out --package-managers SPM
13:04:18.258 [main] INFO org.ossreviewtoolkit.model.config.OrtConfiguration - Using ORT configuration arguments:
ort.analyzer.allowDynamicVersions=true
________ _____________________
\_____ \\______ \__ ___/ the OSS Review Toolkit, version 1f7e11dbb1.
/ | \| _/ | |
/ | \ | \ | | Running 'analyze' under Java 11.0.14.1 on Linux with
\_______ /____|_ / |____| 6 CPUs and a maximum of 1948 MiB of memory.
\/ \/
Environment variables:
ORT_CONFIG_DIR = /root/.ort/config
ORT_DATA_DIR = /root/.ort
JAVA_HOME = /opt/java/openjdk
ANDROID_HOME = /opt/android-sdk
GOPATH = /tmp/go
Looking for analyzer-specific configuration in the following files and directories:
/root/.ort/config/curations.yml (does not exist)
/root/.ort/config/curations (does not exist)
/vapor/.ort.yml (does not exist)
/root/.ort/config/resolutions.yml (does not exist)
The following package managers are activated:
SPM
Analyzing project path:
/vapor
13:04:18.920 [main] INFO org.ossreviewtoolkit.analyzer.PackageManager - Not analyzing directory '/vapor/.build/checkouts/swift-crypto/.git' as it is hard-coded to be ignored.
13:04:18.933 [main] INFO org.ossreviewtoolkit.analyzer.PackageManager - Not analyzing directory '/vapor/.build/checkouts/async-http-client/.git' as it is hard-coded to be ignored.
13:04:18.990 [main] INFO org.ossreviewtoolkit.analyzer.PackageManager - Not analyzing directory '/vapor/.build/checkouts/swift-nio-ssl/.git' as it is hard-coded to be ignored.
13:04:18.993 [main] INFO org.ossreviewtoolkit.analyzer.PackageManager - Not analyzing directory '/vapor/.build/checkouts/websocket-kit/.git' as it is hard-coded to be ignored.
13:04:19.023 [main] INFO org.ossreviewtoolkit.analyzer.PackageManager - Not analyzing directory '/vapor/.build/checkouts/swift-nio-http2/.git' as it is hard-coded to be ignored.
13:04:19.029 [main] INFO org.ossreviewtoolkit.analyzer.PackageManager - Not analyzing directory '/vapor/.build/checkouts/swift-backtrace/.git' as it is hard-coded to be ignored.
13:04:19.033 [main] INFO org.ossreviewtoolkit.analyzer.PackageManager - Not analyzing directory '/vapor/.build/checkouts/swift-metrics/.git' as it is hard-coded to be ignored.
13:04:19.039 [main] INFO org.ossreviewtoolkit.analyzer.PackageManager - Not analyzing directory '/vapor/.build/checkouts/console-kit/.git' as it is hard-coded to be ignored.
13:04:19.041 [main] INFO org.ossreviewtoolkit.analyzer.PackageManager - Not analyzing directory '/vapor/.build/checkouts/routing-kit/.git' as it is hard-coded to be ignored.
13:04:19.047 [main] INFO org.ossreviewtoolkit.analyzer.PackageManager - Not analyzing directory '/vapor/.build/checkouts/multipart-kit/.git' as it is hard-coded to be ignored.
13:04:19.056 [main] INFO org.ossreviewtoolkit.analyzer.PackageManager - Not analyzing directory '/vapor/.build/checkouts/swift-nio-extras/.git' as it is hard-coded to be ignored.
13:04:19.060 [main] INFO org.ossreviewtoolkit.analyzer.PackageManager - Not analyzing directory '/vapor/.build/checkouts/async-kit/.git' as it is hard-coded to be ignored.
13:04:19.064 [main] INFO org.ossreviewtoolkit.analyzer.PackageManager - Not analyzing directory '/vapor/.build/checkouts/swift-nio-transport-services/.git' as it is hard-coded to be ignored.
13:04:19.099 [main] INFO org.ossreviewtoolkit.analyzer.PackageManager - Not analyzing directory '/vapor/.build/checkouts/swift-nio/.git' as it is hard-coded to be ignored.
13:04:19.103 [main] INFO org.ossreviewtoolkit.analyzer.PackageManager - Not analyzing directory '/vapor/.build/checkouts/swift-log/.git' as it is hard-coded to be ignored.
13:04:19.144 [main] INFO org.ossreviewtoolkit.analyzer.PackageManager - Not analyzing directory '/vapor/.git' as it is hard-coded to be ignored.
Found 1 SPM definition file(s) at:
Package.swift
Found 1 definition file(s) from 1 package manager(s) in total.
13:04:19.290 [DefaultDispatcher-worker-3] INFO org.ossreviewtoolkit.analyzer.managers.SPM - Resolving SPM dependencies for 'Package.swift'...
13:04:19.526 [DefaultDispatcher-worker-3] INFO org.ossreviewtoolkit.utils.core.OrtAuthenticator - Authenticator was successfully installed.
13:04:19.555 [DefaultDispatcher-worker-3] INFO org.ossreviewtoolkit.utils.core.OrtProxySelector - Proxy selector was successfully installed.
13:04:19.724 [DefaultDispatcher-worker-3] INFO org.ossreviewtoolkit.utils.core.OrtAuthenticator - Authenticator is already installed.
13:04:19.725 [DefaultDispatcher-worker-3] INFO org.ossreviewtoolkit.utils.core.OrtProxySelector - Proxy selector is already installed.
13:04:23.234 [DefaultDispatcher-worker-3] INFO org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'swift package show-dependencies --format json' in '/vapor'...
13:04:28.767 [DefaultDispatcher-worker-3] INFO org.ossreviewtoolkit.analyzer.managers.SPM - Resolving SPM dependencies for 'Package.swift' took 9.457754970s.
Found 1 project(s) and 15 package(s) in total (not counting excluded ones).
Applied 0 curation(s) from 1 provider(s).
Writing analyzer result to '/vapor/ort-out/analyzer-result.yml'.
Resolved issues: 0 errors, 0 warnings, 0 hints.
Unresolved issues: 0 errors, 0 warnings, 0 hints.
Could you please provide me with a command that you used to spin up the Docker container? Cheers
@UgniusV do you want to move this forward? We're quite interested in SPM support and I could also offer some help if required.
Many thanks for this PR. I rebased it locally on latest master and despite some smaller conflicts SPM.kt needs to change a bit:
Index: analyzer/src/main/kotlin/managers/SPM.kt
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/analyzer/src/main/kotlin/managers/SPM.kt b/analyzer/src/main/kotlin/managers/SPM.kt
--- a/analyzer/src/main/kotlin/managers/SPM.kt (revision ba8f429187caf50d54ecb5ef71ee804db148deaf)
+++ b/analyzer/src/main/kotlin/managers/SPM.kt (date 1668772600414)
@@ -24,6 +24,8 @@
import com.fasterxml.jackson.databind.JsonNode
import com.fasterxml.jackson.databind.ObjectMapper
+import org.apache.logging.log4j.kotlin.Logging
+
import java.io.File
import java.net.URI
@@ -45,8 +47,7 @@
import org.ossreviewtoolkit.model.orEmpty
import org.ossreviewtoolkit.model.utils.DependencyGraphBuilder
import org.ossreviewtoolkit.utils.common.ProcessCapture
-import org.ossreviewtoolkit.utils.core.log
-import org.ossreviewtoolkit.utils.core.normalizeVcsUrl
+import org.ossreviewtoolkit.utils.ort.normalizeVcsUrl
interface SPMCLIExecutor {
fun executeSwift(definitionFile: File): JsonNode
@@ -61,7 +62,7 @@
analyzerConfig: AnalyzerConfiguration,
repoConfig: RepositoryConfiguration,
private val cliExecutor: SPMCLIExecutor
-) : PackageManager(name, analysisRoot, analyzerConfig, repoConfig) {
+) : PackageManager(name, analysisRoot, analyzerConfig, repoConfig), Logging {
private val graphBuilder = DependencyGraphBuilder(SPMDependencyHandler())
@@ -80,7 +81,7 @@
analysisRoot: File,
analyzerConfig: AnalyzerConfiguration,
repoConfig: RepositoryConfiguration
- ) = SPM(managerName, analysisRoot, analyzerConfig, repoConfig, this)
+ ) = SPM(MANAGER_NAME, analysisRoot, analyzerConfig, repoConfig, this)
}
abstract class SPMDependency(private val repositoryURL: String) {
@@ -111,7 +112,7 @@
) : SPMDependency(repositoryURL) {
override fun getVCSInfo(): VcsInfo {
- val vcsInfoFromUrl = VcsHost.toVcsInfo(repositoryURL)
+ val vcsInfoFromUrl = VcsHost.parseUrl(repositoryURL)
if (vcsInfoFromUrl.revision.isBlank()) {
return vcsInfoFromUrl.copy(revision = version)
@@ -149,7 +150,7 @@
}
override fun getVCSInfo(): VcsInfo {
- val vcsInfoFromUrl = VcsHost.toVcsInfo(repositoryURL)
+ val vcsInfoFromUrl = VcsHost.parseUrl(repositoryURL)
if (vcsInfoFromUrl.revision.isBlank() && !state?.revision.isNullOrBlank()) {
return vcsInfoFromUrl.copy(revision = state!!.revision!!)
}
@@ -261,14 +262,14 @@
fun parseAuthorAndProjectFromRepo(repositoryURL: String): Pair<String?, String?> {
val normalizedURL = normalizeVcsUrl(repositoryURL)
- val vcsHost = VcsHost.toVcsHost(URI(normalizedURL))
+ val vcsHost = VcsHost.fromUrl(URI(normalizedURL))
val project = vcsHost?.getProject(normalizedURL)
val author = vcsHost?.getUserOrOrganization(normalizedURL)
if (author.isNullOrBlank()) {
- log.warn { "Unable to parse author name from $repositoryURL VCS URL. Results might be incomplete" }
+ logger.warn { "Unable to parse author name from $repositoryURL VCS URL. Results might be incomplete" }
}
if (project.isNullOrBlank()) {
- log.warn { "Unable to parse project name from $repositoryURL VCS URL. Results might be incomplete" }
+ logger.warn { "Unable to parse project name from $repositoryURL VCS URL. Results might be incomplete" }
}
return author to project
I rebased it locally on latest master and despite some smaller conflicts SPM.kt needs to change a bit:
Thanks @tobiasKaminsky! There's also a newer version of this PR at #6114. Would you be willing to help @MarcelBochtler to move that one forward?
I'll close this PR in favor of the mentioned #6114 to avoid confusion.