Generation Software Bill of Materials (SBOM)

[gamlerhart]

Motivation: In some companies, the development team has to produce Software Bill of Materials (SBOM) for their project for compliance reasons: To track dependencies and licenses across their organisation. Provide a Module that produces SBOMs in JSON format.

Changes in the core: Extended the .getArtifact to return the coursier.Resolution as well. This is then used to get the license information.

Outside the core: Add a SBOM contrib module

• Generate the most basic CycloneDX SBOM files Supporting Java modules for a start
• Provide a basic upload to the Dependency Track server

[gamlerhart]

I've these high level questions

• I implemented the SBOM json from scratch, so that we can use the UPickle etc and do not add libraries: The alternative is to use the CycloneDX 'model' library, that mostly implements the JSON and some hashing. But that then adds this extra library etc: More stuff to download, more stuff in the classpath etc. So, the question is: What is preferred in general: Avoiding external libraries when possible? Or go for maximum comparability and include more external libraries?

• I adding this to the 'contrib' section the right place? The alternative seems to have it as a complete external library. However, that adds extra maintenance burdens: Pushing it to Maven repos, compiling it, versioning it. So, having it in-sync seems better.

• I've extended the returned data from the Coursier .artifacts method. I think that is ok, because that method wasn't yet published in 0.12.9?

(update: Android tests are fixed)

[gamlerhart]

• I'll stick with the upickl JSON for a start. That integrates well with Mill eco system. I consider the CyclonDX library only if there is too painful to directly write the Json.

[gamlerhart]

Rebased & Tests passing =)

[lihaoyi]

@gamlerhart sorry we've overlooked this. Could you rebase this one more time, and flesh out the PR description with a more verbose explanation of what this is all about? I'm not familiar with SBOMs myself and need help understanding what's going on here

[gamlerhart]

Hi everyone. I'll try to get to it and rebase it.

@lihaoyi At this stage the PR is the most minimal possible SBOM. The idea is to get the 'foot in the door' for SBOM support. And then see what people actually need and extend it.

Eg. Missing atm:

• Having different hash flavors (md5, sha1, sha2 etc) for include library. I added SHA2 only.
• Having a report for the build tool itself
• Support for non-maven dependencies.

Anyway, that is something to add if there is demand =).

[lefou]

@gamlerhart To get an idea, how such an SBOM looks like, could you attach an SBOM generated with this PR for Mill itself?

[gamlerhart]

Ok...I'm not sure how to create a SBOM for Mill itself with a dev build:

What I tried:

1. I've mill checked out in a second location
2. I add //| mvnDeps: ["com.lihaoyi::mill-contrib-sbom:$MILL_VERSION"] to the build.mill in that directory, to add the contribution library
3. I then run with this branch /mill dist.run /home/roman/mill-test-checkout resolve dist._

Then I get this error:

============================== resolve dist._ ==============================
[build.mill-60/65] compile
[build.mill-60] [info] compiling 22 Scala sources to /home/roman/dev-temp/mill/out/mill-build/compile.dest/classes ...
[build.mill-60] [error] -- [E046] Cyclic Error: /home/roman/dev-temp/mill/mill-build/src/millbuild/MillScalaModule.scala:12:67 
[build.mill-60] [error] 12 |trait MillScalaModule extends ScalaModule with MillJavaModule with ScalafixModule { outer =>
[build.mill-60] [error]    |                                                                   ^
[build.mill-60] [error]    |                             Cyclic reference involving val <import>
[build.mill-60] [error]    |
[build.mill-60] [error]    |                              Run with -explain-cyclic for more details.
[build.mill-60] [error]    |
[build.mill-60] [error]    | longer explanation available when compiling with `-explain`
[build.mill-60] [error] -- [E46] /home/roman/dev-temp/mill/libs/javalib/package.mill:123:53
[build.mill-60] [error] 123 │  object worker extends MillPublishScalaModule with BuildInfo {
[build.mill-60] [error]     │                                                    ^
[build.mill-60] [error]     │Cyclic reference involving val <import>
[build.mill-60] [error]     │
[build.mill-60] [error]     │ Run with -explain-cyclic for more details.
[build.mill-60] [error] -- [E8] /home/roman/dev-temp/mill/website/package.mill:3:12
[build.mill-60] [error] 3 │import org.jsoup._
[build.mill-60] [error]   │           ^^^^^
[build.mill-60] [error]   │value jsoup is not a member of org
[build.mill-60] [error] -- [E8] /home/roman/dev-temp/mill/runner/meta/package.mill:4:21
[build.mill-60] [error] 4 │import mill.contrib.buildinfo.BuildInfo
[build.mill-60] [error]   │                    ^^^^^^^^^

So, I think my plain dist.run doesn't work with the mill build itself.

[lihaoyi]

@gamlerhart since Mill has a meta-build in mill-build/build.mill, you need to add your dependency there instead of in the build header

[lefou]

1. Make a local release of your development snapshot:

> MILL_STABLE_VERSION=1 mill dist.installLocalCache
...
[7152] /home/lefou/.cache/mill/download/1.0.4-37-a03078

2. Add the plugin to the Mill build

diff --git a/mill-build/build.mill b/mill-build/build.mill
@@ -15,6 +15,7 @@
     // TODO: implement empty version for ivy deps as we do in import parser
     mvn"com.lihaoyi::mill-contrib-buildinfo:${mill.api.BuildInfo.millVersion}",
     mvn"com.goyeau::mill-scalafix_mill1:0.6.0",
-    mvn"org.jsoup:jsoup:1.21.2"
+    mvn"org.jsoup:jsoup:1.21.2",
+    mvn"com.lihaoyi::mill-contrib-sbom:${mill.api.BuildInfo.millVersion}"
   )
 }

3. Use the just released Mill version via MILL_VERSION:

> MILL_VERSION="1.0.4-37-a03078" mill ...

or edit the build.mill:

diff --git a/build.mill b/build.mill
@@ -1,4 +1,4 @@
-//| mill-version: 1.0.4-26-a6e4c1
+//| mill-version: 1.0.4-37-a03078
 //| mill-jvm-opts: ["-XX:NonProfiledCodeHeapSize=250m", "-XX:ReservedCodeCacheSize=500m"]
 //| mill-opts: ["--jobs=0.5C"]
 

[gamlerhart]

Thanks for the help =). That worked:

The SBOM for Mill itself looks like this: sbom.json

Or for example when viewing it in a tool like dependency track:

Reminder: It is a foot in the door state:

• There are tons and tons of data that can be potentially filled. Depends on the feedback on what people actually need.
• Plus: Afaik there is ways to represent sub-modules etc, but the generated SBOM is the flat list. Same: Will see if people actually need that.

From my side: I probably more interested to also get the JavaScript/npm dependencies next. As having JVM backend + NPM in the frontend is so common.