[RFC] `dub list-dependencies` + `dub build-dependency`

[vpanteleev-sym]

trafficstars

Automating efficient builds of Dub packages at scale is a little challenging right now:

• We need to discover all dependencies, and be able to download them ahead of time. (https://github.com/dlang/dub/issues/1723 helps here)
• We would like to be able to build a package's dependencies separately from the package source code, so that we can quickly iterate while developing the package itself without having to rebuild all dependencies
• Ideally, we would like the building of dependencies to be granular, so that bumping one dependency does not require rebuilding all other dependencies

Currently, e.g. in Containerfiles, we do something like the following:

COPY dub.sdl dub.selections.json /build/
# build just the dependencies, to produce a separately-cached layer
RUN cd /build && dub build || true  # ignore failure due to missing source code
# now copy and build the source code
COPY src /build/
RUN cd /build && dub build

This helps a little, but is ugly and it's confusing to see error messages as part of the expected build flow.

To help with the above, I'm thinking that adding two commands would be useful:

1. dub list-dependencies - get a machine-readable list of dependencies.
   • The output should be some kind of list (a JSON array or newline-delimited JSON objects), so that consumers can easily split out individual dependencies.
   • Each item in the list, in one way or another, should be trivially translatable to a dub build-dependency invocation (see below).
   • Ideally, the output should contain an easily understandable recipe for obtaining the dependency source code and verifying its integrity, so that we can provide it to Dub for querying further sub-dependencies and building. (Including this here would absolve having to also parse dub.selections.json.)
   • Ideally, the output should represent or contain an identifier which fully encompasses the dependency's specification and build settings, so that if two packages use the exact same dependency in the exact same way, we could build this dependency only once. This could be a hash of the item itself, assuming that it doesn't contain any environment-specific information such as absolute paths on the local filesystem.
2. dub build-dependency - accept an item from the list produced by list-dependencies, and build it
   • Ideally, we would be able to somehow provide to Dub the package source code, and any sub-dependencies built in the same way, so that this step can run without Internet access.
   • The build output should be in a predictable location so that we can grab it and provide it as a pre-built dependency when building the main package.

Both commands should only need to read dub.sdl / dub.selections.json, without referring to the package source code.

One trivial way to combine them, replacing the method mentioned in the Containerfile example above, could be (assuming list-dependencies emits one line per dependency, and build-dependency builds all lines on standard input):

$ dub list-dependencies | dub build-dependency

More advanced build tools (Nix, buck, etc.) can choose to be more granular by getting the list and then building each dependency separately / in parallel.