Config option to disable the currently-non-optional internal buildsystem

[Abscissa]

This may be controversial and generate a lot of debate, but I've felt for a long time that it's extremely important, so here goes with the ball rolling...

Dub claims (and for very good reason) to be usable as a mere package manager without the (supposedly optional) built-in build system. But there is no mechanism in dub.(sdl|json) to disable the internal build system so that a CLI command to do the build can be provided. So currently, this claim is factually false.

For the sake of NOT continuing to divide the D library ecosystem into "projects that use dub as a buildsystem" vs "projects that use their own choice of buildsystem, such as Ninja, Reggae, rdmd, cmake, etc.and therefore CANNOT participate in the official D package ecosystem", the dub package format needs a way to specify "disable internal dub buildsystem, project/config/whatever will be built using the folling CLI command instead." Naturally, the usual ENVARS provided to prebuild/etc commands would still apply.

[s-ludwig]

To recap, we have the following possibilities now:

• Invoke a foreign build system, let it invoke DUB for a particular dependency
• Invoke DUB, use its build description and its own build system
• Invoke DUB, use its build description, but generate a foreign build description (CMake etc.)
• Invoke DUB, perform the compile step through a custom script, but use DUB's linker phase

So what is missing is just a flag (e.g. customBuild true) to specify that one of the build commands is expected to generate the target binary instead of relying on DUB's linker phase, right? We'd just have to make sure that the path of the target binary is controllable by DUB - for example, its own builder would let the custom command create the binary in .dub/build/xxx/ so that it can take part in the rest of the build process.

(Just as a side note, I think we need to be careful to not actively endorse the use of custom build systems, but the possibility should definitely be there.)

[Abscissa]

First, a few notes on those four options (mainly just FWIW):

1. Invoke a foreign build system, let it invoke DUB for a particular dependency

It's worth keeping in mind that this choice, realistically speaking (albeit not technically), prevents a project from being published in the dub package repository and used as a dependency in other dub-based projects. (Yea, it's technically possible, but not without forcing other projects that depends on the given project to jump through extra hoops.) So while a project is certainly free to do this, DUB should endeavor to be flexible enough that it's never anyone's only option.

2. Invoke DUB, use its build description and its own build system

Right. The "standard" DUB usage.

3. Invoke DUB, use its build description, but generate a foreign build description (CMake etc.)

Should be noted that this approach isn't always appropriate for all buildsystems (ex: reggae or rdmd) or for all situations (ex: when a project requires control over something that DUB doesn't offer control over, or choosing to write the build process description using a tool other than dub).

4. Invoke DUB, perform the compile step through a custom script, but use DUB's linker phase

AFAICT, this isn't currently possible. You can launch a custom compile script via "preBuildCommands" or "postBuildCommands", but dub will still attempt to compile on its own. I've attempted various workarounds in the past, but was ultimately forced to conclude these workarounds were just too brittle and complicated to be a workable approach.

So what is missing is just a flag (e.g. customBuild true) to specify that one of the build commands is expected to generate the target binary instead of relying on DUB's linker phase, right?

Instead of relying on dub's compile AND link phases, yes.

If dub wants to also support selectively disabling JUST dub's built-in compile OR link and leave the other up to a custom script, I don't have a problem with that, but I don't know how useful it would be.

We'd just have to make sure that the path of the target binary is controllable by DUB - for example, its own builder would let the custom command create the binary in .dub/build/xxx/ so that it can take part in the rest of the build process.

AIUI, dub wouldn't be controlling the path of the target binary, but it would be up to the project's developer to ensure the custom buildscript places the target binary wherever dub.sdl/json says it will be. This can be aided by making sure the envvars dub passes to custom scripts includes the target binary path/name (I think this is already the case). The buildscript can simply choose to read that envvar and abide by it.

[s-ludwig]

Invoke DUB, perform the compile step through a custom script, but use DUB's linker phase

AFAICT, this isn't currently possible. You can launch a custom compile script via "preBuildCommands" or "postBuildCommands", but dub will still attempt to compile on its own. I've attempted various workarounds in the past, but was ultimately forced to conclude these workarounds were just too brittle and complicated to be a workable approach.

What I had in mind was something like this, but I admit that I didn't test it yet:

sourcePaths // clear default source paths, so DUB doesn't compile anything
sourceFiles "foo.o"
preBuildCommand "gcc -c source/foo.c -o foo.o"

If dub wants to also support selectively disabling JUST dub's built-in compile OR link and leave the other up to a custom script, I don't have a problem with that, but I don't know how useful it would be.

Since disabling the compile phase is just a matter of specifying no source files, I don't think we need a special flag for that. But maybe to support selectively disabling the linker phase, we need to define a new environment variable that captures the list of generated object files.

We'd just have to make sure that the path of the target binary is controllable by DUB - for example, its own builder would let the custom command create the binary in .dub/build/xxx/ so that it can take part in the rest of the build process.

AIUI, dub wouldn't be controlling the path of the target binary, but it would be up to the project's developer to ensure the custom buildscript places the target binary wherever dub.sdl/json says it will be. This can be aided by making sure the envvars dub passes to custom scripts includes the target binary path/name (I think this is already the case). The buildscript can simply choose to read that envvar and abide by it.

If the feature is not supposed to be disruptive w.r.t. the umbrella build system, this needs to be controllable by DUB or that respective build system. For example DUB itself places the final binary within .dub/.., so that it can be cached for later runs. The actual target file specified in the package recipe then gets hard linked to there.

[Abscissa]

What I had in mind was something like this, but I admit that I didn't test it yet:

sourcePaths // clear default source paths, so DUB doesn't compile anything
sourceFiles "foo.o"
preBuildCommand "gcc -c source/foo.c -o foo.o"

Oh, never occurred to me to clear the source paths like that. I just assumed

sourcePaths //blank

...would be equivalent to omitting sourcePaths, and therefore would include "src" and/or "source" by default.

Last time I tried it, I created "dummy/dummy.d" and set sourcePaths to "dummy", but then newer versions of DUB made some complaint about "dummy" not being in importPaths (and there had been some other awkwardness before that, too, IIRC). I also would've assumed that feeding DUB an empty "dummy" directory, with no "dummy.d" file, would also cause issues, much like running a compiler on the command line without specifying any source files.

I'll have to try that approach and see what happens.

If the feature is not supposed to be disruptive w.r.t. the umbrella build system, this needs to be controllable by DUB or that respective build system. For example DUB itself places the final binary within .dub/.., so that it can be cached for later runs. The actual target file specified in the package recipe then gets hard linked to there.

I suspect we may both be on the (mostly) same page here but with some confusion on "controllable by". Here's how I saw things:

Currently, there are several pieces of info dub passes to preBuildCommand (etc.) process via envvars. IIRC, targetName and targetPath are among them (and if not, they probably should be). If a preBuildCommand instead of DUB is doing the compiling/linking, then the preBuildCommand has a responsibility to generate a target that matches targetName and targetPath.

I didn't know about the hardlink inside .dub/ though. (Is that true on Windows as well?) Maybe, instead, should DUB create the entity inside ./dub as a hardlink to targetPath/targetName, where targetPath/targetName is the one directly generated by the linker? Then, a custom buildscript can generate its own target wherever it needs to and as long as they properly set up targetPath/targetName in dub.sdl, DUB will know what file to hardlink into .dub/...

Or, I guess as (I think) you suggest, DUB could continue telling the linker to generate into .dub/ and then hardlink targetPath/targetName to that. Then, custom buildscripts would have to know to generate to the right filepath inside ./dub instead of directly to targetPath/targetName. Seems a little harder that way to me for buildscript authors, but either way sounds like it would work.

[s-ludwig]

Okay, no I think we are actually on the same page. What I mean with "control" is that it passes the target path/name/filename to the custom build command in the form of env vars, and that it can depend on the generated binary being there. This is simply the approach that results in the most elegant/clean implementation within DUB.

The other direction, where the custom command would write directly to the user supplied path and then DUB would hard link/copy to .dub/ would also work, but would require some hacks to not break the up-to-date checks. It also shifts some of the logic to the custom script, for example in cases where different configurations have different target paths assigned to them, of if we ever implement build type specific target paths, so it would presumably be less future proof.

[MartinNowak]

You'd still get the division as most people don't have ninja, meson, or X installed. This would also make it harder to test dub packages in different environments, e.g. on ci.dlang.io. In fact customization is slightly at odds with a standardized/uniform package ecosystem.

Personally I'd say the latter is by far the more important goal, and there is plenty of precedence from other package managers to integrate a simple build system (Haskell's Cabal, Rust's Cargo, Swift's Package Manager, Gems with Extensions).

Could you list a few arguments why support for external build systems is important?

Also it seemed smarter to finish the C/C++ build feature before making additional architecture decisions.

[Abscissa]

Any build system used by a package, like ninja, meson, etc should be available AS a dub package, therefore the issue is handled via dub's existing dependency-handling.

If there's anything preventing such tools from being packaged and obtained through dub, then naturally that needs to be fixed.

[MartinNowak]

As none of those are D tools, you have a bit of a bootstrap issue. If the build system is defined as a dub package itself (e.g. a binary distribution), then that would be some sort of plugin build system.

naturally that needs to be fixed.

But again, could you please provide some more motivation and arguments why such a feature is necessary.

[Abscissa]

But again, could you please provide some more motivation and arguments why such a feature is necessary.

Frankly, myself and others have been doing that for years only to keep getting roadblocked by more "i disagree just because. try to convince me again." I'm not going to keep indulging that cycle.

[Geod24]

This issue actually motived me to improve dub fetch: https://github.com/dlang/dub/pull/2786

I know that is not what this issue asks, but I don't think having a flag like customBuild makes sense. If one want to use dub as a mere package manager, they would only need a few fields from the package recipe (name, dependencies). Trying to integrate another build system into dub will just not work.

With the aforementioned PR, you can do the dependency resolution independently of the build system. What is missing is an easy way to get the paths Dub owns (package path). dub describe should give you this but I haven't tried it myself.