DUB aggregates and fetch all dependencies regardless of selected configuration

[schveiguy]

Currently when dub is run on a project that has multiple configurations, some of which select optional dependencies, dub will determine packages that should be downloaded in case those dependencies are selected, and download them proactively. However, this makes little sense if the configuration is not intended to be used in common cases. For example, vibe.d's tls:botan configuration will select botan to download, even if you never intend to compile the botan version.

In addition, different configurations may require different versions of dependencies, creating a conflict of which version of a package should be downloaded for those versions. This can even happen within subconfigurations of packages that you will NEVER build. For example, the test configurations for 2 dependency projects may depend on DIFFERENT versions of a subdependency, even if I'm not going to build the specific subconfiguration.

Example:

...
         "dependencies": {
		"mysql-native": "~>2.3.0"
	},
        "configurations" : [
            {
                "name" : "unittest",
                "dependencies" : {
                    "unit-threaded": "==0.7.37"
                }
            }
        ]
...

To solve this problem, I propose that we:

1. Do not download packages until needed for build (this will save from unnecessarily downloading never-used packages)
2. Change dub.selections.json into a key/value array, where the specific dub.selections.json configuration depends on which configurations were selected for building. In practice, you mainly only build one or two configurations, so this will still make the dub.selections.json file relatively static.

For an example of the proposed format, it might be something like:

{
    "fileVersion": 2,
    "selections" : [
        {
            "configuration" : {
                "myproj" : "unittest",
                "mysql-native" : "default",
                "unit-threaded" : "default"
            },
            "versions" : {
                "mysql-native" : "2.3.0",
                "unit-threaded" : "0.7.40"
            }
        },
        ...
    ]
}

[schveiguy]

@wilzbach @ghost91-

[andre2007]

In the FAQ the first 2 topics are related to your proposal and explains the difficulties https://github.com/dlang/dub/wiki/FAQ

Are you aware of this FAQ?

[schveiguy]

Thank you for the FAQ, I was not aware of it.

The topics do not explain why the current model is better than one which would select the packages needed when needed. As an end user, I don't care AT ALL how dub actually figures out what packages to select, and how the internal workings happen. I just want it to build things that make sense, and only download packages which it needs.

The problems identified in the first part are not made up -- if I want to build for example, using unit threaded of a particular version for my project, and another dependency has a NON-SELECTED configuration where it is using a different version of unit-threaded, the build will succeed. BUT dub does not allow this. It's a limitation that is not easily fixed by me, as I can't affect what the dependency puts in their dub.json file.

Note also that downloading packages that are not going to be used can and should be delayed until actually needed. Either one of these problems could potentially be tackled as separate issues, but trying to fix them both at once can result in a much more cohesive solution.

Other language build systems have solved these problems. If dub cannot do this, it simply will stop being used. See threads like this for instance: https://forum.dlang.org/post/[email protected]

[andre2007]

I also want to see this issue fixed. There was a pull request in the past to solve this issue https://github.com/dlang/dub/pull/1148

[schveiguy]

Thanks for the link! I will take a look.

[Kozzi11]

Please this ASAP or I will really need to stop using dub, it is impossible to use right now. Everytime I need tu upgrade to new version of dub packages I will end up with: Could not resolve configuration for package tachograf_bridge

[Geod24]

Please this ASAP or I will really need to stop using dub, it is impossible to use right now.

A workaround we are using is to provide a path-based dependency with a dummy dub.json. E.g. for botan:

{
    "name": "botan",
    "targetType": "none"
}

dub.selections.json:

{
	"fileVersion": 1,
	"versions": {
                // ...
		"botan": { "path": "submodules/botan/" },
                // ...
	}
}

[Geod24]

Renamed and pinned the issue as it is the most common one.

[ichordev]

Does anyone have any plans to resolve this issue or put in a DEP?