add $DPATH environment variable for file locations

[WebFreak001]

fix #229

I chose not to add a setting for this to make it easier for other apps in the ecosystem to understand how to locate and change the dub folder, to parse packages, manipulate them, maintain them, etc.

Having a setting would require other users to potentially locate the config files (which has a bunch of exceptions), parse a JSON and find an optional value. Limiting it to an environment variable means that also tools like scripts can easily find the location of this.

[Geod24]

But an environment variable is prone to other kind of issues. For one it's not persistent unless you edit your shell config, which looses locality. Also names are prone to collision. IMO if one wants to get this value in script, we should have a dub config command.

[WebFreak001]

imo if you set your environment variable only locally then you probably only want DUB to store its data where you specified in that session. (maybe set to /tmp/dub to quickly try out some package without downloading it, or different stores for every different user)

This is similar to the JAVAHOME or GOPATH variables. (go works in a similar way with $GOPATH/pkg/... for downloaded packages)

I think $DPATH is a good fit not only for dub, but also for other tools that we might want to include, if we start with some convention there. In that case it would definitely need to be an environment variable to not be dependent on a single tool.

[WebFreak001]

if we don't plan to add other tools, it might also make sense to define a DUB_HOME environment variable instead, which defaults to ~/.dub / %APPDATA%/dub, so the folder can be overwritten in whole.

[WebFreak001]

after discussing with @Geod24 I changed this to also include a setting instead of only an environment variable. This is so that system-wide changes can be made, e.g. by linux package maintainers, but also kept an environment variable so that it's possible to create a quick temporary clean local dub package storage or in general if it might be needed for some specific user's projects on the system.

To support changing the path of the packages (and also the config), the config in /etc/dub/settings.json (or /usr/...) will be read first and if the dubHome property is set in there, loading of further configs will be redirected to there, as well as the packages being stored there.

[dk949]

Hi 👋

I've been working on something similar (I guess now I'll just wait for this to merge), and just wanted to share some thoughts from my implementation.

I also considered something like DUB_HOME or DUB_PATH, but came to the conclusion that it's far too generic, which means users may have it set to control some other part of their system (DPATH would suffer the most from this).

I am not sure how common this is on Windows or Mac, but a lot of Linux programs use environment variables to control the location of their config/cache/etc files (see this page for some examples).

In the end I settled for DUB_SYSTEM_SETTINGS_PATH, DUB_USER_SETTINGS_PATH and DUB_LOCAL_REPOSITORY_PATH. As I understand you intend other D tooling to use this directory, so maybe something like DLANG_HOME (in line with JDK_HOME)?

As for "...it's not persistent unless you edit your shell config", well yes, but users are already used to doing this. zsh even has a special config file pretty much for this purpose (.zshenv).

My impl for reference.

[WebFreak001]

I made it now so that environment variables can only change both folders at once, but you can use the settings for finer control of user settings and repository path, or use the general "dubHome" setting to set both.

Additionally for package maintainers and sysadmins it's probably interesting to have environment variables in the settings, for example to have a different user store for each user, so I added that as well.

See the changelog entry for details https://github.com/dlang/dub/blob/5addac0e22036cf6b328bda72ac87c47d55d0331/changelog/dpath.dd

[Geod24]

Any chance we could get https://github.com/dlang/dub/pull/2343 before, since it overhaul this part of the code and simplifies the logic ? I can then rebase your branch on top of it, if you'd like.

[WebFreak001]

I just realized we read /etc/ before /var/lib...

I don't think that's the case, the systemSettings path is the very first config that is read, everything after that overrides the configs.

[Geod24]

I don't think that's the case, the systemSettings path is the very first config that is read, everything after that overrides the configs.

My bad, I meant the other way around :facepalm: I think we should read /etc/ first, but we currently don't.

[WebFreak001]

I don't think that's the case, the systemSettings path is the very first config that is read, everything after that overrides the configs.

My bad, I meant the other way around facepalm I think we should read /etc/ first, but we currently don't.

I strongly feel that we should not read /etc first, but /var/lib, like currently already done. I think the idea behind that is:

1. consistency with other programs (e.g. systemd)
2. the /var/lib/... path is maintained by the distro package maintainers and should never be updated by the user, this implies:
   • any setting in /var/lib should not override the user-set settings, thus it must not be loaded after the user settings in /etc

[Geod24]

One use-case that might be important is the default placement path. Currently the the localRepository (user HOME on POSIX) is the default placement. Some users might want to use system, and some might want to use local (e.g. in the package itself). Currently the always have to pass an argument to DUB.

If you prefer we can tackle it in a separate PR, but for me it makes much more sense to think of them together (I thought the scope of this PR was originally bigger).

[WebFreak001]

I think a new setting to set the default placement location should be part of a separate PR.

[dk949]

I though you were planning to tie the location of localRrpository (aka $HOME/.dub, aka default placement) to $DPATH. If this is the case, wouldn't it make it somewhat inseparable from this PR?

(I too have possibly misunderstood the scope of these changes).

[WebFreak001]

rebased to new yaml loader

[WebFreak001]

I though you were planning to tie the location of localRrpository (aka $HOME/.dub, aka default placement) to $DPATH. If this is the case, wouldn't it make it somewhat inseparable from this PR?

(I too have possibly misunderstood the scope of these changes).

no the default placement location setting I was talking about would set a default for the dub --cache=* setting (can set it to local, system or user)

[dk949]

Ohh, that makes sense. My bad.

[WebFreak001]

@Geod24 what's your opinion on this?

[WebFreak001]

adjusted to review, can rebase if it's fine now

[WebFreak001]

fixed left-overs

[WebFreak001]

thanks!