dub

dub copied to clipboard

System information

• dub version: 1.19.0
• OS Platform and distribution: osx catalina 10.15.3
• compiler version v2.090.1

Bug Description

Running dub -v build --single bong.d takes way too long and the package is already there locally so there's no need to go on the network:

This is the output:

Using dub registry url 'https://code.dlang.org/'
Refreshing local packages (refresh existing: true)...
Looking for local package map at /var/lib/dub/packages/local-packages.json
Looking for local package map at /Users/aliak/.dub/packages/local-packages.json
Looking for local package map at /Users/aliak/exploras/ms-chanandler-bong/.dub/packages/local-packages.json
No valid package found in current working directory: No package file found in /Users/aliak/exploras/ms-chanandler-bong/, expected one of dub.json/dub.sdl/package.json
Refreshing local packages (refresh existing: false)...
Looking for local package map at /var/lib/dub/packages/local-packages.json
Looking for local package map at /Users/aliak/.dub/packages/local-packages.json
Looking for local package map at /Users/aliak/exploras/ms-chanandler-bong/.dub/packages/local-packages.json
  Version selection for dependency bolts (bolts) of bong is missing.
  Missing dependency bolts ~>1.5.1 of bong
  Version selection for dependency asdf (asdf) of bong is missing.
  Missing dependency asdf ~>0.5.2 of bong
Refreshing local packages (refresh existing: false)...
Looking for local package map at /var/lib/dub/packages/local-packages.json
Looking for local package map at /Users/aliak/.dub/packages/local-packages.json
Looking for local package map at /Users/aliak/exploras/ms-chanandler-bong/.dub/packages/local-packages.json
  Version selection for dependency bolts (bolts) of bong is missing.
  Missing dependency bolts ~>1.5.1 of bong
  Version selection for dependency asdf (asdf) of bong is missing.
  Missing dependency asdf ~>0.5.2 of bong
Checking for missing dependencies.
Search for versions of bolts (1 package suppliers)
Package supplier registry at https://code.dlang.org/ failed with 'Failed to download https://code.dlang.org/api/packages/infos?packages=%5B%22bolts%22%5D&include_dependencies=true&minimize=true', trying fallbacks.
Fallback package supplier registry at https://code-mirror.dlang.io/ failed with 'Failed to download https://code-mirror.dlang.io/api/packages/infos?packages=%5B%22bolts%22%5D&include_dependencies=true&minimize=true'.
Fallback registry at https://dub-registry.herokuapp.com/ succeeded
Package supplier registry at https://code.dlang.org/ failed with 'Failed to download https://code.dlang.org/api/packages/infos?packages=%5B%22bolts%22%5D&include_dependencies=true&minimize=true', trying fallbacks.
Fallback registry at https://dub-registry.herokuapp.com/ succeeded

My local packages are:

And the preamble of bong.d is:

#!/usr/bin/env dub

/+ dub.sdl:
    dependency "asdf" version="~>0.5.2"
    dependency "bolts" version="~>1.5.1"

    #turns on SSE4.2 optimizations when compiled with LDC
    dflags "-mattr=+sse4.2" platform="ldc"
+/

Removing the ~> doesn't help.

How to reproduce?

I think maybe if you get on a bad network, or code.dlang is acting up it can be reproduced?

Expected Behavior

It should work normally if the package is there?

Logs

In your example you do not use exact versions but ~>. Therefore there might be a new package version available. As there is no upgrade command for single file dub packages, it may makes sense to always check the package versions in case of not exact versions.

But I am not really sure how dub behaves in case of exact versions. Here the call to the server would be really superfluous if the exact version is already available.

This pull request is related to your question https://github.com/dlang/dub/pull/1811 and might improve the problem.

In your example you do not use exact versions but ~>. Therefore there might be a new package version available.

Yeah, fair enough for the ~> but at least with exact version it should not go to network for any reason?

Also, in the case of ~> maybe a timeout of something minimal if it can't connect and there's a version on disc which satisfies would be a good route to go?

I thought I saw in the dub coding s.th. like a 24 hours logic. I should not check if last server call was made within last 24 hours. But if this logic works at all and how it works with single dub file packages which might be build in temp folder, I do not know.

I thought I saw in the dub coding s.th. like a 24 hours logic. I should not check if last server call was made within last 24 hours. But if this logic works at all and how it works with single dub file packages which might be build in temp folder, I do not know.

The primary issue with single-file packages is that they currently don't have an associated "dub.selections.json" file, so that the full dependency resolution process runs for every build. Something like having a sub folder in dub's cache folder that stores selection files of the form <hash_of_file_path>.selections.json and extending the CLI to be able to perform regular upgrades of the form dub upgrade --single somepack.d would fix that.

Specifying exact dependency versions as a workaround can of course only theoretically work as long as all dependencies recursively do the same - which will rarely be the case. In practice that would also require a special dependency resolution pre-pass that recognizes this special case, as otherwise it always starts by fetching the set of available versions for each package and then gradually shrinks those sets.

theoretically work as long as all dependencies recursively do the same

Ahh. Of course!

Just for context, I find working with single dub with a #!/usr/bin/env dub at the top of the file and making it executable amazingly effective. So basically having that just behave well with a bad network would be awesome!

So basically, if one were to implement <hash_of_file_path>.selections.json for single files then the network could be avoided for single files should all packages exist?

Yeah, it would be down to just the optional up-to-date check then, AFAIR.

Dub single file packages are quite useful to me. Avoiding network checks for each dub --single invocation would be very useful. Is there a funding pool I can contribute to for this particular issue? I'd be happy to do so.