A single `dub build` refreshes the local package cache 3 times

[Geod24]

dub build -v inside of a previously build dub folder:

Using dub registry url 'https://code.dlang.org/'
Refreshing local packages (refresh existing: true)...
Looking for local package map at /var/lib/dub/packages/local-packages.json
Looking for local package map at /home/geod24/.dub/packages/local-packages.json
Looking for local package map at /home/geod24/projects/dlang/dub/.dub/packages/local-packages.json
Note: Failed to determine version of package msgpack-d at .. Assuming ~master.
Determined package version using GIT: zmqd 1.2.0+commit.8.gbf88663
Ignoring version specification (>=0.0.0) for path based dependency ./
Ignoring version specification (>=0.0.0) for path based dependency ./
Ignoring version specification (>=0.0.0) for path based dependency ./
Ignoring version specification (*) for path based dependency ../..
Ignoring version specification (*) for path based dependency ../..
Ignoring version specification (*) for path based dependency ../..
Ignoring version specification (*) for path based dependency ../..
Ignoring version specification (*) for path based dependency ../..
Note: Failed to determine version of package msgpack-d at .. Assuming ~master.
Ignoring version specification (>=0.0.0) for path based dependency .
Ignoring version specification (>=0.0.0) for path based dependency .
Ignoring version specification (>=0.0.0) for path based dependency .
Ignoring version specification (>=0.0.0) for path based dependency .
Ignoring version specification (>=0.0.0) for path based dependency .
Ignoring version specification (>=0.0.0) for path based dependency .
Ignoring version specification (>=0.0.0) for path based dependency .
Ignoring version specification (>=0.0.0) for path based dependency .
Determined package version using GIT: windows-headers 1.0.4+commit.2.g4922dbd
Determined package version using GIT: windows-headers 1.0.4+commit.2.g4922dbd
Determined package version using GIT: jwtd 0.5.0
Determined package version using GIT: dub 1.29.1+commit.104.ge95f403e
Refreshing local packages (refresh existing: false)...
Looking for local package map at /var/lib/dub/packages/local-packages.json
Looking for local package map at /home/geod24/.dub/packages/local-packages.json
Looking for local package map at /home/geod24/projects/dlang/dub/.dub/packages/local-packages.json
Ignoring version specification (*) for path based dependency ../..
Ignoring version specification (*) for path based dependency ../..
Ignoring version specification (*) for path based dependency ../..
Ignoring version specification (*) for path based dependency ../..
Ignoring version specification (*) for path based dependency ../..
  Found dependency vibe-d:http 0.9.5
    Found dependency vibe-d:inet 0.9.5
      Found dependency vibe-d:data 0.9.5
        Found dependency vibe-d:utils 0.9.5
          Found dependency stdx-allocator 2.77.5
      Found dependency vibe-d:textfilter 0.9.5
        Found dependency vibe-core 1.22.4
          Found dependency eventcore 0.9.20
            Found dependency taggedalgebraic 0.11.22
            Found dependency libasync 0.8.6
              Found dependency memutils 1.0.4
      Found dependency vibe-d:stream 0.9.5
    Found dependency vibe-d:tls 0.9.5
      Found dependency openssl 3.2.2
    Found dependency vibe-d:crypto 0.9.5
      Found dependency mir-linux-kernel 1.0.1
    Found dependency diet-ng 1.8.1
Refreshing local packages (refresh existing: false)...
Looking for local package map at /var/lib/dub/packages/local-packages.json
Looking for local package map at /home/geod24/.dub/packages/local-packages.json
Looking for local package map at /home/geod24/projects/dlang/dub/.dub/packages/local-packages.json
Ignoring version specification (*) for path based dependency ../..
Ignoring version specification (*) for path based dependency ../..
Ignoring version specification (*) for path based dependency ../..
Ignoring version specification (*) for path based dependency ../..
Ignoring version specification (*) for path based dependency ../..
  Found dependency vibe-d:http 0.9.5
    Found dependency vibe-d:inet 0.9.5
      Found dependency vibe-d:data 0.9.5
        Found dependency vibe-d:utils 0.9.5
          Found dependency stdx-allocator 2.77.5
      Found dependency vibe-d:textfilter 0.9.5
        Found dependency vibe-core 1.22.4
          Found dependency eventcore 0.9.20
            Found dependency taggedalgebraic 0.11.22
            Found dependency libasync 0.8.6
              Found dependency memutils 1.0.4
      Found dependency vibe-d:stream 0.9.5
    Found dependency vibe-d:tls 0.9.5
      Found dependency openssl 3.2.2
    Found dependency vibe-d:crypto 0.9.5
      Found dependency mir-linux-kernel 1.0.1
    Found dependency diet-ng 1.8.1
Generating using build
Configuring dependent dub, deps:
Performing "debug" build using /usr/bin/dmd for x86_64.
dub 1.29.1+commit.104.ge95f403e: target for configuration "application" is up to date.
Using existing build in /home/geod24/projects/dlang/dub/.dub/build/application-debug-linux.posix-x86_64-dmd_v2.100.0-19DC204D1E339FDB712B2C578C4AA2C0/.
Copying target from /home/geod24/projects/dlang/dub/.dub/build/application-debug-linux.posix-x86_64-dmd_v2.100.0-19DC204D1E339FDB712B2C578C4AA2C0/dub to /home/geod24/projects/dlang/dub/bin
To force a rebuild of up-to-date targets, run again with --force.

There is a lot going on there. Every time we see:

Refreshing local packages (refresh existing: true)...
Looking for local package map at /var/lib/dub/packages/local-packages.json
Looking for local package map at /home/geod24/.dub/packages/local-packages.json
Looking for local package map at /home/geod24/projects/dlang/dub/.dub/packages/local-packages.json

It means we re-scan everything in .dub/packages. That means re-reading ALL dub.json / dub.sdl. And the "Ignoring version specification" thing are just showing that, yes, we are re-reading those files.

Step 1: We should not rescan THREE TIMES. Step 2: We should not read package files for packages we do not use.

[Geod24]

Twice now: https://github.com/dlang/dub/pull/2339

[Geod24]

Currently, we need to read the package recipe because we don't rely on the path for the package name or version. So I think the best way forward is to change the way we store packages. We currently store things as $BASE/$PACKAGE_NAME-$VERSION/$PACKAGE_NAME. The $VERSION is sanitized for Optlink, we remove +. For the vast majority of versions, switching to $BASE/$PACKAGE_NAME/$VERSION/$PACKAGE_NAME would allow us to be more lazy.

[WebFreak001]

I think reading it from the path would be a lot better:

• the information is already there
• we are lowering file reads (which is a lot slower, especially on HDD)
• no redundant saving of the version

[Geod24]

Another approach, which I think will be easier in the short term, is to build the expected path from the version we are looking for. Currently we scan the package dir, store package names and versions, then when we need a package, we look it up in the PackageManager. Instead we could just look it up in the PackageManager, and if there's a cache miss, try to look it up on the disk.

We'd still need to scan the custom cache paths because we currently have looser requirements on them, but it would improve things in the common case.