pkg: cannot run `dune pkg lock` while offline

[gridbugs]

trafficstars

This should work as long as the opam repoo is currently checked out. Currently it produces the error:

$ dune pkg lock
fatal: unable to access 'https://github.com/ocaml-dune/opam-overlays.git/': Could not resolve host: github.com
Error: Failed to run external command:
'git ls-remote "https://github.com/ocaml-dune/opam-overlays.git"'
Hint: Check that this Git URL in the project configuration is correct:
"https://github.com/ocaml-dune/opam-overlays.git"
fatal: unable to access 'https://github.com/ocaml/opam-repository.git/': Could not resolve host: github.com
Error: Failed to run external command:
'git ls-remote "https://github.com/ocaml/opam-repository.git"'
Hint: Check that this Git URL in the project configuration is correct:
"https://github.com/ocaml/opam-repository.git

This was on a machine with a copy of the opam repo in ~/.cache/dune/git-repo/, so no new information should have been needed by dune pkg lock requiring internet access.

[rgrinberg]

Did you set the commit hash for the opam repositories that you were using? If not, then this behavior is expected. We refresh the opam repositories on every operation that requires consulting them.

[gridbugs]

I question whether this is the most practical behaviour. It makes dune harder to use in places with spotty internet, and also when trying to get work done on planes (how I came across this issue!). When fetching the repo fails, but there's a local copy of the repo in the expected place, why not print a warning and fall back to the existing repo rather than failing completely.

[rgrinberg]

In opam, updating the opam repository is done manually. While that seems reasonable at first, it's something that surprisingly many users forget to do. See all the issues on discuss that were fixed by running $ opam update. So I thought that in dune, we'd no longer require the user to remember this. That has a trade-off as well of course, as it's not obvious what to do when there's no internet. Though not having internet seems like a more rare situation.

There are two adequate workarounds for this problem:

• Set the opam repository to be something local on the filesystem
• Copy-paste an existing commit hash of the opam repo that already exists in the revision store from previous fetches into the workspace file.

We could improve the error message to tell the user that the above options are possible.

I don't know how to automatically fallback to older version of the repo because I don't know how to detect the internet is not working from running curl.

[gridbugs]

What would be the harm in using the repo in ~/.cache/dune/git-repo if it exists?

[rgrinberg]

Do you mean trying that after we fail to update the opam repository?

[gridbugs]

Yep

[gridbugs]

But also I think we should allow users to do a solve without forcing an update. Imagine you're trying to add a new dependency to a project and you inadvertently introduce a problem caused by switching to a later version of a dependency that introduces a bug. (That's a different issue from the one we're currently discussing.)

[rgrinberg]

Yep

Because updating the repository could fail for multiple reasons. Some of them should not have a fallback like that, and we don't have a way to distinguish the different errors. That's why I proposed the hint to the user.

But also I think we should allow users to do a solve without forcing an update. Imagine you're trying to add a new dependency to a project

Note that we do allow this. It's just not the default behavior. It requires manually setting the commit or the path to the repo.

and you inadvertently introduce a problem caused by switching to a later version of a dependency that introduces a bug. (That's a different issue from the one we're currently discussing.)

This is exactly the reason why our solver shouldn't prefer upper bounds.

[gridbugs]

Because updating the repository could fail for multiple reasons. Some of them should not have a fallback like that, and we don't have a way to distinguish the different errors. That's why I proposed the hint to the user.

What are the ways that updating the repository can fail where a fallback would be inappropriate?

[gridbugs]

and you inadvertently introduce a problem caused by switching to a later version of a dependency that introduces a bug. (That's a different issue from the one we're currently discussing.)

This is exactly the reason why our solver shouldn't prefer upper bounds.

Hang on this is only a problem in the presence of both:

• a solver which prefers upper bounds, and
• a policy of updating the opam repo on every solve

I'm with you in that I'd rather not do the former, but the reality is that opam packages are written and maintained with the assumption that the solver will prefer upper bounds as that's how opam's solver works. We can't change that in dune and still expect opam packages to work as reliably as they do in opam.

[rgrinberg]

What are the ways that updating the repository can fail where a fallback would be inappropriate?

For example, the user set an incorrect commit hash for the opam repository. Or, the user sets an invalid URL for a custom opam repository.

We can't change that in dune and still expect opam packages to work as reliably as they do in opam.

I think we can only gain reliability by preferring lower bounds. If we ever run into a package with some incorrect lower bound that picks a bad package version, then all we have to do is bump our own lower bound on that package to exclude the bad version. You can't do that with upper bounds because nobody writes them in the first place - as that is the custom with opam-repository.