dune pkg lock: cross device link error when refreshing package locks

[avsm]

I'm trying dune package management again and just ran into this error when refreshing a pkg lock in a Dockerfile.

#18 1.525 - zarith.1.14
#18 1.531 Error: rename(dune.lock): Cross-device link
#18 ERROR: process "/bin/sh -c dune pkg lock" did not complete successfully: exit code: 1
------
 > [12/13] RUN dune pkg lock:

This only happens when the lock file already exists, and is presumably because it's trying to rename across a bindmount. The tempfile should probably be in the same directory as the lockfile itself.

The Dockerfile is something like:

# syntax=docker/dockerfile:1.2
FROM ocaml/opam:alpine AS build
RUN curl -fsSL https://get.dune.build/install | sh
WORKDIR /home/ocaml/src
RUN --mount=type=secret,id=GIT_TOKEN git clone <snip...>
ENV PATH=/home/opam/.local/bin:$PATH
RUN sudo apk add libev-dev gmp-dev openssl-dev
RUN ~/.local/bin/dune pkg lock
RUN ~/.local/bin/dune build @pkg-install
RUN git pull
RUN dune pkg lock
RUN dune build

The failure happens on the second lock.

[maiste]

Thanks for reporting! 😄 We will investigate on the problem. Indeed, it might be because we are trying to compute the new lock file inside a /tmp/dune-XXX directory before replacing the old one with the new one.

[maiste]

I thought it would be because of https://github.com/ocaml/dune/pull/10852. However, looking at the code and re experimenting with it, everything happens in the same directory (the one of the project).

To be more explicit, when you run dune pkg lock, it will:

• Rename if it exists the dune.lock to .dune.pkg
• Create a temporary directory at the root and build everything in it
• Rename the temp directory into dune.lock
• Delete the old .dune.lock

By any chance, do you have access to a more complete reproduction case with Docker? It would help to try and reproduce locally because I'm not able to.

[mtelvers]

Here's a minimal example showing the failure:

# syntax=docker/dockerfile:1.2
FROM ocaml/opam:alpine AS build
RUN curl -fsSL https://get.dune.build/install | sh
WORKDIR /home/opam
RUN git clone https://github.com/ocurrent/ocaml-dockerfile
ENV PATH=/home/opam/.local/bin:$PATH
RUN sudo apk add libev-dev gmp-dev openssl-dev
WORKDIR /home/opam/ocaml-dockerfile
RUN dune pkg lock
RUN dune build @pkg-install
RUN git pull
RUN dune pkg lock
RUN dune build

Here's the log from Docker

$ docker build --progress plain --no-cache .
#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 502B done
#1 DONE 0.0s

#2 [auth] docker/dockerfile:pull token for registry-1.docker.io
#2 DONE 0.0s

#3 resolve image config for docker-image://docker.io/docker/dockerfile:1.2
#3 DONE 1.0s

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 502B done
#1 DONE 0.0s

#4 docker-image://docker.io/docker/dockerfile:1.2@sha256:e2a8561e419ab1ba6b2fe6cbdf49fd92b95912df1cf7d313c3e2230a333fdbcc
#4 CACHED

#5 [internal] load build definition from Dockerfile
#5 DONE 0.0s

#6 [internal] load .dockerignore
#6 transferring context: 2B done
#6 DONE 0.0s

#7 [auth] ocaml/opam:pull token for registry-1.docker.io
#7 DONE 0.0s

#8 [internal] load metadata for docker.io/ocaml/opam:alpine
#8 DONE 0.4s

#9 [ 1/11] FROM docker.io/ocaml/opam:alpine@sha256:7733fde49f79a37ff8aa68a51656bebf6bbc4fbef91220ebfc452aa1de4ea9ea
#9 CACHED

#10 [ 2/11] RUN curl -fsSL https://get.dune.build/install | sh
######################################################################## 100.0%
#10 0.744 dune x86_64-unknown-linux-musl was installed successfully to ~/.local/bin/dune 
#10 0.745 
######################################################################## 100.0%
#10 8.784 Revision cache was populated successfully 
#10 8.790 To use dune you will need to source the file "$HOME/.local/share/dune/env/env.bash" (or similar as appropriate for your shell)
#10 8.790   export PATH="/home/opam/.local/bin:$PATH" 
#10 8.790 
#10 8.790 To get started, run: 
#10 8.790   dune --help 
#10 DONE 9.6s

#11 [ 3/11] WORKDIR /home/opam
#11 DONE 0.6s

#12 [ 4/11] RUN git clone https://github.com/ocurrent/ocaml-dockerfile
#12 0.180 Cloning into 'ocaml-dockerfile'...
#12 DONE 1.0s

#13 [ 5/11] RUN sudo apk add libev-dev gmp-dev openssl-dev
#13 0.200 fetch https://dl-cdn.alpinelinux.org/alpine/v3.21/main/x86_64/APKINDEX.tar.gz
#13 0.356 fetch https://dl-cdn.alpinelinux.org/alpine/v3.21/community/x86_64/APKINDEX.tar.gz
#13 0.642 fetch https://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz
#13 0.720 fetch https://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
#13 0.988 fetch https://dl-cdn.alpinelinux.org/alpine/edge/testing/x86_64/APKINDEX.tar.gz
#13 1.372 (1/5) Installing libgmpxx (6.3.0-r2)
#13 1.401 (2/5) Installing gmp-dev (6.3.0-r2)
#13 1.454 (3/5) Installing libev (4.33-r1)
#13 1.461 (4/5) Installing libev-dev (4.33-r1)
#13 1.468 (5/5) Installing openssl-dev (3.3.4-r0)
#13 1.503 OK: 312 MiB in 107 packages
#13 DONE 2.8s

#14 [ 6/11] WORKDIR /home/opam/ocaml-dockerfile
#14 DONE 1.4s

#15 [ 7/11] RUN dune pkg lock
#15 22.05 Solution for dune.lock:
#15 22.05 - alcotest.1.9.0
#15 22.05 - arch-x86_64.1
#15 22.05 - astring.0.8.5
#15 22.05 - base.v0.17.3
#15 22.05 - base-threads.base
#15 22.05 - base-unix.base
#15 22.05 - bos.0.2.1
#15 22.05 - cmdliner.1.3.0
#15 22.05 - conf-mingw-w64-gcc-i686.1
#15 22.05 - conf-mingw-w64-gcc-x86_64.1
#15 22.05 - csexp.1.5.2
#15 22.05 - dune-configurator.3.19.1
#15 22.05 - flexdll.0.44
#15 22.05 - fmt.0.10.0
#15 22.05 - fpath.0.7.3
#15 22.05 - logs.0.9.0
#15 22.05 - num.1.6
#15 22.05 - ocaml.5.3.0
#15 22.05 - ocaml-base-compiler.5.3.0
#15 22.05 - ocaml-compiler.5.3.0
#15 22.05 - ocaml-compiler-libs.v0.17.0
#15 22.05 - ocaml-config.3
#15 22.05 - ocaml-env-mingw32.1
#15 22.05 - ocaml-env-mingw64.1
#15 22.05 - ocaml-syntax-shims.1.0.0
#15 22.05 - ocaml-version.4.0.1
#15 22.05 - ocaml_intrinsics_kernel.v0.17.1
#15 22.05 - ocamlbuild.0.16.1+dune
#15 22.05 - ocamlfind.1.9.8+dune
#15 22.05 - parsexp.v0.17.0
#15 22.05 - ppx_derivers.1.2.1
#15 22.05 - ppx_sexp_conv.v0.17.1
#15 22.05 - ppxlib.0.36.0
#15 22.05 - ppxlib_jane.v0.17.4
#15 22.05 - re.1.13.2
#15 22.05 - rresult.0.7.0
#15 22.05 - seq.base
#15 22.05 - sexplib.v0.17.0
#15 22.05 - sexplib0.v0.17.0
#15 22.05 - stdlib-shims.0.3.0
#15 22.05 - system-mingw.1
#15 22.05 - topkg.1.0.8
#15 22.05 - uutf.1.0.4
#15 DONE 22.4s

#16 [ 8/11] RUN dune build @pkg-install
#16 3.188  Downloading ocaml-compiler.5.3.0
#16 5.806     Building ocaml-compiler.5.3.0
#16 139.2     Building ocaml-base-compiler.5.3.0
#16 139.4     Building ocaml-config.3
#16 139.4     Building ocaml.5.3.0
#16 140.0  Downloading num.1.6
#16 140.1     Building num.1.6
#16 140.7  Downloading sexplib0.v0.17.0
#16 140.8     Building sexplib0.v0.17.0
#16 141.8  Downloading parsexp.v0.17.0
#16 141.9     Building parsexp.v0.17.0
#16 143.7  Downloading sexplib.v0.17.0
#16 143.8     Building sexplib.v0.17.0
#16 145.0     Building base-unix.base
#16 145.3  Downloading csexp.1.5.2
#16 145.3     Building csexp.1.5.2
#16 145.3  Downloading ocaml-compiler-libs.v0.17.0
#16 145.4     Building ocaml-compiler-libs.v0.17.0
#16 145.4  Downloading stdlib-shims.0.3.0
#16 145.4     Building stdlib-shims.0.3.0
#16 145.4  Downloading ocaml_intrinsics_kernel.v0.17.1
#16 145.4  Downloading ppx_derivers.1.2.1
#16 145.4     Building ocaml_intrinsics_kernel.v0.17.1
#16 145.4     Building ppx_derivers.1.2.1
#16 147.9  Downloading dune-configurator.3.19.1
#16 149.8  Downloading ppxlib.0.36.0
#16 150.0     Building ppxlib.0.36.0
#16 150.2     Building dune-configurator.3.19.1
#16 152.8  Downloading base.v0.17.3
#16 153.0     Building base.v0.17.3
#16 165.4  Downloading ppxlib_jane.v0.17.4
#16 165.6     Building ppxlib_jane.v0.17.4
#16 167.3  Downloading ppx_sexp_conv.v0.17.1
#16 167.7     Building ppx_sexp_conv.v0.17.1
#16 171.0  Downloading ocaml-version.4.0.1
#16 171.0     Building ocaml-version.4.0.1
#16 171.4     Building base-threads.base
#16 171.7  Downloading cmdliner.1.3.0
#16 171.7     Building cmdliner.1.3.0
#16 172.1  Downloading ocamlbuild.0.16.1+dune
#16 172.2     Building ocamlbuild.0.16.1+dune
#16 172.4  Downloading ocamlfind.1.9.8+dune
#16 172.4     Building ocamlfind.1.9.8+dune
#16 178.5  Downloading topkg.1.0.8
#16 178.5     Building topkg.1.0.8
#16 183.5  Downloading rresult.0.7.0
#16 183.6  Downloading fmt.0.10.0
#16 183.6     Building rresult.0.7.0
#16 183.6  Downloading astring.0.8.5
#16 183.6     Building fmt.0.10.0
#16 183.7     Building astring.0.8.5
#16 185.3  Downloading logs.0.9.0
#16 185.4     Building logs.0.9.0
#16 185.7  Downloading fpath.0.7.3
#16 185.7     Building fpath.0.7.3
#16 187.0  Downloading bos.0.2.1
#16 187.1     Building bos.0.2.1
#16 190.5     Building seq.base
#16 190.6  Downloading uutf.1.0.4
#16 190.7     Building uutf.1.0.4
#16 190.7  Downloading ocaml-syntax-shims.1.0.0
#16 190.7     Building ocaml-syntax-shims.1.0.0
#16 191.0  Downloading re.1.13.2
#16 191.0     Building re.1.13.2
#16 192.7  Downloading alcotest.1.9.0
#16 192.9     Building alcotest.1.9.0
#16 DONE 194.5s

#17 [ 9/11] RUN git pull
#17 0.522 Already up to date.
#17 DONE 0.5s

#18 [10/11] RUN dune pkg lock
#18 2.819 Solution for dune.lock:
#18 2.819 - alcotest.1.9.0
#18 2.819 - arch-x86_64.1
#18 2.819 - astring.0.8.5
#18 2.819 - base.v0.17.3
#18 2.819 - base-threads.base
#18 2.819 - base-unix.base
#18 2.819 - bos.0.2.1
#18 2.819 - cmdliner.1.3.0
#18 2.819 - conf-mingw-w64-gcc-i686.1
#18 2.819 - conf-mingw-w64-gcc-x86_64.1
#18 2.819 - csexp.1.5.2
#18 2.819 - dune-configurator.3.19.1
#18 2.819 - flexdll.0.44
#18 2.819 - fmt.0.10.0
#18 2.819 - fpath.0.7.3
#18 2.819 - logs.0.9.0
#18 2.819 - num.1.6
#18 2.819 - ocaml.5.3.0
#18 2.819 - ocaml-base-compiler.5.3.0
#18 2.819 - ocaml-compiler.5.3.0
#18 2.819 - ocaml-compiler-libs.v0.17.0
#18 2.819 - ocaml-config.3
#18 2.819 - ocaml-env-mingw32.1
#18 2.819 - ocaml-env-mingw64.1
#18 2.819 - ocaml-syntax-shims.1.0.0
#18 2.819 - ocaml-version.4.0.1
#18 2.819 - ocaml_intrinsics_kernel.v0.17.1
#18 2.819 - ocamlbuild.0.16.1+dune
#18 2.819 - ocamlfind.1.9.8+dune
#18 2.819 - parsexp.v0.17.0
#18 2.819 - ppx_derivers.1.2.1
#18 2.819 - ppx_sexp_conv.v0.17.1
#18 2.819 - ppxlib.0.36.0
#18 2.819 - ppxlib_jane.v0.17.4
#18 2.819 - re.1.13.2
#18 2.819 - rresult.0.7.0
#18 2.819 - seq.base
#18 2.819 - sexplib.v0.17.0
#18 2.819 - sexplib0.v0.17.0
#18 2.819 - stdlib-shims.0.3.0
#18 2.819 - system-mingw.1
#18 2.819 - topkg.1.0.8
#18 2.819 - uutf.1.0.4
#18 2.825 Error: rename(dune.lock): Cross-device link
#18 ERROR: process "/bin/sh -c dune pkg lock" did not complete successfully: exit code: 1
------
 > [10/11] RUN dune pkg lock:
2.819 - re.1.13.2
2.819 - rresult.0.7.0
2.819 - seq.base
2.819 - sexplib.v0.17.0
2.819 - sexplib0.v0.17.0
2.819 - stdlib-shims.0.3.0
2.819 - system-mingw.1
2.819 - topkg.1.0.8
2.819 - uutf.1.0.4
2.825 Error: rename(dune.lock): Cross-device link
------
Dockerfile:12
--------------------
  10 |     RUN dune build @pkg-install
  11 |     RUN git pull
  12 | >>> RUN dune pkg lock
  13 |     RUN dune build
--------------------
ERROR: failed to solve: process "/bin/sh -c dune pkg lock" did not complete successfully: exit code: 1

[gridbugs]

I thought it would be because of #10852. However, looking at the code and re experimenting with it, everything happens in the same directory (the one of the project).

@maiste I think you were on the right track with this hunch after all. Even though when renaming dune.lock to .dune.lock the source and destination are in the same directory, due to how some docker storage drivers work this does not necessarily mean they are on the same (logical) volume, as different volumes can be merged into a single directory hierarchy with a union filesystem.

More to the point, the overlayfs storage driver docs state:

OverlayFS does not fully support the rename(2) system call. Your application needs to detect its failure and fall back to a "copy and unlink" strategy.

Dune performs two renames while relocking.

1. dune.lock -> .dune.lock to get it out of the way of the new lockdir
2. dune.tmp.xxxx.lock -> dune.lock (after generating a new lockdir in the former)

This is done to avoid interfering with the watch mode server. IIRC recursively deleting dune.lock and creating a new lock directory in its place was found to create a lot of file system events which caused problems with watch mode. @maiste I don't understand why deleting .dune.lock and creating the new lockdir in a temporary directory (in the same project directory) avoided the problem with watch mode. Do you happen to remember why that works?

I'm able to reproduce the issue by running dune pkg lock twice in two separate RUN lines of a dockerfile and running docker build. Changing dune to just delete the old lockdir and create a new one in its place fixed the problem. However issues with watch mode are notoriously hard to catch with tests so I'm conscious that the simple solution may introduce problems with watch mode. Manually testing my change with watch mode I didn't notice any problems. @maiste do you remember the problems with watch mode that led us to update the lockdir atomically in #10852?

[maiste]

From what I found in my notes, the reason behind was to avoid dune to restart the watch mode on a partial set of dependencies. It was mostly done because it was a lock dir and not a lock file. The "worst case scenario" it was supposed to solve was:

1. You run dune build --watch.
2. You change the dependency set in dune-project.
3. It prints the error "Out-of-sync dependencies".
4. You run dune pkg lock while the watch mode is still running (which you are allowed to).
5. The watch mode picks the change, but the files are not all written yet.
6. It changes again, and you end up in a state where the watch mode is lost because dune-project is not different, but the file system is.

To be honest, I don't know today how much this scenario is still accurate. I would imagine having the lock directory inside _build removes this problem.

[Alizter]

This problem should entirely go away when lock directories become actual targets (https://github.com/ocaml/dune/issues/12098). Then doing dune pkg lock will be just another build target which will work in tandem with watch mode.