dune format-dune-file: use Dune lang version of current Dune project

[nojb]

This PR extends dune format-dune-file so that the Dune version of the current Dune project is used when formatting. If a file is passed as argument, then it is the Dune project that owns the file that is used. If formatting standard input, it is the Dune project containing the current directory that is used.

This fixes the isssue discussed in the thread starting at https://github.com/ocaml/dune/pull/10892#issuecomment-2503762387.

This also opens the door to honouring the (formatting) settings of the current Dune project (cc @Khady, cf #3516).

[nojb]

I added a warning when using dune format-dune-file inside Dune, as this may not give the expected results (root detection behaves differently in that case) and the (format-dune-file) action seems a better option in that case. https://github.com/ocaml/dune/pull/11865/commits/dfc602742e849337e523d932ea7d2a79960efdf1

[rgrinberg]

Tests are failing in this one

[nojb]

Tests are failing in this one

Thanks, coming back to this PR after the holidays. The test error was due to INSIDE_DUNE being set in the tests so that any use of dune format-dune-file triggers a new warning suggesting to use (format-dune-file) instead. I just accepted the affected tests: https://github.com/ocaml/dune/pull/11865/commits/ccb86c3afdc02a5b4e83819757174bb7ef3d6fb5

[mbarbin]

Thank you @nojb for this work. I wanted to note I'll be a happy a user of this as I have started some work for fixing https://github.com/mbarbin/dunolint/issues/88 but it seems with this change there will be no further change required in dunolint.

However, I have some reservation about the warning, and that is due to the use case on itself, as discussed in #11605 I would like to have a supported way (and official!) to format dune files from OCaml. I would like to add a way to silence the warning, as a way to acknowledge that I am not doing something that will annoy the dune-devs! Thanks a lot.

[nojb]

Thank you @nojb for this work. I wanted to note I'll be a happy a user of this as I have started some work for fixing mbarbin/dunolint#88 but it seems with this change there will be no further change required in dunolint.

Would you be able to test this PR and report back if indeed it works as expected? It would be nice to have some confirmation from an actual user.

However, I have some reservation about the warning, and that is due to the use case on itself, as discussed in #11605 I would like to have a supported way (and official!) to format dune files from OCaml. I would like to add a way to silence the warning, as a way to acknowledge that I am not doing something that will annoy the dune-devs! Thanks a lot.

If I understand correctly from #11605, the issue is only the potential triggering of this warning in tests of your code, right? If so, you can disable the warning simply by unsetting the variable INSIDE_DUNE. Would that work?

[mbarbin]

Thank you. I'll attempt to find a slot for this in the afternoon and try the integration in the tests as well.

Thanks for pointing me to the env var :+1: I'll check it out.

Besides the technical consideration of making it work today, I guess I am also trying to do some due diligence that my use case aligns with your longer term vision. Seeing a warning in my use to me is a hint that something may go wrong later on, hence my feedback. Does it make sense, or am I being too cautious? Thanks a lot!

[mbarbin]

@nojb I am getting started with the testing. It's been a while since I built dune from source, I have some questions - hopefully this won't be too noisy for this threads, otherwise let me know I'm happy to PM you instead with my progress.

I'm at ccb86c3afdc02a5b4e83819757174bb7ef3d6fb5 tip of this PR as of now.

make bootstrap + make dev gets me a binary which I can use in tests. I noticed a new _build/log file generated in tests whereas before there was none. The log contains:

 |  $ cat _build/log
+|  # dune format-dune-file
+|  # OCAMLPARAM: unset
+|  # Shared cache: enabled-except-user-rules
+|  # Shared cache location: /home/mathieu/.cache/dune/db
+|  # Workspace root:
+|  # $TESTCASE_ROOT

Is the fact that a log is created a behavior change of 3.20, or perhaps an artifact of building in dev mode? To be clear, I don't mind the log on its own, I just need to understand what I can reasonable commit to my cram tests, and what dune version I should require in my dev-package.

If I try to build your PR in release mode, I get this:

$ make release
Internal error, please report upstream including the contents of _build/log.
Description:
  ("Unexpected find result", { found = Not_found; lib.name = "pp" })
Raised at Stdune__Code_error.raise in file "stdune__Code_error.ml", line 11,
  characters 30-62
Called from Fiber__Scheduler.exec in file "fiber__Scheduler.ml", line 77,
  characters 8-11
-> required by ("<unnamed>", ())
-> required by ("<unnamed>", ())
-> required by ("<unnamed>", ())
-> required by ("load-dir", In_build_dir "default/test/blackbox-tests/utils")
-> required by ("toplevel", ())

I must not crash.  Uncertainty is the mind-killer. Exceptions are the
little-death that brings total obliteration.  I will fully express my cases. 
Execution will pass over me and through me.  And when it has gone past, I
will unwind the stack along its path.  Where the cases are handled there will
be nothing.  Only I will remain.
Internal error, please report upstream including the contents of _build/log.
Description:
  ("Unexpected find result", { found = Not_found; lib.name = "csexp" })
Raised at Stdune__Code_error.raise in file "stdune__Code_error.ml", line 11,
  characters 30-62
Called from Fiber__Scheduler.exec in file "fiber__Scheduler.ml", line 77,
  characters 8-11
-> required by ("<unnamed>", ())
-> required by ("<unnamed>", ())
-> required by
   ("load-dir", In_build_dir "default/test/expect-tests/dune_rpc_e2e")
-> required by ("toplevel", ())
make: *** [Makefile:38: release] Error 1        

In a switch where I have pp=2.0.0 and csexp=1.5.2

$ cat _build/log
# ./_boot/dune.exe build @install -p dune --profile dune-bootstrap
# OCAMLPARAM: unset
# Shared cache: enabled-except-user-rules
# Shared cache location: /home/mathieu/.cache/dune/db
# Workspace root: /home/mathieu/dev/forks/dune/main
# Auto-detected concurrency: 8
# Dune context:
#  { name = "default"
#  ; kind = "default"
#  ; profile = User_defined "dune-bootstrap"
#  ; merlin = true
#  ; fdo_target_exe = None
#  ; build_dir = In_build_dir "default"
#  ; instrument_with = []
#  }
$ /home/mathieu/.opam/5.3.0/bin/ocamlc.opt -config > /tmp/dune_650f61_output

Thanks for your help!

[mbarbin]

@nojb apart from the warning and/or the _build/log question, I report that this PR fixes the formatting issue encountered with dune.3.20 when using dunolint as an Emacs reformatter.

I tested with dunolint.0.0.20250804 and a dune executable in the PATH built from your PR + 1 commit to bring back the formatting changes that were recently reverted:

:heavy_check_mark: From Emacs when saving a file enclosed in a dune-project with 3.20, the compact formatting is used. When saving a file enclosed in a dune-project, say with 3.17, the usual former format is used.

(I added the one commit on top to actually have a difference in formatting, otherwise it was hard to get a visual confirmation, but because this PR and the formatting are orthogonal, I consider this is a satisfactory test).

[nojb]

I report that this PR fixes the formatting issue encountered with dune.3.20 when using dunolint as an Emacs reformatter.

Thanks for the confirmation! I haven't had time to look at your question yet, but I will get to it soon-ish :)

[nojb]

Is the fact that a log is created a behavior change of 3.20, or perhaps an artifact of building in dev mode? To be clear, I don't mind the log on its own,

I believe it is a behaviour change of this PR in fact. In order to have dune format-dune-file detect the version of the Dune project containing the Dune file being formatted, more of the standard build infrastructure needs to be invoked, and this includes initializing the logging subsystem.

[nojb]

If I try to build your PR in release mode, I get this:

make release is pretty finicky regarding dependencies. I normally use make dev for testing. In any case, I can make it work if I use make dev-deps before make release.

[nojb]

Besides the technical consideration of making it work today, I guess I am also trying to do some due diligence that my use case aligns with your longer term vision. Seeing a warning in my use to me is a hint that something may go wrong later on, hence my feedback. Does it make sense, or am I being too cautious? Thanks a lot!

Actually, adding the warning is probably me being overly cautious that something else. Technically, Dune may not work very well if it is called recursively on the same workspace. However, special support could be added for this if/as needed (basically to have the "inner" Dune invocation emit an RPC request to the "outer one"). So perhaps the warning is premature and may not be needed in the end.

I'll remove the warning for now. Thanks for the review!

[mbarbin]

Great, thank you!

Looking at your PR now, my take is that I am satisfied with it as far as the requirements I stated in #11605 so I will close that issue once this lands. Also, this fixes https://github.com/mbarbin/dunolint/issues/88 which is nice (less immediate work to do in dunolint for me, yeah!). The timing of this PR is also on point, as we've seen the potential user-facing impact of formatting changes in dune files. In sum, I am thankful for your work!

more of the standard build infrastructure needs to be invoked, and this includes initializing the logging subsystem

I cannot quite explain why, but I feel a slight reluctance to this part. I was hoping the dune command would be comparable to, say ocamlformat FILE and not requiring to create files on the side, but I don't think this is a big deal. I'll leave that simply as a thought from an external user who doesn't know the implementation constraints of the code. Does that part happen when the --dune-version is supplied?

By the way, currently the help says:

$ dune format-dune-file --help
...
OPTIONS
       --dune-version=VERSION (absent=3.20)
           Which version of Dune language to use.

I'm not sure if the new version documentation takes the changes into account as I didn't see any new text in the diff.

[mbarbin]

By the way, I just saw @Khady express a desire to keep the formatting changes in 3.20. Because this breaks my dunolint editor workflow, my perspective is that, whatever you decide, I would like to request that you include this PR with the formatting changes whenever you release it (that being in 3.20.1 or 3.21). Thanks a lot!

[nojb]

I think all issues raised so far have been addressed, so the PR could be merged in principle. @rgrinberg: what do you think?

[nojb]

Thanks @Alizter!