merlin icon indicating copy to clipboard operation
merlin copied to clipboard
verified publisher icon ocaml

Jumping to `Core.Md5` fails with `Several source files in your path have the same name`

Open voodoos opened this issue 9 months ago • 10 comments

Initially reported by @ddickstein.

In the source file:

module M = Core.Md5

Running ocamlmerlin single locate -position 2:18 -filename bin/main.ml <bin/main.ml | jq'.value' returns:

"Several source files in your path have the same name, and merlin doesn't know which is the right one: /.../.opam/5.3.0/lib/core/md5.ml, /.../opam/5.3.0/lib/core/md5.mli, /.../.opam/5.3.0/lib/bin_prot/md5.ml, /.../.opam/5.3.0/lib/bin_prot/md5.mli"
Full log 
# 0.03 locate - reconstructed identifier
Core.Md5
# 0.03 locate - from_string
inferred context: module path
# 0.03 locate - from_string
looking for the source of 'Core.Md5' (prioritizing .mli files)
# 0.04 locate - uid_of_path
Unaliased declaration uid: Core.88 -> Core__Md5
# 0.04 locate - find_loc_of_comp_unit
Got the uid of a compilation unit: Core__Md5
# 0.04 locate - find_file_with_path
Try find "Core__Md5"
# 0.04 locate - find_file_with_path
Trying "Core__Md5.cmti"
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/tmp/repro-core/_build/default/bin/.main.eobjs/byte/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/variantslib/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/typerep/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/time_now/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/stdio/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/splittable_random/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/sexplib0/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/sexplib/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppxlib/print_diff/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_string/runtime/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_stable_witness/stable_witness/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_stable_witness/runtime/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_sexp_conv/runtime-lib/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_module_timer/runtime/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_log/types/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_log/syntax/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_inline_test/runtime-lib/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_inline_test/config/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_here/runtime-lib/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_hash/runtime-lib/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_expect/runtime/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_expect/make_corrected_file/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_expect/config_types/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_expect/config/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_enumerate/runtime-lib/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_diff/diffable_cinaps/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_diff/diffable/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_compare/runtime-lib/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_bench/runtime-lib/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ppx_assert/runtime-lib/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/parsexp/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/ocaml_intrinsics_kernel/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/jane-street-headers/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/int_repr/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/gel/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/fieldslib/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/core/validate/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/core/univ_map/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/core/heap_block/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/core/filename_base/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/core/command/Core__Md5.cmti
# 0.04 locate - find_in_path_uncap
Failed to load /Users/ulysse/.opam/5.3.0/lib/core/base_for_tests/Core__Md5.cmti
# 0.04 locate - load_cmt
Found "Core__Md5" at path "/Users/ulysse/.opam/5.3.0/lib/core/core__Md5.cmti"
# 0.04 locate - File_switching.move_to
file: /Users/ulysse/.opam/5.3.0/lib/core/core__Md5.cmti
digest: 897cbc74854e265574e70c752c09cc57
# 0.04 locate - find_source
attempt to find "core/src/md5.pp.mli"
# 0.04 locate - find_source
initial path: "/Users/ulysse/.opam/5.3.0/lib/core/core__Md5.cmti"
# 0.04 locate - find_all_in_path_uncap
Looking for file "Md5.mli" in path:
/Users/ulysse/tmp/repro-core/bin/Users/ulysse/.opam/5.3.0/lib/ocaml
/Users/ulysse/.opam/5.3.0/lib/base
/Users/ulysse/.opam/5.3.0/lib/base/base_internalhash_types
/Users/ulysse/.opam/5.3.0/lib/base/md5
/Users/ulysse/.opam/5.3.0/lib/base/shadow_stdlib
/Users/ulysse/.opam/5.3.0/lib/base_bigstring
/Users/ulysse/.opam/5.3.0/lib/base_quickcheck
/Users/ulysse/.opam/5.3.0/lib/base_quickcheck/ppx_quickcheck/runtime
/Users/ulysse/.opam/5.3.0/lib/bin_prot
/Users/ulysse/.opam/5.3.0/lib/bin_prot/shape
/Users/ulysse/.opam/5.3.0/lib/core
/Users/ulysse/.opam/5.3.0/lib/core/base_for_tests
/Users/ulysse/.opam/5.3.0/lib/core/command
/Users/ulysse/.opam/5.3.0/lib/core/filename_base
/Users/ulysse/.opam/5.3.0/lib/core/heap_block
/Users/ulysse/.opam/5.3.0/lib/core/univ_map
/Users/ulysse/.opam/5.3.0/lib/core/validate
/Users/ulysse/.opam/5.3.0/lib/fieldslib/Users/ulysse/.opam/5.3.0/lib/gel
/Users/ulysse/.opam/5.3.0/lib/int_repr
/Users/ulysse/.opam/5.3.0/lib/jane-street-headers
/Users/ulysse/.opam/5.3.0/lib/ocaml_intrinsics_kernel
/Users/ulysse/.opam/5.3.0/lib/parsexp
/Users/ulysse/.opam/5.3.0/lib/ppx_assert/runtime-lib
/Users/ulysse/.opam/5.3.0/lib/ppx_bench/runtime-lib
/Users/ulysse/.opam/5.3.0/lib/ppx_compare/runtime-lib
/Users/ulysse/.opam/5.3.0/lib/ppx_diff/diffable
/Users/ulysse/.opam/5.3.0/lib/ppx_diff/diffable_cinaps
/Users/ulysse/.opam/5.3.0/lib/ppx_enumerate/runtime-lib
/Users/ulysse/.opam/5.3.0/lib/ppx_expect/config
/Users/ulysse/.opam/5.3.0/lib/ppx_expect/config_types
/Users/ulysse/.opam/5.3.0/lib/ppx_expect/make_corrected_file
/Users/ulysse/.opam/5.3.0/lib/ppx_expect/runtime
/Users/ulysse/.opam/5.3.0/lib/ppx_hash/runtime-lib
/Users/ulysse/.opam/5.3.0/lib/ppx_here/runtime-lib
/Users/ulysse/.opam/5.3.0/lib/ppx_inline_test/config
/Users/ulysse/.opam/5.3.0/lib/ppx_inline_test/runtime-lib
/Users/ulysse/.opam/5.3.0/lib/ppx_log/syntax
/Users/ulysse/.opam/5.3.0/lib/ppx_log/types
/Users/ulysse/.opam/5.3.0/lib/ppx_module_timer/runtime
/Users/ulysse/.opam/5.3.0/lib/ppx_sexp_conv/runtime-lib
/Users/ulysse/.opam/5.3.0/lib/ppx_stable_witness/runtime
/Users/ulysse/.opam/5.3.0/lib/ppx_stable_witness/stable_witness
/Users/ulysse/.opam/5.3.0/lib/ppx_string/runtime
/Users/ulysse/.opam/5.3.0/lib/ppxlib/print_diff
/Users/ulysse/.opam/5.3.0/lib/sexplib/Users/ulysse/.opam/5.3.0/lib/sexplib0
/Users/ulysse/.opam/5.3.0/lib/splittable_random
/Users/ulysse/.opam/5.3.0/lib/stdio/Users/ulysse/.opam/5.3.0/lib/time_now
/Users/ulysse/.opam/5.3.0/lib/typerep
/Users/ulysse/.opam/5.3.0/lib/variantslib
# 0.04 locate - find_all_in_path_uncap
Looking for file "Md5.rei" in path:
/Users/ulysse/tmp/repro-core/bin/Users/ulysse/.opam/5.3.0/lib/ocaml
/Users/ulysse/.opam/5.3.0/lib/base
/Users/ulysse/.opam/5.3.0/lib/base/base_internalhash_types
/Users/ulysse/.opam/5.3.0/lib/base/md5
/Users/ulysse/.opam/5.3.0/lib/base/shadow_stdlib
/Users/ulysse/.opam/5.3.0/lib/base_bigstring
/Users/ulysse/.opam/5.3.0/lib/base_quickcheck
/Users/ulysse/.opam/5.3.0/lib/base_quickcheck/ppx_quickcheck/runtime
/Users/ulysse/.opam/5.3.0/lib/bin_prot
/Users/ulysse/.opam/5.3.0/lib/bin_prot/shape
/Users/ulysse/.opam/5.3.0/lib/core
/Users/ulysse/.opam/5.3.0/lib/core/base_for_tests
/Users/ulysse/.opam/5.3.0/lib/core/command
/Users/ulysse/.opam/5.3.0/lib/core/filename_base
/Users/ulysse/.opam/5.3.0/lib/core/heap_block
/Users/ulysse/.opam/5.3.0/lib/core/univ_map
/Users/ulysse/.opam/5.3.0/lib/core/validate
/Users/ulysse/.opam/5.3.0/lib/fieldslib/Users/ulysse/.opam/5.3.0/lib/gel
/Users/ulysse/.opam/5.3.0/lib/int_repr
/Users/ulysse/.opam/5.3.0/lib/jane-street-headers
/Users/ulysse/.opam/5.3.0/lib/ocaml_intrinsics_kernel
/Users/ulysse/.opam/5.3.0/lib/parsexp
/Users/ulysse/.opam/5.3.0/lib/ppx_assert/runtime-lib
/Users/ulysse/.opam/5.3.0/lib/ppx_bench/runtime-lib
/Users/ulysse/.opam/5.3.0/lib/ppx_compare/runtime-lib
/Users/ulysse/.opam/5.3.0/lib/ppx_diff/diffable
/Users/ulysse/.opam/5.3.0/lib/ppx_diff/diffable_cinaps
/Users/ulysse/.opam/5.3.0/lib/ppx_enumerate/runtime-lib
/Users/ulysse/.opam/5.3.0/lib/ppx_expect/config
/Users/ulysse/.opam/5.3.0/lib/ppx_expect/config_types
/Users/ulysse/.opam/5.3.0/lib/ppx_expect/make_corrected_file
/Users/ulysse/.opam/5.3.0/lib/ppx_expect/runtime
/Users/ulysse/.opam/5.3.0/lib/ppx_hash/runtime-lib
/Users/ulysse/.opam/5.3.0/lib/ppx_here/runtime-lib
/Users/ulysse/.opam/5.3.0/lib/ppx_inline_test/config
/Users/ulysse/.opam/5.3.0/lib/ppx_inline_test/runtime-lib
/Users/ulysse/.opam/5.3.0/lib/ppx_log/syntax
/Users/ulysse/.opam/5.3.0/lib/ppx_log/types
/Users/ulysse/.opam/5.3.0/lib/ppx_module_timer/runtime
/Users/ulysse/.opam/5.3.0/lib/ppx_sexp_conv/runtime-lib
/Users/ulysse/.opam/5.3.0/lib/ppx_stable_witness/runtime
/Users/ulysse/.opam/5.3.0/lib/ppx_stable_witness/stable_witness
/Users/ulysse/.opam/5.3.0/lib/ppx_string/runtime
/Users/ulysse/.opam/5.3.0/lib/ppxlib/print_diff
/Users/ulysse/.opam/5.3.0/lib/sexplib/Users/ulysse/.opam/5.3.0/lib/sexplib0
/Users/ulysse/.opam/5.3.0/lib/splittable_random
/Users/ulysse/.opam/5.3.0/lib/stdio/Users/ulysse/.opam/5.3.0/lib/time_now
/Users/ulysse/.opam/5.3.0/lib/typerep
/Users/ulysse/.opam/5.3.0/lib/variantslib
# 0.04 locate - find_source(Md5)
multiple matches in the source path : /Users/ulysse/.opam/5.3.0/lib/core/md5.ml , /Users/ulysse/.opam/5.3.0/lib/core/md5.mli , /Users/ulysse/.opam/5.3.0/lib/bin_prot/md5.ml , /Users/ulysse/.opam/5.3.0/lib/bin_prot/md5.mli
# 0.04 locate - find_source
... trying to use source digest to find the right one
# 0.04 locate - find_source
Source digest: 897cbc74854e265574e70c752c09cc57
# 0.04 locate - find_source
  /Users/ulysse/.opam/5.3.0/lib/core/md5.ml (5c07d21953176a8bb5701f0361525a6b)
# 0.04 locate - find_source
  /Users/ulysse/.opam/5.3.0/lib/core/md5.mli (e400638f83e6028dfd761ba11c9373f7)
# 0.04 locate - find_source
  /Users/ulysse/.opam/5.3.0/lib/bin_prot/md5.ml (83bb88d9053d76a01ca15ae73e8588c6)
# 0.04 locate - find_source
  /Users/ulysse/.opam/5.3.0/lib/bin_prot/md5.mli (4073e45e18be5ef6d3f65954a6d8a912)
# 0.04 locate - find_source
... found no files with matching digest
# 0.04 locate - find_source
... using heuristic to select the right one
# 0.04 locate - find_source
we are looking for a file named core/src/md5.pp.mli in /Users/ulysse/.opam/5.3.0/lib/core

voodoos avatar May 15 '25 09:05 voodoos

I think this is due to the fact that the files are preprocessed. The cmt for core__Md5 actually refers to core/src/md5.pp.ml and it's digest. Since this file is not part of the installed files Merlin cannot find it and has no way to infer the correct one.

voodoos avatar May 15 '25 09:05 voodoos

Archeology moment:

This issue was already raised in: https://github.com/ocaml/merlin/issues/894

And @ddickstein proposed a fix in https://github.com/ocaml/merlin/pull/1219

It looks like Merlin has become stricter in matching digests since then, which broke the fix in #1219 ?

edit: or more probably merlin fails to find the "exact match" to make a digest here: https://github.com/ocaml/merlin/blob/main/src/analysis/locate.ml#L319

voodoos avatar May 15 '25 09:05 voodoos

Same issue as #1821

voodoos avatar May 20 '25 12:05 voodoos

I think I found the issue: we rely on the build directory stored in the cmt file to look for the exact path of an original .ml file when we are looking for its corresponding .pp.ml file. But today this information is non-sensical: When in a test it litterally contains $TESTCASE_ROOT/_build/default and when in a normal project it literrally contains /workspace_root. These obviously do not refer to existing paths and this prevent the heuristic proposed in #1219 from working.

This is due to the fact that OCaml honors the BUILD_PATH_PREFIX_MAP https://github.com/ocaml/ocaml/pull/1515. Dune use it to sanitize paths, this is enabled since Dune 3.0.

https://dune.readthedocs.io/en/stable/reference/dune-project/map_workspace_root.html

I am not sure yet what is the best way to work around that, especially for installed artifacts.

voodoos avatar May 21 '25 14:05 voodoos

I was going to suggest, for monorepos a workaround using (map_workspace_root false) in the dune-project file, but that setting doesn't impact vendored directories...

An option might be to add the build root as a configuration directive sent by dune to Merlin. An other one is to do some parsing based on the cmt paths, but that's not very satisfying.

voodoos avatar May 21 '25 15:05 voodoos

If I understand this issue correctly, I think the heuristic from https://github.com/ocaml/merlin/pull/1882 would help here. That heuristic only kicks in when there are multiple files with a matching digest. But maybe it should also kick in if no files have a matching digest?

Concretely, I'm proposing replacing the heuristic on line 444 with the score_file heuristic.

(I have not tested whether this actually works for this case.)

If you agree this would be a good idea, I'm happy to open a pr to do so.

liam923 avatar Jun 16 '25 02:06 liam923

An option might be to add the build root as a configuration directive sent by dune to Merlin. An other one is to do some parsing based on the cmt paths, but that's not very satisfying.

I quite like this approach and perhaps it could be generalized for merlin to be able to reverse all the mangling done with BUILD_PATH_PREFIX_MAP. At least the alternative is far worse: (map_workspace_root false) makes the dune cache completely useless. Note that ocamldebug has the same problem too.

rgrinberg avatar Jun 16 '25 23:06 rgrinberg

Internally at Jane Street, we're planning on changing the build rules so that the compiler records the relative path to the source file from the project root. And then Merlin can use the SOURCE_ROOT directive (which dune already includes for us internally) to reconstruct the full path.

liam923 avatar Jun 17 '25 01:06 liam923

perhaps it could be generalized for merlin to be able to reverse all the mangling done with BUILD_PATH_PREFIX_MAP. At least the alternative is far worse: (map_workspace_root false) makes the dune cache completely useless. Note that ocamldebug has the same problem too.

I agree that this sounds like a sane fix for that issue.

Meanwhile I proposed a workaround (in e4cfc069b4f89cc24a69fc3e822f190d8ee29880) that simply re-roots the _build folder to the SOURCE_ROOT when possible. It should work in much more cases than the current version (which never does).

@liam923 I am not sure your proposition would fix the issue here, but I happy to try a POC if you have one :-)

voodoos avatar Jun 23 '25 14:06 voodoos

@liam923 I am not sure your proposition would fix the issue here, but I happy to try a POC if you have one :-)

Yes I agree, I've realized that I had misunderstood the issue.

liam923 avatar Jun 23 '25 15:06 liam923