[macOS] `opam` 2.1.5 fails to build: `sed: opam.install: No such file or directory` `make: *** [processed-opam.install] Error 1`

[barracuda156]

Unable to build opam 2.1.5 on macOS:

make[1]: Leaving directory `/opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_ocaml_opam/opam/work/opam-full-2.1.5/src_ext'
dune build --profile=release --root .   -- opam-installer.install opam.install
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
File "src/core/opamStubsTypes.ml", line 1:
Warning 70 [missing-mli]: Cannot find interface file.
File "src/tools/opam_installer.ml", line 1:
Warning 70 [missing-mli]: Cannot find interface file.
File "src/solver/opamCudfSolverSig.ml", line 1:
Warning 70 [missing-mli]: Cannot find interface file.
dune build --profile=release --root .   -- opam-installer.install opam.install
sed -f process.sed opam.install > processed-opam.install
sed: opam.install: No such file or directory
make: *** [processed-opam.install] Error 1
make: Leaving directory `/opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_ocaml_opam/opam/work/opam-full-2.1.5'
Command failed:  cd "/opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_ocaml_opam/opam/work/opam-full-2.1.5" && /usr/bin/make -j1 -w lib-ext all 
Exit code: 2

Tested on:

macOS 14.2.1 / arm64 / Apple clang 15 macOS 10.6 / ppc / gcc 13.2.0

Versions used: ocaml 4.14.1 dune 3.12.1

See also: https://trac.macports.org/ticket/68586

[kit-ty-kate]

what does ocamlc -config say? How did you build opam? If you used the vendored ocaml compiler (using make cold or make compiler), what does ./bootstrap/ocaml/bin/ocamlc -config say?

[barracuda156]

@kit-ty-kate This is on a PowerPC machine:

36-29% /opt/local/bin/ocamlc -config
version: 4.14.1
standard_library_default: /opt/local/lib/ocaml
standard_library: /opt/local/lib/ocaml
ccomp_type: cc
c_compiler: /opt/local/bin/gcc-mp-13 -arch ppc
ocamlc_cflags: -O2 -fno-strict-aliasing -fwrapv -pthread  -pipe -Os -arch ppc
ocamlc_cppflags: -D_FILE_OFFSET_BITS=64 -I/opt/local/include
ocamlopt_cflags: -O2 -fno-strict-aliasing -fwrapv -pthread  -pipe -Os -arch ppc
ocamlopt_cppflags: -D_FILE_OFFSET_BITS=64 -I/opt/local/include
bytecomp_c_compiler: /opt/local/bin/gcc-mp-13 -arch ppc -O2 -fno-strict-aliasing -fwrapv -pthread  -pipe -Os -arch ppc -D_FILE_OFFSET_BITS=64 -I/opt/local/include
native_c_compiler: /opt/local/bin/gcc-mp-13 -arch ppc -O2 -fno-strict-aliasing -fwrapv -pthread  -pipe -Os -arch ppc -D_FILE_OFFSET_BITS=64 -I/opt/local/include
bytecomp_c_libraries: -lm  -lpthread
native_c_libraries: -lm 
native_pack_linker: /opt/local/bin/gcc-mp-13 -arch ppc -Wl,-r -o 
architecture: power
model: ppc
int_size: 31
word_size: 32
system: rhapsody
asm: /opt/local/bin/gcc-mp-13 -arch ppc -Wno-trigraphs -c
asm_cfi_supported: false
with_frame_pointers: false
ext_exe: 
ext_obj: .o
ext_asm: .s
ext_lib: .a
ext_dll: .so
os_type: Unix
default_executable_name: a.out
systhread_supported: true
host: powerpc-apple-darwin10.0.0d2
target: powerpc-apple-darwin10.0.0d2
flambda: false
safe_string: true
default_safe_string: true
flat_float_array: true
function_sections: false
afl_instrument: false
windows_unicode: false
supports_shared_libraries: true
naked_pointers: true
exec_magic_number: Caml1999X031
cmi_magic_number: Caml1999I031
cmo_magic_number: Caml1999O031
cma_magic_number: Caml1999A031
cmx_magic_number: Caml1999Y031
cmxa_magic_number: Caml1999Z031
ast_impl_magic_number: Caml1999M031
ast_intf_magic_number: Caml1999N031
cmxs_magic_number: Caml1999D031
cmt_magic_number: Caml1999T031
linear_magic_number: Caml1999L031

This is on aarch64 with Sonoma:

svacchanda@43-202 ~ % /opt/local/bin/ocamlc -config
version: 4.14.1
standard_library_default: /opt/local/lib/ocaml
standard_library: /opt/local/lib/ocaml
ccomp_type: cc
c_compiler: /usr/bin/clang -arch arm64
ocamlc_cflags: -O2 -fno-strict-aliasing -fwrapv -pthread  -pipe -Os -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk -arch arm64
ocamlc_cppflags: -D_FILE_OFFSET_BITS=64 -I/opt/local/include -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk
ocamlopt_cflags: -O2 -fno-strict-aliasing -fwrapv -pthread  -pipe -Os -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk -arch arm64
ocamlopt_cppflags: -D_FILE_OFFSET_BITS=64 -I/opt/local/include -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk
bytecomp_c_compiler: /usr/bin/clang -arch arm64 -O2 -fno-strict-aliasing -fwrapv -pthread  -pipe -Os -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk -arch arm64 -D_FILE_OFFSET_BITS=64 -I/opt/local/include -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk
native_c_compiler: /usr/bin/clang -arch arm64 -O2 -fno-strict-aliasing -fwrapv -pthread  -pipe -Os -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk -arch arm64 -D_FILE_OFFSET_BITS=64 -I/opt/local/include -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk
bytecomp_c_libraries: -lm  -lpthread
native_c_libraries: -lm 
native_pack_linker: /usr/bin/clang -arch arm64 -Wl,-r -o 
architecture: arm64
model: default
int_size: 63
word_size: 64
system: macosx
asm: /usr/bin/clang -arch arm64 -Wno-trigraphs -c
asm_cfi_supported: true
with_frame_pointers: false
ext_exe: 
ext_obj: .o
ext_asm: .s
ext_lib: .a
ext_dll: .so
os_type: Unix
default_executable_name: a.out
systhread_supported: true
host: aarch64-apple-darwin23.0.0
target: aarch64-apple-darwin23.0.0
flambda: false
safe_string: true
default_safe_string: true
flat_float_array: true
function_sections: false
afl_instrument: false
windows_unicode: false
supports_shared_libraries: true
naked_pointers: true
exec_magic_number: Caml1999X031
cmi_magic_number: Caml1999I031
cmo_magic_number: Caml1999O031
cma_magic_number: Caml1999A031
cmx_magic_number: Caml1999Y031
cmxa_magic_number: Caml1999Z031
ast_impl_magic_number: Caml1999M031
ast_intf_magic_number: Caml1999N031
cmxs_magic_number: Caml1999D031
cmt_magic_number: Caml1999T031
linear_magic_number: Caml1999L031

I build opam via Macports. (The build on PowerPC needs two fixes which are not yet in Macports, but since we have the identical failure on the latest macOS, this is irrelevant here.)

[kit-ty-kate]

on PPC, the error comes system: rhapsody. OCaml should’ve detected system: macosx but somehow there was a problem in the OCaml configure script. So this is an issue in the OCaml compiler, not opam. You can open a ticket at https://github.com/ocaml/ocaml/issues

Do you also get the same error on macOS/arm64 ? It works just fine for me. I’m not seeing any logs refering to arm64 in the link above

[barracuda156]

@kit-ty-kate rhapsody is the correct value for macOS on PowerPC. It comes from some ancient times due to https://en.wikipedia.org/wiki/Rhapsody_(operating_system)

The log in https://trac.macports.org/ticket/68586 was from Sonoma (arm64), but here is the new one (same outcome): opam_2.1.5_Sonoma_fail.log

[modlfo]

I'm experiencing the same issue on macOs (arm64) when running make cold on 2.1.5. This is in a build machine where no OCaml compiler is available.

I tested building 2.2.0~alpha3 and it works fine.

[barracuda156]

@modlfo Thank you for the info. Good to know it is not happening only to us in Macports.

(I do have OCaml available on both machines where opam in being built though.)

[kit-ty-kate]

I found the reasons for this issue.

The first one is that on 2.1.5 we try to detect if dune supports --promote-install-files using an outdated method (pre dune 2.9.1) and that fails if dune >= 2.9.1 is already installed.

The second one is that for some reason we do not build dune locally when using make cold when dune is already installed. That’s an issue as make cold is supposed to make the compilation of opam reproducible.

We’ll try to fix those two issues in the coming days.

[kit-ty-kate]

In the meantime simply removing dune from the PATH should be enough.

[barracuda156]

@kit-ty-kate I tried deactivating ocaml-dune port and building opam, and got some sort of a new disaster:

rm dune-local.download
make[1]: Leaving directory `/opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_ocaml_opam/opam/work/opam-full-2.1.5/src_ext'
cd src_ext/dune-local && ocaml bootstrap.ml
ocamlc -output-complete-exe -w -24 -g -o .duneboot.exe -I boot unix.cma boot/libs.ml boot/duneboot.ml
./.duneboot.exe
cd _boot && /opt/local/bin/ocamlc -c -g -I +threads fsevents_stubs.c
src/fsevents/fsevents_stubs.c:67:5: error: 'kFSEventStreamEventFlagItemCreated' undeclared here (not in a function); did you mean 'kFSEventStreamEventFlagUserDropped'?
src/fsevents/fsevents_stubs.c:67:42: error: 'kFSEventStreamEventFlagItemRemoved' undeclared here (not in a function); did you mean 'kFSEventStreamEventFlagUserDropped'?
src/fsevents/fsevents_stubs.c:68:5: error: 'kFSEventStreamEventFlagItemRenamed' undeclared here (not in a function); did you mean 'kFSEventStreamEventFlagRootChanged'?
src/fsevents/fsevents_stubs.c:68:42: error: 'kFSEventStreamEventFlagItemModified' undeclared here (not in a function); did you mean 'kFSEventStreamEventFlagUserDropped'?
src/fsevents/fsevents_stubs.c: In function 'dune_fsevents_create':
src/fsevents/fsevents_stubs.c:160:44: error: 'kFSEventStreamCreateFlagFileEvents' undeclared (first use in this function); did you mean 'kFSEventStreamCreateFlagNoDefer'?
src/fsevents/fsevents_stubs.c:160:44: note: each undeclared identifier is reported only once for each function it appears in
src/fsevents/fsevents_stubs.c: In function 'dune_fsevents_set_exclusion_paths':
src/fsevents/fsevents_stubs.c:185:14: warning: implicit declaration of function 'FSEventStreamSetExclusionPaths' [-Wimplicit-function-declaration]
src/fsevents/fsevents_stubs.c: In function 'dune_fsevents_kind':
src/fsevents/fsevents_stubs.c:256:15: error: 'kFSEventStreamEventFlagItemIsDir' undeclared (first use in this function); did you mean 'kFSEventStreamEventFlagUnmount'?
src/fsevents/fsevents_stubs.c: At top level:
src/fsevents/fsevents_stubs.c:298:5: error: 'kFSEventStreamEventFlagItemInodeMetaMod' undeclared here (not in a function); did you mean 'kFSEventStreamEventFlagUserDropped'?
src/fsevents/fsevents_stubs.c:301:5: error: 'kFSEventStreamEventFlagItemFinderInfoMod' undeclared here (not in a function); did you mean 'kFSEventStreamEventFlagKernelDropped'?
src/fsevents/fsevents_stubs.c:302:5: error: 'kFSEventStreamEventFlagItemChangeOwner' undeclared here (not in a function); did you mean 'kFSEventStreamEventFlagRootChanged'?
src/fsevents/fsevents_stubs.c:303:5: error: 'kFSEventStreamEventFlagItemXattrMod' undeclared here (not in a function); did you mean 'kFSEventStreamEventFlagUnmount'?
src/fsevents/fsevents_stubs.c:304:5: error: 'kFSEventStreamEventFlagItemIsFile' undeclared here (not in a function); did you mean 'kFSEventStreamEventFlagUnmount'?
src/fsevents/fsevents_stubs.c:305:5: error: 'kFSEventStreamEventFlagItemIsDir' undeclared here (not in a function); did you mean 'kFSEventStreamEventFlagUnmount'?
src/fsevents/fsevents_stubs.c:306:5: error: 'kFSEventStreamEventFlagItemIsSymlink' undeclared here (not in a function); did you mean 'kFSEventStreamEventFlagUnmount'?
src/fsevents/fsevents_stubs.c:307:5: error: 'kFSEventStreamEventFlagOwnEvent' undeclared here (not in a function); did you mean 'kFSEventStreamEventFlagUnmount'?
src/fsevents/fsevents_stubs.c:308:5: error: 'kFSEventStreamEventFlagItemIsHardlink' undeclared here (not in a function); did you mean 'kFSEventStreamEventFlagHistoryDone'?
src/fsevents/fsevents_stubs.c:309:5: error: 'kFSEventStreamEventFlagItemIsLastHardlink' undeclared here (not in a function); did you mean 'kFSEventStreamEventFlagHistoryDone'?
make: *** [src_ext/dune-local/dune.exe] Error 2

[barracuda156]

Looks like these things should be provided if missing. See: https://github.com/macports/macports-ports/commit/ce4020d5d4e70cc3600fb916e0bc6c51c8628471#commitcomment-71571495

And https://github.com/git/git/commit/5ff01b1f1e8e42805bdf98cb0bd3277d18543b07

[kit-ty-kate]

that’s a dune issue. Could you open a ticket at https://github.com/ocaml/dune/issues?

[barracuda156]

It has been fixed in Dune quite some time ago: https://github.com/ocaml/dune/pull/6515

(I recalled that after I commented about the error – it looked vaguely familiar and I looked for issues mentioning it with Dune repo.)

[barracuda156]

@kit-ty-kate Could you please say what is the status of this?

Will you merge a PR if I backport a fix from dune for the second failure? Or you plan to update dune bundled source to a version which contains upstream fix anyway?

UPD. Ah, it is not possible to fix it here, since you simply pull dune source: https://github.com/ocaml/opam/blob/master/src_ext/Makefile.dune Then the only simple way will be to update dune here, is that possible?

[barracuda156]

It has been fixed in Dune quite some time ago: ocaml/dune#6515

The link, however, points to an early version which did not work. Currently upstream has: https://github.com/ocaml/dune/blob/1bff4990a3db7ae1876a2e9220ee0701fb762ff2/src/fsevents/fsevents_stubs.c

[barracuda156]

@kit-ty-kate I actually made it work.

1. You are right, deactivating dune solves the issue of building opam 2.1.5.
2. Moving the following patch to the patches folder applies the fix to bundled dune source: fix-bundled-dune.patch

[kit-ty-kate]

Sorry for the lack of update. I've opened https://github.com/ocaml/opam/pull/5870 (for 2.1) and https://github.com/ocaml/opam/pull/5869 (for master/2.2) and those should fix the issues you were having.

[barracuda156]

@kit-ty-kate Thank you very much!

P. S. Right now away from my testing hardware, will be back in a week.

[kit-ty-kate]

opam 2.1.6 has been released. I believe everything should be fixed so I'm going to close this. If you're still seeing issues with the new release please feel free to reopen.

Thanks for the report

[barracuda156]

@kit-ty-kate Looks like it is working now without a patch and with active Dune in the path, so all good. Thank you!