ocamlfind will not build on OSX Catalina if CLICOLOR=1 and/or CLICOLOR_FORCE=1

[philCryoport]

tl;dr:

When building ocamlfind on OSX:

• First: check to see if ls site-lib-src outputs colorized text.
  • If the output is not colorized text, then keep going
  • Otherwise, see if gls (gnu-ls) is installed.
    • If gls is installed, keep going
    • If gls is not installed, tell the user they need to install Gnu's ls, for instance, via brew install coreutils
• Second: you need to unroll https://github.com/ocaml/ocamlfind/blob/6c8438c13b68e5cbc48ea558906046e38a7f8d72/Makefile#L99 to properly capture the exit code from seeing if a META file exists in a sub-sub-directory of site-lib-src

STORY:

I come here after much messing around trying to get opam install csvtool to work on OSX Catalina on an Intel Mac.

It was failing on ocaml install topkg. Working with @kit-ty-kate and @dbuenzli starting at https://github.com/dbuenzli/topkg/issues/119#issuecomment-1073389576 we took a look at ocamlfind

This line in the Makefile appears to have an issue: https://github.com/ocaml/ocamlfind/blob/6c8438c13b68e5cbc48ea558906046e38a7f8d72/Makefile#L99

If I'm understanding of that line correctly, the goal is:

1. Iterate over a list of site-lib-src directories; call the current iterated value x
2. If site-lib-src/x/META IS a regular file, return exit code 1; else return exit code 0
3. If the exit code is non-zero -- meaning that IT WAS a regular file -- then echo that x into Makefile.packages.in -- else go to next entry from the site-lib-src directory

After doing

./configure
make all
make opt
make install

I find there are 3 files that qualify for that META:

➜ find site-lib-src -name "META" -print
site-lib-src/bytes/META
site-lib-src/stdlib/META
site-lib-src/dynlink/META

And if I run test ! -f "site-lib-src/bytes/META" || echo "bytes" >> Makefile.packages.in, indeed I do get:

➜ cat Makefile.packages.in
bytes

The problem? As soon as one puts that test ! -f "site-lib-src/$$x/META" || echo $$x >> Makefile.packages.in line inside a for loop, OSX always considers the referenced META file returns 0.

Every. Single. Time.

...which means that the Makefile.packages.in file is never created...and the Makefile.packages file ends up just being:

➜ cat Makefile.packages
SITELIB_META = %

You would think the easy answer would be to -- within the for loop:

1. Run the test command by itself, sending the output to /dev/null
2. Assign $? to a variable
3. Now process accordingly

It gets even more complicated:

In OSX, if one is using a fancy-schmancy terminal program -- like iterm2 -- with a nice shell and prompt -- like oh-my-zsh with the spaceship-prompt -- then ls is colorized due to a forced alias of ls -G...

...and the ls site-lib-src output can't be properly understood by the test command because the characters are in blue.

...which means on OSX we need to de-colorize the ls site-lib-src output before using it.

"Oh", you think, "let's just use ls --color=never", right?

Doesn't work.

How about \ls?

Nope -- nor "ls" nor 'ls'

So far the only way I've found to do that do that is to brew install coreutils -- which then provides gls -- which then offers a non-colorized output of site-lib-src.

Unrolled bash script: https://gist.github.com/philCryoport/5565b8fae4b58b5a26823860f28691ad

Resulting Makefile.packages.in:

➜ cat Makefile.packages.in
bigarray
bytes
compiler-libs
dynlink
ocamldoc
stdlib
str
threads
unix

Please do fix when you get a moment?

Thanks.

[dbuenzli]

"Oh", you think, "let's just use ls --color=never", right?

Here TERM=dumb ls -G seems to work in not outputing colors.

[philCryoport]

"Oh", you think, "let's just use ls --color=never", right?

Here TERM=dumb ls -G seems to work in not outputing colors.

Nice @dbuenzli , that'll take care of the colorization issue.

However it still fails to build. Gist of:

make clean
./configure
make all
make opt
make install

https://gist.github.com/philCryoport/1486159bcaed6c245dbab65a8e945787

[philCryoport]

Ok if I:

• add in the TERM=dumb preface to ls site-lib-src and
• modify the Makefile to now assign the exit code to a variable while unrolling that for loop

diff --git a/Makefile b/Makefile
index d57f259..0d1e8ae 100644
--- a/Makefile
+++ b/Makefile
@@ -96,7 +96,13 @@ check-installation:
     test -f "site-lib-src/num/META" || rm -f "site-lib-src/num-top/META"; \
   fi
        echo 'SITELIB_META =' > Makefile.packages.in
-       for x in `ls site-lib-src`; do test ! -f "site-lib-src/$$x/META" || echo $$x >> Makefile.packages.in; done
+       for x in `TERM=dumb ls site-lib-src`; do \
+               test ! -f "site-lib-src/$$x/META" >/dev/null; \
+               rc=$$?; \
+         if [ $$rc -ne 0 ]; then \
+                 echo $$x >> Makefile.packages.in; \
+               fi; \
+       done;
        tr '\n' ' ' < Makefile.packages.in > Makefile.packages
        rm Makefile.packages.in

...then Makefile.packages.in does appear to be generated -- as Makefile.packages does end up with a space-separated values list:

➜ find site-lib-src -name "META"
site-lib-src/bytes/META
site-lib-src/stdlib/META
site-lib-src/dynlink/META
(base)
ocamlfind on  master [!?] via 🅒 base
➜ cat Makefile.packages
SITELIB_META = bytes dynlink stdlib %

However, it's still failing to make install after make clean; ./configure; make all; make opt;:

➜ make install
if [ "0" -eq 1 ]; then \
    for x in camlp4 dbm graphics labltk num ocamlbuild; do \
      if [ -f "/Users/<user>/.opam/4.13.1/lib/ocaml/site-lib/$x/META" ]; then \
        if ! grep -Fq '[distributed with Ocaml]' "//Users/<user>/.opam/4.13.1/lib/ocaml/site-lib/$x/META"; then \
          rm -f site-lib-src/$x/META; \
        fi; \
      fi; \
    done; \
    test -f "site-lib-src/num/META" || rm -f "site-lib-src/num-top/META"; \
  fi
echo 'SITELIB_META =' > Makefile.packages.in
for x in `TERM=dumb ls site-lib-src`; do \
		test ! -f "site-lib-src/$x/META" >/dev/null; \
		rc=$?; \
	  if [ $rc -ne 0 ]; then \
		  echo $x >> Makefile.packages.in; \
		fi; \
	done;
tr '\n' ' ' < Makefile.packages.in > Makefile.packages
rm Makefile.packages.in
mkdir -p "/Users/<user>/.opam/4.13.1/bin"
mkdir -p "/Users/<user>/.opam/4.13.1/man"
/Library/Developer/CommandLineTools/usr/bin/make install-config
mkdir -p "`dirname \"/Users/<user>/.opam/4.13.1/lib/findlib.conf\"`"
!!! Keeping old /Users/<user>/.opam/4.13.1/lib/findlib.conf !!!
test -f "/Users/<user>/.opam/4.13.1/lib/findlib.conf" || cp findlib.conf "/Users/<user>/.opam/4.13.1/lib/findlib.conf"
for p in findlib; do ( cd src/$p; /Library/Developer/CommandLineTools/usr/bin/make install ); done
mkdir -p "/Users/<user>/.opam/4.13.1/lib/ocaml/site-lib/findlib"
mkdir -p "/Users/<user>/.opam/4.13.1/bin"
test 1 -eq 0 || cp topfind "/Users/<user>/.opam/4.13.1/lib/ocaml"
files=` ../../tools/collect_files ../../Makefile.config \
	findlib.cmi findlib.mli findlib.cma findlib.cmxa findlib.a findlib.cmxs \
	findlib_config.cmi findlib_config.ml topfind.cmi topfind.mli \
	fl_args.cmi fl_lint.cmi fl_meta.cmi fl_split.cmi fl_topo.cmi ocaml_args.cmi \
	fl_package_base.mli fl_package_base.cmi fl_metascanner.mli fl_metascanner.cmi \
	fl_metatoken.cmi findlib_top.cma findlib_top.cmxa findlib_top.a findlib_top.cmxs \
	findlib_dynload.cma findlib_dynload.cmxa findlib_dynload.a findlib_dynload.cmxs fl_dynload.mli fl_dynload.cmi \
	META` && \
	cp $files "/Users/<user>/.opam/4.13.1/lib/ocaml/site-lib/findlib"
f="ocamlfind"; { test -f ocamlfind_opt && f="ocamlfind_opt"; }; \
	cp $f "/Users/<user>/.opam/4.13.1/bin/ocamlfind"
# the following "if" block is only needed for 4.00beta2
if [ 1 -eq 0 -a -f "/Users/<user>/.opam/4.13.1/lib/ocaml/compiler-libs/topdirs.cmi" ]; then \
	    cd "/Users/<user>/.opam/4.13.1/lib/ocaml/compiler-libs/"; \
	    cp topdirs.cmi toploop.cmi "/Users/<user>/.opam/4.13.1/lib/ocaml/site-lib/findlib/"; \
	fi
/Library/Developer/CommandLineTools/usr/bin/make install-meta
for x in bytes dynlink stdlib ; do mkdir -p "/Users/<user>/.opam/4.13.1/lib/ocaml/site-lib/$x"; cp site-lib-src/$x/META "/Users/<user>/.opam/4.13.1/lib/ocaml/site-lib/$x/META.tmp" && mv "/Users/<user>/.opam/4.13.1/lib/ocaml/site-lib/$x/META.tmp" "/Users/<user>/.opam/4.13.1/lib/ocaml/site-lib/$x/META"; done
mkdir -p "/Users/<user>/.opam/4.13.1/lib/ocaml/site-lib/findlib"; cp Makefile.packages "/Users/<user>/.opam/4.13.1/lib/ocaml/site-lib/findlib/Makefile.packages"
test ! -f 'site-lib-src/num-top/META' || { cd src/findlib; /Library/Developer/CommandLineTools/usr/bin/make install-num-top; }
test ! -f 'site-lib-src/camlp4/META' ||	cp tools/safe_camlp4 "/Users/<user>/.opam/4.13.1/bin"
/Library/Developer/CommandLineTools/usr/bin/make install-doc
mkdir -p "/Users/<user>/.opam/4.13.1/man/man1" "/Users/<user>/.opam/4.13.1/man/man3" "/Users/<user>/.opam/4.13.1/man/man5"
cp doc/ref-man/ocamlfind.1 "/Users/<user>/.opam/4.13.1/man/man1"
cp: doc/ref-man/ocamlfind.1: No such file or directory
make[1]: [install-doc] Error 1 (ignored)
cp doc/ref-man/META.5 doc/ref-man/site-lib.5 doc/ref-man/findlib.conf.5 "/Users/<user>/.opam/4.13.1/man/man5"
cp: doc/ref-man/META.5: No such file or directory
cp: doc/ref-man/site-lib.5: No such file or directory
cp: doc/ref-man/findlib.conf.5: No such file or directory
make[1]: [install-doc] Error 1 (ignored)

[dbuenzli]

In OSX, if one is using a fancy-schmancy terminal program -- like iterm2 -- with a nice shell and prompt -- like oh-my-zsh with the spaceship-prompt -- then ls is colorized due to a forced alias of ls -G...

Btw. I'm not exactly sure what is happening here. I do have:

> command -v ls
alias ls='ls -G'

and everything's working right.

Besides I really don't get what is going on with that loop (but with shell scripting I usually prefer not to know). Do you maybe have a test executable in your PATH that does fancy things ?

However, it's still failing to make install after make clean; ./configure; make all; make opt;:

[...]

cp doc/ref-man/ocamlfind.1 "/Users//.opam/4.13.1/man/man1" cp: doc/ref-man/ocamlfind.1: No such file or directory make[1]: [install-doc] Error 1 (ignored) cp doc/ref-man/META.5 doc/ref-man/site-lib.5 doc/ref-man/findlib.conf.5 "/Users//.opam/4.13.1/man/man5" cp: doc/ref-man/META.5: No such file or directory cp: doc/ref-man/site-lib.5: No such file or directory cp: doc/ref-man/findlib.conf.5: No such file or directory

These things are generated by doc/Makefile, but need more tools. Apparently they exist in the release tarball.

[gerdstolpmann]

I also don't get what is happening here. Normally, ls falls back to the behavior of ls -1 (and disabling all fanciness) when stdout is a pipe (and not a terminal). E.g. try

ls -G | cat

and you'll see that there are no colors, and everything is in one column. The same happens when capturing the output, like

v="$(ls -G)"
echo $v

(I tested this on MacOS Monterey.)

Are you setting SHELL?

[philCryoport]

I also don't get what is happening here. Normally, ls falls back to the behavior of ls -1 (and disabling all fanciness) when stdout is a pipe (and not a terminal). E.g. try

ls -G | cat

and you'll see that there are no colors, and everything is in one column. The same happens when capturing the output, like

v="$(ls -G)"
echo $v

(I tested this on MacOS Monterey.)

When I run that within either iTerm or within OSX's own Terminal, I get colors:

Only thing I can guess is that it's a result of using oh-my-zsh and the spaceship-prompt.

Are you setting SHELL?

My system (Catalina) automatically does that. It's not in .zshrc; perhaps it's setting it because I made zsh my default shell...

➜ echo $SHELL
/bin/zsh

[philCryoport]

In OSX, if one is using a fancy-schmancy terminal program -- like iterm2 -- with a nice shell and prompt -- like oh-my-zsh with the spaceship-prompt -- then ls is colorized due to a forced alias of ls -G...

Btw. I'm not exactly sure what is happening here. I do have:

> command -v ls
alias ls='ls -G'

and everything's working right.

Besides I really don't get what is going on with that loop (but with shell scripting I usually prefer not to know). Do you maybe have a test executable in your PATH that does fancy things ?

ocamlfind on  master [!?] via 🅒 base
➜ which test
test: shell built-in command
(base)
ocamlfind on  master [!?] via 🅒 base
➜
(base)
ocamlfind on  master [!?] via 🅒 base
➜ whereis test
/bin/test
(base)
ocamlfind on  master [!?] via 🅒 base
➜ ls -l /bin/test
-rwxr-xr-x  1 root  wheel  121120 Jan  1  2020 /bin/test
(base)

However, it's still failing to make install after make clean; ./configure; make all; make opt;:

[...]

cp doc/ref-man/ocamlfind.1 "/Users//.opam/4.13.1/man/man1" cp: doc/ref-man/ocamlfind.1: No such file or directory make[1]: [install-doc] Error 1 (ignored) cp doc/ref-man/META.5 doc/ref-man/site-lib.5 doc/ref-man/findlib.conf.5 "/Users//.opam/4.13.1/man/man5" cp: doc/ref-man/META.5: No such file or directory cp: doc/ref-man/site-lib.5: No such file or directory cp: doc/ref-man/findlib.conf.5: No such file or directory

These things are generated by doc/Makefile, but need more tools. Apparently they exist in the release tarball.

Yeah it looks like it needs openjade and readme.

• openjade doesn't want to build from source. It complains about time; my system clock is synchronized, so dunno what's going on...

~/Downloads/openjade-1.3.2 via 🅒 base
➜ make clean
make: *** No rule to make target `clean'.  Stop.
(base)
~/Downloads/openjade-1.3.2 via 🅒 base
➜ ./configure
checking for a BSD-compatible install... /usr/local/bin/ginstall -c
checking whether build environment is sane... configure: error: ls -t appears to fail.  Make sure there is not a broken
alias in your environment
configure: error: newly created file is older than distributed files!
Check your system clock
(base)
~/Downloads/openjade-1.3.2 via 🅒 base
➜ date
Tue Mar 22 10:12:57 PDT 2022
(base)

• My google-fu is failing me; I can't find where to get readme.

[dbuenzli]

(Forget about that openjade thing)

Only thing I can guess is that it's a result of using oh-my-zsh and the spaceship-prompt.

I think you should really sort that out.

If ls | cat shows up with colors you are not only going to get problems with ocamlfind… Start by disabling these things and see if you succeed.

My system (Catalina) automatically does that. It's not in .zshrc; perhaps it's setting it because I made zsh my default shell...

That's ok I have that here too by default (i.e. SHELL=/bin/zsh on 11.6) and it's not posing any problem.

I'm starting to think we need to fix your system rather than ocamlfind here :–)

[philCryoport]

(Forget about that openjade thing)

Only thing I can guess is that it's a result of using oh-my-zsh and the spaceship-prompt.

I think you should really sort that out.

If ls | cat shows up with colors you are not only going to get problems with ocamlfind… Start by disabling these things and see if you succeed.

My system (Catalina) automatically does that. It's not in .zshrc; perhaps it's setting it because I made zsh my default shell...

That's ok I have that here too by default (i.e. SHELL=/bin/zsh on 11.6) and it's not posing any problem.

I'm starting to think we need to fix your system rather than ocamlfind here :–)

Ok I figured out what was tripping up my system: these lines in my .zshrc:

# Colors.
unset LSCOLORS
export CLICOLOR=1
export CLICOLOR_FORCE=1

If I comment out those three lines and then open a fresh terminal, then opam install csvtool -- the end goal of all of this -- that works without an issue.

That was ANNOYING.

I have yet to see anything else tripped by this on my system:

• Homebrew apps/casks (I have more than 400 installed)
• My custom Ruby scripts
• My custom PHP scripts
• My custom Java scripts
• Python packages (I have more than 200 installed)
• My custom pure shell scripts
• ...I'd have to go back and look, but I am certain I have had to build other projects with autoconf and those also built without an issue.

Therefore -- since ocamlfind and potential other packages might get tripped up by CLICOLOR=1 and (more likely) CLICOLOR_FORCE=1 -- it's probably best that defensive programming modifications be made to unset those variables and/or prepend command line calls with TERM=dumb (as you suggested above) whenever ocamlfind and other packages are tapping into the output of ls or anything else that might bring in colorized text -- unless the expectation for that call is explictly to handle colorized text.

[philCryoport]

Check this out from GNU's autoconf manual: https://www.gnu.org/software/autoconf/manual/autoconf-2.68/html_node/Special-Shell-Variables.html

CLICOLOR_FORCE When this variable is set, some implementations of tools like ls attempt to add color to their output via terminal escape sequences, even when the output is not directed to a terminal, and can thus cause spurious failures in scripts. Configure scripts use M4sh, which automatically unsets this variable.

[philCryoport]

@dbuenzli looks like you dealt with a similar issue previously: https://github.com/ocaml/opam-repository/issues/10737#issuecomment-345533638

[philCryoport]

The opam repo makes sure CLICOLOR_FORCE is unset: https://github.com/ocaml/opam/blob/5ad1b5093f801420ca8308652af45358304ac4c9/configure#L280

[dbuenzli]

Indeed though that was a grep env var. I overlooked that CLICOLOR_FORCE in the env dump you provided me.

That being said I'm still unclear why one would want CLICOLOR_FORCE to be set by default (note that CLI_COLOR=1 ls | cat works).

Not everything is configured, writing a simple for loop in your shell with ls will fail, there's ton of scripts out there that will never bother to unset that variable.

Unless you are in to debug obscure fail paths (as you have now witnessed :–) I would highly suggest to remove that line from your .zshrc.

At that point I must still quote:

We build our computers the way we build our cities – over time, without a plan, on top of ruins. – Ellen Ullman

and wish you happy OCaml hacking !

[dbuenzli]

(oh and so that you don't run into other obscurities since you are on catalina please do ~~opam~~ brew install mandoc if you ever wish to use the latest version of the cmdliner package, sorry about that 😬)

[philCryoport]

(oh and so that you don't run into other obscurities since you are on catalina please do opam install mandoc if you ever wish to use the latest version of the cmdliner package, sorry about that 😬)

➜ opam install mandoc
[ERROR] No package named mandoc found.

Did you mean brew install mandoc?

[dbuenzli]

Yes sorry.

Btw. is this ohmyzsh thing doing this CLICOLOR_FORCE setup automatically ? Or are people cargo culting what they find in this issue ?

In the first case it might be worth opening an issue there, in the second case it might be worth mentioning that CLICOLOR_FORCE=1 can break shell scripts on that issue.

[philCryoport]

Yes sorry.

Btw. is this ohmyzsh thing doing this CLICOLOR_FORCE setup automatically ? Or are people cargo culting what they find in this issue ?

In the first case it might be worth opening an issue there, in the second case it might be worth mentioning that CLICOLOR_FORCE=1 can break shell scripts on that issue.

I'll ping the oh-my-zsh people to find out what's going on.

[dra27]

Blimey, so much for POSIX 🙂 Two thoughts:

1. I'm not certain, but given that we control the files, I think good old POSIX globbing is safe here? Given that there are no files with spaces, etc., for x in site-lib-src/*; do test ! -f "$$x/META" || echo $$x >> Makefile.packages.in; done should be equivalent to for x in ls site-lib-src; do test ! -f "site-lib-src/$$x/META" || echo $$x >> Makefile.packages.in; done
2. Rather than being forced to mess around with ever more environment variables and commands, might it be actually be simpler to adopt GNU make and just use $(wildcard ... for all its many warts, GNU make is really rather portable...!

[dbuenzli]

1. I'm not certain, but given that we control the files, I think good old POSIX globbing is safe here?

I actually played with that yesterday the problem it's brittle in the face of nothing to glob:

> sh --posix -c 'for f in heyhobla/*; do echo $f; done' 
heyhobla/*

Ruins.

There's also:

3. Simplify all that by upstreaming these META files where they belong to (see e.g. https://github.com/ocaml/ocaml/pull/11007)

[dra27]

Indeed, but there's never going to be nothing in site-lib-src for site-lib-src/* to be mis-globbed, just as there's never going to be a directory name with a space - the present ls is brittle in the face of actual things which get broken (inadvertently) by users, as opposed to things which {w,d}on't get broken by findlib's maintainers 🙂

It'll certainly be good to set a better future with OCaml installing the META files itself, but that still needs ocamlfind fixed until all the previous versions of OCaml aren't used any more!