Doesn't install "out of the box" on a mac: magicffi error 'magic.h' file not found [fixed on master]

[agam]

When I tried (cloning and then loading via quicklisp) it got stuck at loading the magicffi package.

Specifically, at this step:

clang -o /Users/agambrahma/.cache/common-lisp/sbcl-2.2.4-macosx-arm64/Users/agambrahma/quicklisp/dists/quicklisp/software/magicffi-20210531-git/src/grovel__grovel-tmpTT1OWD6.o -c -I/usr/include -I/usr/local/include -g -Wall -Wundef -Wsign-compare -Wpointer-arith -O3 -g -Wall -fdollars-in-identifiers -fno-omit-frame-pointer -fPIC -I/Users/agambrahma/quicklisp/dists/quicklisp/software/cffi_0.24.1/ /Users/agambrahma/.cache/common-lisp/sbcl-2.2.4-macosx-arm64/Users/agambrahma/quicklisp/dists/quicklisp/software/magicffi-20210531-git/src/grovel__grovel.c
/Users/agambrahma/.cache/common-lisp/sbcl-2.2.4-macosx-arm64/Users/agambrahma/quicklisp/dists/quicklisp/software/magicffi-20210531-git/src/grovel__grovel.c:6:10: fatal error: 'magic.h' file not found
#include <magic.h>
         ^~~~~~~~~
1 error generated.

Ciel is a great idea for making CL approachable, perhaps it should be reduced to a version that "just works" ?

(or maybe I'm missing something obvious -- if anyone's got this working out-of-the-box on a mac, lemme know !)

[vindarel]

Following magicffi's README https://github.com/guicho271828/magicffi, we read:

The following system packages are also required:

    libmagic-dev
    libc6-dev
    gcc

Does that make it work for you?

Then there are several approaches we could study to make it work out of the box: make the .asd install system dependencies, create a minimal CIEL system that would not rely on those, ship a ready-to-use binary and image for all platforms.

(edit) and ship a Docker image, of course.

[patrixl]

I managed to install a package, using brew, that has the magic.h file, the package is: libmagic

It contains:

/opt/homebrew/Cellar/libmagic/5.44/include/magic.h
/opt/homebrew/Cellar/libmagic/5.44/lib/libmagic.1.dylib
/opt/homebrew/Cellar/libmagic/5.44/lib/pkgconfig/libmagic.pc
/opt/homebrew/Cellar/libmagic/5.44/lib/ (2 other files)
/opt/homebrew/Cellar/libmagic/5.44/share/man/ (2 files)
/opt/homebrew/Cellar/libmagic/5.44/share/misc/ (339 files)

However, my lisp implementation (roswell + sbcl) does not look for files in these directories when compiling magicffi, but I feel like I'm one step closer...

debugger invoked on a CFFI-GROVEL:GROVEL-ERROR in thread #<THREAD "main thread" RUNNING {7005200923}>: Subprocess #<UIOP/LAUNCH-PROGRAM::PROCESS-INFO {70089E5BE3}>
 with command ("cc" "-o" "/Users/patrix/.cache/common-lisp/sbcl-2.2.6-macosx-arm64/Users/patrix/.roswell/lisp/quicklisp/dists/quicklisp/software/magicffi-20210531-git/src/grovel__grovel-tmpAAURSO1.o" "-c" "-I/usr/include" "-I/usr/local/include" "-g" "-Wall" "-Wundef" "-Wsign-compare" "-Wpointer-arith" "-O3" ...)
 exited with error code 1

[patrixl]

I've forked magicffi to add the include dirs and lib dirs in the magicffi.asd:

https://github.com/patrixl/magicffi

This still requires manually symlinking libmagic.dylib to libmagic.so, if I can find how to tell cc to use .dylib instead of .so that won't be necessary anymore

(now there are other issues following this, so might open a new issue)

[patrixl]

Ok so here were the issues, which are now resolved:

1. ciel has a dependency which seems to require package local nicknames, therefore needs sbcl 2.3.x (or a newer asdf?)
2. ciel depends on "vgplot", which is broken under sbcl 2.3.x but works under 2.2.x

After posting a bug report to the SBCL mailing list, the bug was quickly fixed and should roll out in 2.3.8.

Meanwhile, I:

• cloned https://gitlab.common-lisp.net/asdf/asdf in ~/quicklisp/local-projects/ to get the most recent asdf (quicklisp's was slightly too old? weird...)
• installed sbcl "head" using brew
• installing my fork of magicffi (I should probably send a PR...)

... and I was able to load the "ciel" system!

Maybe I'll try nuking my quicklisp installation and try again just to be sure all those steps were necessary (especially the asdf update)

[vindarel]

Thanks!

needs sbcl 2.x (or a newer asdf?)

yes, you need a newer ASDF to get package-local-systems. There's a one-liner in the README to install it:

    $ ( cd ~/common-lisp/ && wget https://asdf.common-lisp.dev/archives/asdf-3.3.5.tar.gz  && tar -xvf asdf-3.3.5.tar.gz && mv asdf-3.3.5 asdf )

bloken vgplot under sbcl 2.3.x … fix

ah damn… thanks for taking care of it.

magicffi (I should probably send a PR...)

yes :)

---

A simpler CIEL system or installation methods would still be appropriate.

[bo-tato]

I also was getting error about magic.h not found (on debian unstable) despite having libmagic-dev installed and /usr/include/magic.h present. I edited src/grovel.lisp to change (include "magic.h") to (include "/usr/include/magic.h") and then it built. Then I changed it back and tried clearing ~/.cache/common-lisp and deleting magicffi from quicklisp downloaded packages, and reinstalled it and weirdly I couldn't reproduce the issue again, it build fine after

[vindarel]

thanks for the feedback. Guys, are your fixes worth a PR to magicffi, or a discussion there? (ah, there are no issues because it's a fork, we should ask the author)

[bo-tato]

I've been unable to reproduce my issue again, so I don't even know if it was a problem in magicffi or some weird temporary problem with configuration or something of asdf or cffi-grovel

[vindarel]

magicffi

I recently removed the dependency on FOF (file object finder) which relied on magicffi. I'll replace it by the simplified https://github.com/lisp-maintainers/file-finder

I'll have to double-check magicffi isn't a transitive dependency of another library, but I don't think so.

[vindarel]

I removed the fof dependency, normally the only one relying on magicffi, and added the lightweight file-finder.

thanks for the feedback.