Use of dlopen on macOS

[ryandesign]

I understand that e2fsprogs can use libmagic if available. How can I discover whether e2fsprogs has found libmagic? Looking at the code, I believe it is not working on macOS. I'm the maintainer of e2fsprogs and libmagic in MacPorts.

The configure script checks for libmagic, and that seems to be fine, but if it also detects that dlopen exists, which it does on macOS, then it decides to use that method to load libmagic at runtime. There does not appear to be a configure flag to disable this behavior and just dynamically link the library normally, which would seem to be the most straightforward thing to do.

The code that loads libmagic at runtime using dlopen appears to be hardcoded to use a Linux shared library filename:

https://github.com/tytso/e2fsprogs/blob/0352d353adbe6c5d6f1937e12c66e599b8657d72/lib/support/plausible.c#L65

This won't work on macOS where it would be called libmagic.1.dylib and its absolute path would need to be specified. macOS doesn't ship from Apple with libmagic, but the user might have it installed manually or with MacPorts or Homebrew.

I see that there is also similar code for loading libreadline/libedit:

https://github.com/tytso/e2fsprogs/blob/25ad8a431331b4d1d444a70b6079456cc612ac40/lib/ss/get_readline.c#L40

[tytso]

For Linux, dlopen() will search the list of shared library directories, and the list of potential shared library names because there are a number of different of implementation of the readline, mostly for Licensing reasons. There is a GNU readline, a BSD editline, and there are multiple ABI changes mostly for the command completion (which debugfs doesn't use).

Is the issue for MacOS that there isn't a standard shared library name, or that there isn't a standard shared library directory where it is installed, and/or dlopen() doesn't seaarch a set of directories? If we can make dlopen() on MacOS do the right thimg, that's probably better for the user experience. If we can't, we can certainly add an override which forces using a static link of the library somehow.

The reason why we try to use dlopen() on Linux is that e2fsprogs needs to be installed for all/most root file systems and certainly for most rescue images (floppies, CDROM's, USB thumbdrives, etc.) So I try to avoid dragging in unnecessary dependencies if at all possible. With the BSD/MacOS Ports system, where users are doing a custom build of e2fsprogs, this is much less important ---- although I would imagine that if someone first built e2fsprogs, and then later built readline, it would be nice if (a) building e2fsprogs did not forcibly drag in readline, and (b) e2fsprogs could automatically get the benefits of readline regardless of the order in which they were built, and (c) we reduced the configuration complexity of whether the readline or editline or lineedit library is desired to be installed by the user (perhaps because of licensing preferences/concerns).

[ryandesign]

Is the issue for MacOS that there isn't a standard shared library name,

macOS dynamic library filenames end with .dylib (not .so like on Linux). The library major version number appears before the .dylib extension (not after the .so extension like on Linux) and there is typically a symlink pointing from the unversioned library to the latest versioned library.

or that there isn't a standard shared library directory where it is installed,

Dynamic libraries could be installed anywhere but are usually in a lib directory. For example macOS ships from Apple with /usr/lib/libedit.dylib which is a symlink to /usr/lib/libedit.3.dylib. Apple does not ship a copy of libmagic but the user could compile libmagic from source and install it anywhere (except for protected system directories like /usr/lib) or install it using MacPorts or Homebrew which could be configured to install anywhere.

and/or dlopen() doesn't seaarch a set of directories?

There is allegedly a search process, according to dlopen(3).

SEARCHING
     dlopen() searches for a compatible Mach-O file in the directories specified by a set
     of environment variables and the process's current working directory.  When set, the
     environment variables contain a colon-separated list of directory paths, which can be
     absolute or relative to the current working directory.

     When path does not contain a slash character (i.e. it is just a leaf name), dlopen()
     searches the following until it finds a compatible Mach-O file: $LD_LIBRARY_PATH,
     $DYLD_LIBRARY_PATH, current working directory, $DYLD_FALLBACK_LIBRARY_PATH.

     When path looks like a framework path (e.g. /stuff/foo.framework/foo), dlopen()
     searches the following until it finds a compatible Mach-O file: $DYLD_FRAMEWORK_PATH
     (with framework partial path from path ), then the supplied path (using current
     working directory for relative paths).

     When path contains a slash but is not a framework path (i.e. a full path or a partial
     path to a dylib), dlopen() searches the following until it finds a compatible Mach-O
     file: $DYLD_LIBRARY_PATH (with leaf name from path ), then the supplied path (using
     current working directory for relative paths).

     Note: If DYLD_FALLBACK_LIBRARY_PATH is not set, dlopen operates as if
     DYLD_FALLBACK_LIBRARY_PATH was set to $HOME/lib:/usr/local/lib:/usr/lib.

     Note: If DYLD_FALLBACK_FRAMEWORK_PATH is not set, dlopen operates as if
     DYLD_FALLBACK_FRAMEWORK_PATH was set to
     $HOME/Library/Frameworks:/Library/Frameworks:/Network/Library/Frameworks:/System/Library/Frameworks.

     Note: There are no configuration files to control dlopen searching.

     Note: If the main executable is a set[ug]id binary or codesigned with entitlements,
     then all environment variables are ignored, and only a full path can be used.

     Note: Mac OS X uses "universal" files to combine 32-bit and 64-bit libraries.  This
     means there are no separate 32-bit and 64-bit search paths.

However in this stack overflow question setting the relevant environment variables did not work, and in any case on macOS it is not expected for the user to need to configure anything for a program to find its dynamic libraries; the program should be able to do so on its own. From Jeremy's answer there:

You need to pass the full path to the library you want to open to the first argument of dlopen().

If you really need to use dlopen() to load this, you should be passing dlopen("/opt/local/lib/libsass.dylib", RTLD_NOW).

However, dlopen() is generally frowned upon as it bypasses a lot of the performance and correctness benefits of static linkage. You should aim to use static linkage (ie: pass -lsass at build time) wherever possible.

Jeremy is a longtime MacPorts contributor who maintains our X11 ports as well as the standalone X11 distribution XQuartz and is an engineer at Apple and I completely trust his advice on this.

If we can make dlopen() on MacOS do the right thimg, that's probably better for the user experience. If we can't, we can certainly add an override which forces using a static link of the library somehow.

I think the right thing to do for macOS is not to use dlopen and to do normal dynamic linking with whatever libraries are found at configure time (e.g. -lmagic -ledit). That's what pretty much every other program does.