work around Mac OS dlopen restrictions

[rmculpepper]

Recent versions of Mac OS (by 11.2, but probably several versions before that) restrict the OS library search path for dlopen. The man page for dlopen has the following note:

Note: If the main executable is a set[ug]id binary or codesigned with entitlements, then all environment variables are ignored, and only a full path can be used.

Thus a call to ffi-lib with a plain library name, where the library is expected to be in a system location like /usr/local/lib, will succeed when run with a locally-built Racket but fail when run with a (signed) release Racket.

The note above isn't completely accurate: the entitlement com.apple.security.cs.allow-dyld-environment-variables, which Racket has, has the following effect:

• DYLD_LIBRARY_PATH is not ignored; if set to /usr/local/lib:/usr/lib then Racket does find libraries in those locations, but the variable is not set by default.
• DYLD_FALLBACK_LIBRARY_PATH is visible to Racket's getenv, but it is ignored by dlopen.

This commit works around this behavior by explicitly searching the OS library locations (set by the DYLD_FALLBACK_{LIBRARY,FRAMEWORK}_PATH variables).

---

See rmculpepper/racket-zeromq#6.

This PR still needs documentation. Would a parameter be better?

[mflatt]

I think I understand the logic here, but I have some reservations.

Although Racket must sometimes bend over backwards to conform to OS requirements, I can't think of a place that actively subverts the intent of OS restrictions like this — at least not in this way of duplicating OS functionality that was intentionally disabled. That doesn't mean it's definitely wrong, here, but it certainly gives pause.

Meanwhile, if I understand what's going on, this doesn't solve the problem for ARM Macs with Homebrew, since the ARM version installs into "/opt/homebrew/lib". (Granted, the change solves a problem of making entitled and unentitled Racket behave more the same.)

So, I'm wondering whether requiring users to set DYLD_LIBRARY_PATH to find Homebrew and similar installs might be the better choice, since (I think) that will work more often, it seems like generally the right thing separate from Racket, and it doesn't duplicate or work against OS functionality.

Another possibility would be to make it more convenient to update Racket's 'lib-search-dirs configuration. That could be more convenient than DYLD_LIBRARY_PATH in many cases, although it's still in the realm of having users take a specific step instead of trying to automatically solve the problem.

[rmculpepper]

I had hoped to avoid exposing the issue to users, but I understand your reservation, and the point about Homebrew on ARM makes it sound like this PR is too narrow. I do wonder if this is evidence that distributing Racket as a .dmg with individually signed executables is inconsistent with how Apple thinks about security.

In that case, adding paths to 'lib-search-dirs sounds like a good idea, if the user is able and willing to change the Racket installation's global configuration. A finer-grained solution is to symlink individual libraries from the user-specific Racket lib directory. The environment variable approach does seem inconvenient and possibly fragile.

I'll put together a draft package for automating this and some related issues and link it here for discussion.

[LiberalArtist]

I was thinking about this again and thought I should leave a pointer here to https://github.com/racket/racket/issues/3834#issuecomment-866941280 by @mflatt, about a similar way we're currently using 'lib-search-dirs in Guix's packaging of Racket:

I think you're right that 'so mode for define-runtime-path will pick up the libraries. I agree that it's not clear whether that's the right thing. If you want to compile for generic Linux using a Guix Racket installation, maybe raco cross is the way to go.