Makefile / configure.ac: realpath is not available on macOS

[tiran]

Bug report

Makefile.pre.in and configure.ac are using the realpath command for some targets. For example WASI HOSTRUNNER uses realpath --relative-to to get the working directory relative to the srcdir root directory. macOS seems to lack realpath, breaking WASI tests, see https://github.com/python/cpython/pull/95828#issuecomment-1213579582

For WASI HOSTRUNNER we need to replace realpath somehow. Or find a different approach to fix testing with OOT builds. The HOSTRUNNER for WASI is complicated because we have to make srcdir available to WASI runtime environment. --mapdir /::$(srcdir) maps the srcdir to root / inside the runtime environment. wasmtime sets cwd to /, so our logic to find pybuilddir.txt does not work. That's why the HOSTRUNNER sets --env PYTHONPATH=/$(shell realpath --relative-to $(abs_srcdir) $(abs_builddir))/$(shell cat pybuilddir.txt):/Lib, which resolves to --env PYTHONPATH=/builddir/wasi/build/lib.wasi-wasm32-3.12:/Lib on my system. The prefix depends on the OOT builddir. The suffix is read from pybuilddir.txt, which is generated at the end of the build process.

[ronaldoussoren]

macOS does not have a realpath command (at least not on macOS 12). There is a manual page, but that doesn't mention a relative-to option (it basically describes a command-line wrapper around realpath(3)).

UPDATE: The beta for the next major release of macOS does include a realpath(1) command, but that doesn't have the --relative-to option and hence isn't usable for your use case.

[erlend-aasland]

@brettcannon, is this still relevant?

[brettcannon]

@brettcannon, is this still relevant?

I have no idea (I certainly don't need it as I generate a shell script to avoid such issues), so I'm going to close it as unnecessary.

[ronaldoussoren]

configure.ac still contains code using realpath options not supported on macOS:

dnl TODO: support other WASI runtimes
    dnl wasmtime starts the proces with "/" as CWD. For OOT builds add the
    dnl directory containing _sysconfigdata to PYTHONPATH.
    [WASI/*], [HOSTRUNNER='wasmtime run --env PYTHONPATH=/$(shell realpath --relative-to $(abs_srcdir) $(abs_builddir))/$(shell cat pybuilddir.txt):/Lib --mapdir /::$(srcdir) --'],

[brettcannon]

I believe @ronaldoussoren is referring to:

https://github.com/python/cpython/blob/939fc6d6eab9b7ea8c244d513610dbdd556503a7/configure.ac#L1417-L1420

I personally think that should be changed to error out if the host runner is not specified for WASI. With Tools/wasm/wasi.py (and the old Tools/wasm/wasm_build.py), there's enough help in setting that value externally. Otherwise we perhaps should have a --host-runner option to ./configure?

[ronaldoussoren]

I believe @ronaldoussoren is referring to:

That's correct.

https://github.com/python/cpython/blob/939fc6d6eab9b7ea8c244d513610dbdd556503a7/configure.ac#L1417-L1420

I personally think that should be changed to error out if the host runner is not specified for WASI. With Tools/wasm/wasi.py (and the old Tools/wasm/wasm_build.py), there's enough help in setting that value externally. Otherwise we perhaps should have a --host-runner option to ./configure?

I don't have an opinion on that, WASI/WebAsm support is interesting from the distance but I've never looked at it beyond reading articles ("how cool, CPython in a browser").

[ronaldoussoren]

There are three workarounds for this on MacOS:

1. Install coreutils if you're a homebrew user
2. https://codereview.stackexchange.com/questions/242050/realpath-substitute-with-relative-to-and-relative-base-support has a shell script implementation of realpath with --relative-to support
3. Implementing realpath in Python should be easy enough (and Apple's compiler ships with Python 3, which means anyone playing with this should have Python 3 available)

For (3) something like this should work (I may have gotten the two arguments backwards):

HOSTRUNNER='python3 -c "import sys, pathlib; print(pathlib.Path(sys.argv[1]).resolve().relative_to(sys.argv[2]), walk_up=True))" "$(abs_builddir)" "$(abs_srcdir)"'

Just dropping automatically settings HOSTRUNNER for WASI is probably easier ;-)