Build issue on Mac M1

[grig-guz]

Hi, I get the following import error:

>>> import dmlab2d 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/grigoriiguz/opt/anaconda3/envs/rlenv/lib/python3.9/site-packages/dmlab2d/__init__.py", line 21, in <module>
    import dmlab2d.dmlab2d_pybind as dmlab2d_pybind
ImportError: dlopen(/Users/grigoriiguz/opt/anaconda3/envs/rlenv/lib/python3.9/site-packages/dmlab2d/dmlab2d_pybind.so, 0x0002): tried: '/Users/grigoriiguz/opt/anaconda3/envs/rlenv/lib/python3.9/site-packages/dmlab2d/dmlab2d_pybind.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')), '/usr/local/lib/dmlab2d_pybind.so' (no such file), '/usr/lib/dmlab2d_pybind.so' (no such file)

after building dmlab2d as specified here. It looks like the reason is that it generates a x_86_64 binary in bazel_bin/dmlab2d. Do you know how to generate arm64 binary instead?

[tkoeppe]

I'm afraid I don't have an M1 development environment and can't test that. I don't know why building from source on an M1 would result in an x86_64 binary, though; that seems implausible. Did you actually build the wheel file from source on the M1, or did you download a prebuilt version from our release on GitHub?

[grig-guz]

Here is what I am doing:

(rlenv) grigoriiguz@vpn3 lab2d % LUA_VERSION=lua5_1                                                    
(rlenv) grigoriiguz@vpn3 lab2d %   C=clang CXX=clang++ bazel build \
      --compilation_mode=opt \
      --dynamic_mode=off \
      --config="${LUA_VERSION}" \
      --verbose_failures \
      --experimental_ui_max_stdouterr_bytes=-1 \
      --sandbox_debug \
      //dmlab2d:dmlab2d_wheel

INFO: Analyzed target //dmlab2d:dmlab2d_wheel (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //dmlab2d:dmlab2d_wheel up-to-date:
  bazel-bin/dmlab2d/dmlab2d-1.0-cp39-cp39-macosx_12_0_x86_64.whl
  bazel-bin/dmlab2d/dmlab2d-1.0-cp39-cp39-macosx_12_0_x86_64.whlname
INFO: Elapsed time: 0.156s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action

I get dmlab2d-1.0-cp39-cp39-macosx_12_0_x86_64.whl file in bazel-bin/dmlab2d/

[tkoeppe]

Did you run that on an actual M1 machine? That's very odd -- maybe that's a bug in the packaging module? Could you directly run the packaging query for the SOABI tag, like in https://github.com/deepmind/lab2d/blob/89f3dd2c498410b5201d271ca50860f311025ca6/python_system.bzl#L38-L40, and see what that returns?

Can you also confirm that the DSO is actually using the AARCH64 architecture?

If this is just a labelling bug, try renaming the wheel file (or use the "any" tag, e.g. via https://github.com/deepmind/lab2d/blob/89f3dd2c498410b5201d271ca50860f311025ca6/dmlab2d/BUILD#L115-L130).

[grig-guz]

Here is the output from packaging:

PY_TAGS = struct(interpreter = "cp39", abi = "cp39", platform = "macosx_12_0_x86_64")

and running arch outputs arm64. After building the local wheel, I get this error now:

>>> import dmlab2d
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/grigoriiguz/projects/speech_acts/meltingpot/lab2d/dmlab2d/__init__.py", line 21, in <module>
    import dmlab2d.dmlab2d_pybind as dmlab2d_pybind
ModuleNotFoundError: No module named 'dmlab2d.dmlab2d_pybind'
>>> 

[tkoeppe]

Interesting. That looks like a bug in packaging for sure. Would you perhaps mind reporting that?

I don't know what the correct SOABI tag should be. If you can find out, try renaming the wheel file accordingly.

As for why the any wheel doesn't work (and I assume you pip-installed it after building it?), I'm afraid that's not enough information to tell. I'd try to debug why the module doesn't load: on Linux, I'd perhaps start with strace and LD_DEBUG to find out if a) the DSO is actually found in the right place, and if yes, b) what goes wrong during loading. I don't know the corresponding MacOS/BSD tools, though.

[grig-guz]

Hmm I ran this below and it looks like the issue is with my installation of Python itself?

>>> import sysconfig
>>> sysconfig.get_platform()
'macosx-10.9-x86_64'

as for the wheel itself, I also tried pip installing the standard dmlab2d-1.0-cp39-cp39-macosx_12_0_x86_64.whl one I am getting, and now I have the same no module named 'dmlab2d.dmlab2d_pybind' error. Thanks for help, I'll try to find the analogous tracing tools for MacOS.

[grig-guz]

Nevermind, please ignore the above. The no module named 'dmlab2d.dmlab2d_pybind' error was because I was inside the lab2d repo, in all cases. But I still have the ImportError, even if I pip install the local wheel.

[grig-guz]

Ok, I finally figured out the issue. I had x86_64 version of Anaconda, and in case someone is having the same issue, the arm64 version can be found here. After installing this, bazel was still generating x86_64 wheel and I was getting that this version is not compatible with the current platform. However, installing the local wheel worked perfectly and I can import dmlab2d just fine now.

@tkoeppe feel free to close this issue if you don't have any questions.

[tkoeppe]

Great! Could you check the SOABI tag reported by packaging again for the corrected Python setup?

[grig-guz]

I get this now:

PY_TAGS = struct(interpreter = "cp39", abi = "cp39", platform = "macosx_12_0_arm64")

[tkoeppe]

Looks like this was ultimately sorted out.