Cannot Import Library on Mac

[NateSolon]

Description

When I tried to run the carrets_demo notebook, I was unable to import pydp.

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-2-9b364303ce89> in <module>
----> 1 import pydp as dp

/opt/anaconda3/lib/python3.7/site-packages/pydp/__init__.py in <module>
----> 1 from .pydp import *

ImportError: dlopen(/opt/anaconda3/lib/python3.7/site-packages/pydp/pydp.so, 2): no suitable image found.  Did find:
	/opt/anaconda3/lib/python3.7/site-packages/pydp/pydp.so: unknown file type, first eight bytes: 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x00
	/opt/anaconda3/lib/python3.7/site-packages/pydp/pydp.so: unknown file type, first eight bytes: 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x00

How to Reproduce

1. Clone PyDP repo.
2. Install PyDP with pip install python-dp. I also tried python -m pip install python-dp --force-reinstall.
3. Try to run import pydp as dp.

Expected Behavior

The library imports without error.

System Information

• OS: macOS
• OS Version: Mojave 10.14.6
• Language Version: Python 3.7.4
• Package Manager Version: Conda 4.8.3

[8bitmp3]

Hi @NateSolon, are you by any chance running PyDP in a separate virtual environment to have the packages localized to your PyDP projects?

To replicate your issue, I set up a separate new virtualenv on macOS with Python 3.7, activated it, then ran pip3 install python-dp. The library worked OK when running import pydp as pd and showed no errors.

[chinmayshah99]

Hi @NateSolon , Thanks for reporting this. Our CIs use the latest OSX for building and release. While we investigate and find a way to solve this, can you try building it from the source. What you'll need to run is:

- prereqs_mac.sh
- build_PyDP.sh
- make dist
- pip install dist/*.whl

Or, what you could do is directly use Google Collab or Docker.

[chinmayshah99]

The simplest solution is to move our release version to OSX 10.13. OSX is a forward-compatible OS and hence this should do the trick. One can try making this change here: https://github.com/OpenMined/PyDP/blob/dev/.github/workflows/pypipublish_osx.yml#L9. But this needs to be tested.

If someone can edit it on their folk and upload on the test.pypi server to demonstrate if this works, it would be great!

Note: In the PR please make sure that the package is tested on OSX 10.13, 10.14 and 10.15 and (or) share the corresponding test pypi project link. More on test pypi here: https://packaging.python.org/guides/using-testpypi/

[madhavajay]

Just some additional context to help with this ticket.

I believe all that is needed is to set the MACOSX_DEPLOYMENT_TARGET environment variable during build time. Like:

MACOSX_DEPLOYMENT_TARGET=10.13

I found this project which makes cross platform builds on CI easy: https://github.com/joerick/cibuildwheel

But, looking at the docs and source code it looks like the thing you need is that ENV flag from above: https://github.com/joerick/cibuildwheel/blob/7afdd40714f7567c595081500aafd66a58b31611/docs/cpp_standards.md

My guess is that xcode build tool chain compiles the MacOS libraries in a way that is backwards compatible to the minimum version set in that env flag.

It also looks like for C++17 you might want a minimum of 10.13 which is High Sierra:

To get C++17 support, Xcode 9.3+ is needed, requiring at least macOS 10.13 on the build machine. To use C++17 library features and link against the C++ runtime library, set MACOSX_DEPLOYMENT_TARGET to "10.13" or "10.14" (or higher) - macOS 10.13 offers partial C++17 support (e.g., the filesystem header is in experimental, offering #include <experimental/filesystem> instead of #include ); macOS 10.14 has full C++17 support.

However, if only C++17 compiler and standard template library (STL) features are used (not needing a C++17 runtime) it might be possible to set MACOSX_DEPLOYMENT_TARGET to a lower value, such as "10.9". To find out if this is the case, try compiling and running with a lower MACOSX_DEPLOYMENT_TARGET: if C++17 features are used that require a more recent deployment target, building the wheel should fail.

So it might be worth going even lower and just seeing what the minimum possible target version is.

It's also important to remember that the name of the python wheel indicates the system and architecture versions that it is compatible with so it might also need to be renamed to allow install: https://lepture.com/en/2014/python-on-a-hard-wheel

[chinmayshah99]

It's also important to remember that the name of the python wheel indicates the system and architecture versions that it is compatible with so it might also need to be renamed to allow install:

The OS name seems to be enough for the same. Consider the example of tf here: https://pypi.org/project/tensorflow/#files

[chinmayshah99]

Hi @NateSolon, is this still an issue?