[Question] Library not loaded on macOS and Homebrew Python

[SEngelnkemper]

I am trying to import an obfuscated package under macOS with Python3.10 installed via Homebrew. I tested this on multiple system (aarch64 and x86_64) and constantly get the error

pytransform/darwin_x86_64/pytransform.so, 0x0002): Library not loaded: '@rpath/Frameworks/Python.framework/Versions/3.10/Python'

As the pytransform.so seems to seek for libpython in the standard system python installation path. As a workaround, I tried to add the location via install_name_tool and -rpath to the pytransform.so, but this destroys the signature, so the pytransform.so is not trusted anymore.

Here is the obfuscation command I use:

pyarmor obfuscate --platform darwin.aarch64 --platform darwin.x86_64 --platform linux.x86_64 --platform windows.x86_64 --advanced 2 --exclude tests,examples,docs,setup.py --with-license licenses/r001/license.lic --recursive --output ./temp ./input

Is there a way to make the pytransform libraries find libpython also when installed via homebrew without too much modifications of the target system?

[jondy]

Please search install_name_tool in the closed issues to find the solution.

[SEngelnkemper]

As I mentioned, I know about the general issue with libpython and the workaround to use install_name_tool to add/modify paths to the pytransform.so.

However, this seems to destroy the signature and the pytransform.so is not trusted anymore (at least on aarch64 systems). Also this workaround has the downside to be not universal and needs some installation script on the user system to figure out the actual location of libpython and add it to the pytransform.so.

Can the pytransform libraries be modified on your side in a way that they figure out the libpython path dynamically rather than expecting the libpython to be in the fixed path?

[jondy]

Generally it could be fixed by adding an extra rpath something like @loader_path/../.., but there is no latest home-brew installed in my machine, I don't know how many ../ are required in this extra rpath. Could you test it? Just run

install_name_tool -add_rpath @loader_path/../.. dist/pytransform.so

If it doesn't work, try more ../ to make sure @rpath/Frameworks/Python.framework/Versions/3.10/Python exists.

install_name_tool -add_rpath @loader_path/../../.. dist/pytransform.so

[SEngelnkemper]

Thanks for the hint, but the problem is, that the Homebrew installation is not always in a fixed path. I have two test systems (x86_64 and aarch64) and these two already have different locations of the Homebrew installation, i.e. libpython.

Also, as mentioned, adding an rpath to the library invalidates the signature, so the interpreter gets killed on aarch64 systems when the library is loaded.

[jondy]

It's not the problem, actually there are many rpath in the current pytransform.so,

-add_rpath @loader_path/.. -add_rpath /System/Library -add_rpath /Library

It seems @loader_path/.. is just for old HomeBrew, but Homebrew changes its default location recently.

So adding more rpath for different Homebrew, then sign it.

[SEngelnkemper]

I didn't find a good workaround for this, yet.

1. It's not possible to add all the possible paths to the pytransform.so, especially when also covering pyenv installations, where the corresponding path is within the user folder, which is of course not universal.
2. On darwin_aarch64, adding paths doesn't work at all. The interpreter gets killed even if I codesign the library after adding the paths (with codesign -s - pytransform.so). Btw, after adding the paths the codesign command still states pytransform.so: is already signed.

Any idea how to get around this ?

[jondy]

The only solution what I known is to use @loader_path for random python installations, only if python interpreter path is fixed relative to Frameworks path.

[SEngelnkemper]

@loader_path always points to the location of the pytransform.so, so how can this be universal, if the package location is non-universal?

Edit: As @loader_path always points to the location of the pytransform.so (at least on my system), I tried @executable_path instead. With that, I was able to solve this on at least one system for now (with @executable_path/../../../../../../../.. as an additional rpath for a Homebrew installation). Do you have a macOS test system to verify the @loader_path vs. @executable_path behavior?

Also, do you know the root cause of the 2. issue I mentioned above ?

[SEngelnkemper]

I successfully tested the fix with @executable_path/../../../../../../../.. now on two systems, also for pyenv installations. Maybe this should be also integrated to your core libraries?

[jondy]

Thinks for your efforts, and @executable_path really gives me a new idea. Here are sample commands to replace the runtime file pytransform.so:

cd dist/
install_name_tool -change @rpath/Frameworks/Python.framework/Versions/3.9/Python @rpath/Python -add_rpath @executable_path/../../../.. pytransform.so
codesign -f -s - pytransform.so

So only one rpath @executable_path/../../../.., combine it with @rpath/Python, not @rpath/Frameworks/xxxxxxx/Python. It works in my MacOs. Does it work in your MacOs and pyenv environments?

[AmberSahdev]

@SEngelnkemper sorry I didn't follow your solution, what did you end up doing to resolve the library not loaded error? Thanks!