Add an option to put libraries into a library sub directory

[mozbugbox]

trafficstars

Try to do issue #1048, #2468.

Add a option --lib-subdir libs option to pyi-makespec.

Known Issues:

• Only tested on Linux
• Cannot put the dir of a module into the library subdir. When numpy/ was put into the lib subdir, it cannot be properly imported. Don't know why.
• libpythonXX.so and base_library.zip are still in the home dir.
• the DYLD_LIBRARY_PATH is not touched on MacOS. It seemed there are some special way to treat DLL loading on Mac.

So basically, most dynamic modules are put into a lib sub directory. packages are left to the home directory.

[bwoodsend]

Woah hoah, that's a bold one!

Only tested on Linux

Well we have CI/CD for that. Github have added this approve and run feature which means that, as you're a first time contributor, every time you want to run CI I have to click yes do it. Normally this is useful but I think it's going to be a pain here. You might find it easier to enable actions on your own fork then temporarily create a pull request to your fork's develop branch. That way, you CI will run on your fork without having to wait for me to tell it that it's ok to run.

Cannot put the dir of a module into the library subdir. When numpy/ was put into the lib subdir, it cannot be properly imported. Don't know why.

Is this specifically numpy or does it not work for even really simple libraries containing the odd data file? You'd need to add the subdirectory to sys.path at runtime (which could be done in here) but other than that it should be moveable.

the DYLD_LIBRARY_PATH is not touched on MacOS. It seemed there are some special way to treat DLL loading on Mac.

They are treated weirdly but I'd rather that they weren't. Every DLL gets monkey-patched in a way I can't remember (cc @rokm help me out here) and never really understood anyway.

[rokm]

This is indeed a bold move (especially for the first contribution), but the situation with shared libraries in PyInstaller is rather complicated...

---

So basically, most dynamic modules are put into a lib sub directory. packages are left to the home directory.

That's going to wreak havoc on macOS, because library paths of all shared libraries AND extensions (including those that remain in the package directories) are rewritten to be relative to the application root.

EDIT: or rather, if A (shared library or extension, potentially located anywhere) is linked against B (shared library collected in the application's root), A's link path to B is modified so that it points to B in application's root (_MEIPASS) but as relative path to the A's location.

That is indeed really ugly and inconvenient, but I think it was designed this way to deal with shared libraries that get pulled from non-package directories in site-packages (e.g., site-packages/blspy.libs). So the path rewriting code will need to be adjusted for the new location, but this can be done only if all libraries (except Python) are moved to the new location. (As a side note, we should eventually upgrade this path rewriting procedure with an ability to infer actual library locations from the TOC - then we could break free from the assumption that everything is in _MEIPASS, or in some other pre-determined location).

---

I suspect it will also break ctypes and possibly cffi in frozen application, because those also have special provisions for looking for dynamic libraries in the _MEIPASS .

And I wouldn't be surprised if there were other obscure corners of PyInstaller that depend on shared libraries being in the application's root directory...

---

I'm not particularly fond of having this as an option, either. Having two possible deployment layouts with potentially subtly different behaviors will be a nightmare to support. So either we keep collecting stuff into _MEIPASS, or we decide to revamp that, and live with it. Point in the case: the current batch of tests on CI is probably running without the new relocation, so no new issues will be uncovered.

---

All in all, I'd prefer if this started smaller. Perhaps with trying to relocate just the .pyd extensions on Windows. On linux and macOS, we already divert the python extensions coming from python's lib-dynload directory into _MEIPASS/lib-dynload subdirectory. Perhaps something similar could be done for .pyd extensions coming from python's DLLs directory? (Maybe move them into lib-dynload as well?)

[mozbugbox]

The code strictly follows the steps described in issue #1048. So the changes are straight forward and small. It just did:

• Add a lib_subdir option to EXE() and COLLECT(). In the COLLECT.assemble(), prepend the sub_dir to tofnm filename if the type of the TOC is either EXTENSION or BINARY.
• In the bootloader, Adjust LD_LIBRARY_PATH and sys.path if the lib_subdir is set.
• Add the cmdline option to the pyi-makespec and the related template file.

That's all about it.

As for the _MEIPASS. I think it's a one-file thing. The intend of this patch is for cleanup the home directory for the one-folder mode. Does _MEIPASS has much impact on one-folder mode?

The option does not have to take effect on Mac if it is too much a hassle.

[mozbugbox]

And for the problem with relocating directories like numpy/ extension package, I've traced it to the FrozenImporter.exec_module() in pyimod03_importers.py: https://github.com/pyinstaller/pyinstaller/blob/e5d69327ef602180bdbe443964d0a64b30408dea/PyInstaller/loader/pyimod03_importers.py#L525-L539

That __path__ thing is passed from the parent module to find_spec() of meta_path finders to find submodules. Since python modules are frozen and given the path of dirname(module.__file__), which for numpy means the home/module_path directory. Therefore the submodules of numpy are always searched under the home directory.

Change the __path__ to include home/lib_subdir/module_path makes relocate packages to a sub dir work alright:

            module_dir = pyi_os_path.os_path_dirname(module.__file__)
            path = pyi_os_path.os_path_join(sys.prefix, "libs")
            path = pyi_os_path.os_path_join(path, module_dir[len(sys.prefix)+1:])
            module.__path__ = [module_dir, path]

The "libs" is hard coded here. Is there anyway in the python bootloader part to fetch EXE OPTIONS? like the pyi_arch_get_option() rom the binary bootloader? Else the pyi-lib-subdir has to be passed through sys module?

[rokm]

The "libs" is hard coded here. Is there anyway in the python bootloader part to fetch EXE OPTIONS? like the pyi_arch_get_option() rom the binary bootloader? Else the pyi-lib-subdir has to be passed through sys module?

This would probably be best passed through sys module, same as how it is done for sys._MEIPASS (which denotes application's top-level directory in both onedir and onefile applications).

[lillithkt]

Is there any update on the progress of this?