pyinstaller

pyinstaller copied to clipboard

Is your feature request related to a problem? Please describe.

For macOS framework builds, the Qt hook copies various Qt frameworks to two different locations, so that they are accessible from the QtWebEngineProcess:

https://github.com/pyinstaller/pyinstaller/blob/63f30ef57a37dfc088672ec04169b9da3e92ce9b/PyInstaller/utils/hooks/qt.py#L726-L762

This however blows up the total app bundle a lot - with my project with PyQt5, from 400 MB (with those files replaced by symlinks, see below) to 630 MB.

Describe the solution you'd like

Instead of copying those files, PyInstaller could symlink them. I didn't figure out how to accomplish this, however, because the hook only seems to add them to datas, and I don't see a way for a hook to modify the resulting bundle.

Describe alternatives you've considered

Manually patching the resulting .app: qutebrowser's build_release.py.

This has served me well for a few years, but recently introduced the issue mentioned below, and doesn't quite seem to work yet for Qt 6.

Additional context

There seems to be somewhat related discussion here:

• #6612

I suspect my manual modifying to cause this:

• https://github.com/qutebrowser/qutebrowser/issues/6771

Hmm, if PyInstaller's Bundle() indexed all the files it collects, grouped duplicates and then replaced all but the first of each group with symlinks to the first, we could probably solve both this problem and the more general problem of when a library gets collected as both a data file and a .so. That kinda sounds like it might actually work... :thinking:

...at least on macOS and Linux, I suppose. No idea how the situation looks for symlinks on Windows. I'm also not sure what else it would break, I don't think everything handling files can necessarily handle symlinks as well.

For a more fine-grained approach, however, I suppose a hook would need a way to "postprocess" the generated bundle. Also see #6612 which requires similar post-processing I suppose, and isn't solvable by a "link duplicates together" approach.

The QtWebEngine problem on macOS is a bit more specific; we should not be breaking up the .framework bundle as this breaks the QtWebEngineProcess helper, and is the reason why we need to collect the .framework bundles as additional copies (and as data) in the first place.

The origin location of the dylib matters in this case, too; it should be the one in the .framework bundle, and the copy in _MEIPASS could be replaced with symlink to it. The problem is, the copy in _MEIPASS is the result of binary analysis and is unavoidable as long as library path rewriting is there and we keep collecting shared libraries in _MEIPASS because of it...

But even if we kept the .framework bundles intact, they would probably still be causing problems with notarization, because they should probably be moved to Contents/Frameworks and symlinked back to their original location.

@rokm You appear to be saying that no possible setup involving QtWebEngine and notarization can work?

No, it should be possible with sufficient amount of file relocation and symlinking to get around the dot-in-name problem; at least according to the feedback we got from users who were asking about help with notarization.

---

The point I was trying to make is that the duplication in this case was very deliberate and was result of how PyInstaller currently works (binary analysis collecting shared libs into _MEIPASS). So I very much dislike the idea of introducing additional de-duplication post-processing, because it is basically applying a work-around on top of a work-around instead of addressing the core problem... this particular problem (if we forget about notarization for a moment) needs to be fixed by leaving shared libs in their original places (and revising path rewriting behavior to account for that, or creating symlinks to _MEIPASS to account for the current behavior and assumptions). Doing so for example also makes QtWebEngine work on macOS in onefile builds.

Doing the right thing and leaving the shared libraries in their original directory structure, though, will likely run afoul of notarization, due to no-dot-in-name requirements (and wheels using .libs, .dylibs, etc., to store their dylibs in). That's not specific to Qt and QtWebEngine, although there are several sub-directories in addition to .framework bundles that are likely problematic due to dot in their name.

No idea how the situation looks for symlinks on Windows.

If you don't have Developer Mode turned on, you can't create symlinks if you aren't an admin. In other words, try and avoid them if it involves the symlinks leaving your machine.