Provide framework build on macOS?

[cbrnr]

I was wondering if providing a framework build on macOS would make sense. I'm not entirely sure how this works exactly, but a framework build seems to be necessary to access the macOS GUI reliably. I've tried this with my own projects (using PySide6 mostly), and even though my GUI app works fine in most cases with a non-framework build, sometimes there are no menus in the global menubar (it's really hard to reproduce). Furthermore, the official Python installers for macOS ship a framework build, so I'm guessing that there are reasons why this is a good idea in general. However, there might be downsides that I'm not aware of, but I think it would be good to discuss this rather sooner than later.

[glyph]

I tried to write up the vague constellation of reasons you might want this, and I'd love to see it. You don't need it for GUI access, but "reliably" is a good word: https://blog.glyph.im/2024/09/python-macos-framework-builds.html

[hoechenberger]

Super friendly bump – are there any news on that front, or is this planned?

[zanieb]

There's no progress here yet, it hasn't been a priority because there haven't been many concrete bugs related to it.

From @glyph's article, the following seems concerning:

Framework builds do have some small drawbacks. They tend to be larger, they can be a bit more annoying to relocate, they typically want to live in a location like /Library or ~/Library. You can move Python.framework into an application bundle according to certain rules, as any bundling tool for macOS will have to do, but it might not work in random filesystem locations. This may make managing really large number of Python versions more annoying.

I'm also not sure what value-prop there is here, since the Python org publishes framework build installers for macOS already, right?

[indygreg]

Historically I never needed a framework build of Python, so I never produced it.

To echo Zanie's thoughts, if you need a framework build, why not use the official Python.org framework builds?

So far nobody has articulated a value proposition for why this project should publish a framework build.

[glyph]

So far nobody has articulated a value proposition for why this project should publish a framework build.

I am sorry in advance for this brick of words; Pascal's apology applies :)

Here's my concern: Astral has built a great toolchain for building everything Python-related. The community tends to converge: we don't want to have 10 different ways to install Python depending on peculiar nuances of our projects. This means that recommendations, tutorials, workflows, standards, and just general ~vibes~ throughout the community will tend to simplify decisions like "how do I get Python installed?" Both the asker and the answerer of that question want the answer to be a simple statement, like uv python install, and not an hour-long consultation where you ask what the asker's needs are.

UI use-cases for Python are already relatively niche, and so a lot of people who get into using Python for desktop apps, or games, get into it because they were doing something else, and hey presto, Python magically does this other cool thing too. I think that we as a community should generally try to preserve this versatility, because that sort of pleasant surprise is great, and it's something that Python has more often than almost any other ecosystem except maybe JavaScript. Few languages can be used in so many different places.

If this project becomes the load-bearing way that everybody gets Python because it's the easiest answer for "how should I install Python", macs are popular with developers, and maybe even the official upstream build atrophies and eventually gets discontinued (python.org doesn't distribute binaries for Linux, if the audience for Mac builds shrinks to a very fringe use-case because everyone is using this project instead, perhaps they give up on that platform too), then various UX automation workflows won't work, and they won't work in a peculiar, half-broken way because UNIX-ish executables are peculiar and half-broken on macOS, then we end up in a situation where, in the rare case, when a developer veers out of their data-science lane to try out some PyPI package with some GUI or a game made with PyGame, something is wrong, the window doesn't take focus properly, the app icon is blank, or whatever, then folk wisdom develops "oh, you can't use Python for this, it's just kind of janky and broken. you should learn Swift instead" and the community loses a valuable direction for growth and cross-pollination.

Now, all of that doesn't have to be your project's problem, per se. Perhaps python.org should just fix their official builds to be relocatable, and then uv python could just install those instead. Or perhaps the overhead of grabbing the official build is not all that hard, and desktop things could keep doing it, and perhaps we could just build some conventions around detecting build-framework-ness where libraries could issue a clear warning rather than silently breaking in some way. Maybe that last one is the right one because some of these things also have code-signing and entitlements requirements that aren't fully resolved by frameworkifying build; this is just one step down an series of annoying rabbit holes.

But, all things being equal, if it isn't an unreasonable amount of work for you, I would personally love it if we could all stop worrying about the framework/non-framework distinction because everybody just stopped making non-framework builds and they all worked for every purpose without some people needing to learn and re-learn this unfortunate and extremely esoteric detail.

[cbrnr]

FWIW, have you considered working with the PSF on this one? I remember Brett Cannon talking about this topic on Mastodon (he made a poll if people would find it useful to have this option), but I cannot find that post anymore.

[hugovk]

@cbrnr This one? https://fosstodon.org/@brettcannon/113148479890148086

[zanieb]

FWIW, have you considered working with the PSF on this one? I remember Brett Cannon talking about this topic on Mastodon (he made a poll if people would find it useful to have this option), but I cannot find that post anymore.

Yep we're talking to Brett and other CPython developers.

[zanieb]

It seems possible that the solution is that uv should install macOS framework builds from python.org instead of adding framework builds here? Unsure. Is it feasible to make the framework builds relocatable?

[glyph]

Is it feasible to make the framework builds relocatable?

Build tools like py2app and briefcase have to rewrite framework builds to make them relocatable, because you can't dictate where a user puts an app bundle, and the framework needs to live in that bundle. Not sure how https://github.com/beeware/briefcase does it, but py2app uses "macholib" to rewrite the binary relative to the executable path, like this: https://github.com/ronaldoussoren/macholib/blob/e3c7ece6d39afbdcff75e8a2075798953918502d/macholib/MachOStandalone.py#L110-L115

I think that you could build a framework build the usual way, then do something similar as a post-build fixup? As I understand it the main relocatability issue is actually the Python binary itself, which links against the static /Library path. libpython has a self-reference which it seems in your builds is already the invalid /install/lib/libpython… so this wouldn't be any worse; just rewrite to a similar @executable_path with macholib and the result should be the same.

To be clear, I just know that it's possible to do because these tools do it, and I know that it's possible to do with a small reliable binary patch because I know that py2app does it. However, I don't know if the thing py2app produces is fully compliant with other constraints that this project might have (I suspect not, as it strips down many Python features, doesn't include most of the headers, etc, in service of just having the relatively minimal stuff you need at runtime rather than the full development install of Python)

[glyph]

It seems possible that the solution is that uv should install macOS framework builds from python.org instead of adding framework builds here?

The only issue with this is that it would require admin access to install the macOS framework builds, on account of requiring write access to /Library.

(Perhaps making the first-party framework builds properly relocatable at build time is actually just a thing Python.org could do, and then uv could install those without worry?)

[freakboy3742]

I've mentioned this to @zanieb in a separate forum, but this is a topic that I'm interested in helping to drive this from the CPython perspective (and it's nominally my "3.14 project", along with the iOS and Android "release artefact" analogs). See python/cpython#86680 for the details on that end.

As a tl;dr - It's not possible to make the "default" python installer relocatable because of the amount of existing code that is dependent on the /Library installed location. It's likely that CPython will end up with a different downloadable artefact for a relocatable Framework build, analogous to the Windows "embeddable" build. The major holdup on that front is that the release process for macOS builds needs some automation work, especially regarding the binary dependencies (xz, openssl, etc).

[glyph]

the amount of existing code that is dependent on the /Library installed location

is this code within CPython itself, within the build automation, or within the broader ecosystem?

[freakboy3742]

the amount of existing code that is dependent on the /Library installed location

is this code within CPython itself, within the build automation, or within the broader ecosystem?

AIUI, the broader ecosystem.

[glyph]

AIUI, the broader ecosystem.

Ah, so that separate downloadable artifact could just be the .framework directory itself, which could work fine standalone, but the .pkg installer (that installs e.g. an app shortcut for IDLE , the Python Launcher, etc) would be a superset of that and the other things it contains, and still hard-code /Library? In my mind "relocatable" was referring to just "will the framework itself break if you move it without patching it"

[freakboy3742]

Ah, so that separate downloadable artifact could just be the .framework directory itself, which could work fine standalone, but the .pkg installer (that installs e.g. an app shortcut for IDLE , the Python Launcher, etc) would be a superset of that and the other things it contains, and still hard-code /Library? In my mind "relocatable" was referring to just "will the framework itself break if you move it without patching it"

That's the idea, yes (or, at least, the idea that I'm working towards) - a "macOS Embedded" release artefact that is essentially an unversioned, relocatable Python.framework that could be dropped into an Xcode project, Briefcase stub executable, etc.

[zanieb]

@freakboy3742 do you think some of the build automation work here could be helpful to address those shortcomings?

[freakboy3742]

do you think some of the build automation work here could be helpful to address those shortcomings?

Possibly.

If there's an impediment to using this work as-is, I suspect it will be because of friction with the existing CPython build and release processes. I'm not completely up to speed on those at this point, so I'm not in a position to evaluate that right now; but I suspect I will be getting up to speed in the coming months.

[andythomas]

I got interested in this issue, because we would like to simplify the installation of our PyQt6 GUI app. However, while I intentionally tried to find something that would not work, I could not! Then, I included the linked clipboard issue and that worked [sic!] as well.

I hope, I got everything correctly. I used uv to install a standalone python. Then I ran the script as attached. The menu was changed and a cmd-v in a text editor revealed:

It does have some quirks, for example, the standalone version already replaced some of the occurrences (hide and quit) of Python in the menu, so only the top-most, bold one was replaced. Here one more sanity check:

╰─➤  ls -la /Users/thomasa/.venv/matr1x/bin/python
lrwxr-xr-x  1 thomasa  staff  87 Jul 26 17:16 /Users/thomasa/.venv/matr1x/bin/python -> /Users/thomasa/.local/share/uv/python/cpython-3.12.11-macos-aarch64-none/bin/python3.12

I do not know, if anything was changed, but for me it seems to work out of the box! The only obvious difference I see is between PySide and PyQt.

import sys

from PyQt6.QtWidgets import QApplication, QMainWindow, QLabel

# conditional import for Mac: Required to correctly set the app name and left-most menu
if sys.platform == "darwin":
    from AppKit import NSApplication
    from Foundation import NSBundle


def set_correct_mac_appname(name: str) -> None:
    """
    Set the correct app name on a Mac.

    Parameters
    ----------
    name : str
        The desired name of the application.
    """
    bundle = NSBundle.mainBundle()
    if bundle:
        info_dict = bundle.localizedInfoDictionary() or bundle.infoDictionary()
        info_dict["CFBundleName"] = name
    # Correct the menu
    app = NSApplication.sharedApplication()
    mainMenu = app.mainMenu()
    # Get left-most menu with app-specific items
    app_menu = mainMenu.itemAtIndex_(0).submenu()
    for i in range(app_menu.numberOfItems()):
        item = app_menu.itemAtIndex_(i)
        item.setTitle_(item.title().replace("Python", name))


def main():
    """Set the basic GUI parameters and run."""
    app = QApplication(sys.argv)
    title = "myNewTitle"
    if sys.platform == "darwin":
        set_correct_mac_appname(title)
    window = QMainWindow()
    window.setWindowTitle(title)
    window.setCentralWidget(QLabel("Hello, World!"))
    clipboard = QApplication.clipboard()
    clipboard.setText("I am in the clipboard.")
    window.show()
    sys.exit(app.exec())


if __name__ == "__main__":
    main()

[glyph]

Trivial examples are hard to find, but one that routinely shows the issue for me is that these broken non-bundles can't do things like request location-monitoring permissions. For example, a script just lying around on my hard drive:

import sys
import objc
import CoreLocation
import Foundation

class MyCLDelegate(Foundation.NSObject):

    def locationManager_didEnterRegion_(self, mgr, region):
        pass

    def locationManager_didExitRegion_(self, mgr, region):
        pass

    def locationManager_monitoringDidFailForRegion_withError_(self, mgr, region, err):
        pass

    def locationManager_didStartMonitoringForRegion_(self, mgr, region):
        pass

    @objc.typedSelector(b"v@:@i")
    def locationManager_didChangeAuthorizationStatus_(self, mgr, status):
        sys.stdout.write("Authorization Status Changed: {}\n".format(status))
        sys.stdout.flush()
        if status == CoreLocation.kCLAuthorizationStatusNotDetermined:
            mgr.requestWhenInUseAuthorization()

    def locationManager_didUpdateToLocation_fromLocation_(self, mgr, toloc,
                                                          fromloc):
        sys.stdout.write(u"Update:\n  From: {}\n  To: {}\n"
                         .format(fromloc, toloc))
        sys.stdout.flush()

    def locationManager_didUpdateLocations_(self, mgr, loc):
        ""
        print("diul", loc)

    def locationManager_didFailWithError_(self, mgr, err):
        sys.stdout.write(u"Error: {}\n".format(err))
        sys.stdout.flush()

#myCLManager.startMonitoringSignificantLocationChanges()

if __name__ == '__main__':
    from twisted.internet import cfreactor
    #import PyObjCTools.AppHelper
    reactor = cfreactor.install()#(runner=PyObjCTools.AppHelper.runEventLoop)
    myCLManager = CoreLocation.CLLocationManager.alloc().init()
    myCLDelegate = MyCLDelegate.alloc().init()
    myCLManager.setDelegate_(myCLDelegate)
    myCLManager.startUpdatingLocation()
    reactor.run()

Unfortunately, the exact behavior of mgr.requestWhenInUseAuthorization is quite dependent upon local system state, so it's not clear whether you'll get a prompt or not and I can't predict exactly what you'll see when you run this. But, with a framework build of Python, either you'll get a prompt or you will see a "Python" entry with an appropriate icon in the "Privacy & Security → Location Services" section of System Settings, and if you check it off you'll get a lat/long coordinate printed to the console. With a non-framework build, there is no icon nor any app name that the privacy settings pane can even display to the user, so you just get a blanket refusal.

The fully correct way to address this of course is to have your own app bundle that has its own bundle ID and code-signing and everything, so even Python's framework-build stub launcher application is a little bit a "works by accident" scenario (and certainly allowing every Python script you ever run access to your location is not the best security setup) but it does at least work.

[andythomas]

have your own app bundle that has its own bundle ID and code-signing and everything,

I believe it requires only the first half to be functional. It needs to be bundled, but does not need to be signed.

I made it work the following way: I added a shebang to your script that points to the pbs python (even in a venv), in my case:

#!/Users/thomasa/.venv/matr1x/bin/python3

EDIT: chmod 755 location.py

Then, I bundled it using a little wrapper I made to have proper Desktop integration, open a datafile by double clicking it, etc. I made a small package for this, you can find it via PyPi. If I name the scriptlocation.py

script2bundle -e location.py -d user -f Locationtest

Then, you can start it by double clicking the app called 'Locationtest' in ~/Applications This even works for editable python scripts (PEP660) which is kind of neat.

The wrapper makes the proper directory structure and the plist file for you. It is quite small in this case:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>CFBundleDisplayName</key>
	<string>Locationtest.app</string>
	<key>CFBundleExecutable</key>
	<string>location.py</string>
	<key>CFBundleIdentifier</key>
	<string>org.script2bundle.location.py</string>
	<key>CFBundlePackageType</key>
	<string>APPL</string>
</dict>
</plist>

I remember it being quite painful to figure it out ;)

Of course, now the outputs gets lost and it does not work if I launch it via a terminal --terminal in script2bundle, which is interesting I might look into that. But, if you can make a little Qt script that shows the same info in a QPlainTextEdit or similar, I am moderately confident that it might work. I would also be interesting to see if one can now really only allow this script to use the location info, but not other scripts.

[andythomas]

My understanding of the locations framework is not really good, but if I attempt to make a qt app, I see the following:

after pressing it, it is switched on in the System Settings, but I see

[andythomas]

I looked a little bit further into it. I believe there are at least 4 different cases that might or might not obfuscate the core of the issue.

1. python itself can be bundled (or not)
2. any python script (at least a GUI one) can also be bundled (or not). This is independent of 1!

Only bundles are visible in, for example, the 'system Settings'. 'Privacy Settings', 'Location Services'. This is achievable for scripts using 'uv install python' versions and the state of the switch is passed onto the app. As you can see in the above screenshot, the authorization status became '3', which seems to allow location services. However, the location itself I could still (so far) only see utilizing the system python the mac came with. So far so good (or bad ;).

This python is a 'framework python' as far as I understood this whole thread, it is bundled and it is signed (by apple if I am not mistaken). While I hope, I do not make a fool out of myself, but is it obvious that being a framework is the deciding factor? Maybe, bundled (and possibly signed) is the deciding factor?

[cbrnr]

Yep, you need a bundle in order to be a proper macOS app, but a framework is also a bundle.

[andythomas]

You guys have to stop me when this is pestering, but a self-signed. self-bundled app using a 'non-framework' python installed via uv does work for me!

EDIT: Unsigned and bundled works as well, but the WiFi has to be on ...

[glyph]

Then, I bundled it using a little wrapper I made to have proper Desktop integration, open a datafile by double clicking it, etc. I made a small package for this, you can find it via PyPi. If I name the scriptlocation.py

This is super interesting! I tried doing something like this once with my venvdotap package but it never really worked properly. I suspect that without team-ID code signing some of the other issues I keep noticing (like keychain access controls breaking) will not actually be fixed, but the ability to do development with an editable install would be interesting; I'm going to have to try it out. Thanks for sharing!

[glyph]

a framework is also a bundle

While this is technically true, I don't think a Framework can be a main bundle for a process, and thus show up in places like the system preferences panes under discussion; hence the presence of the Python Launcher app bundle within said framework.

I'd be interested to hear from @ronaldoussoren about whether the presence of the python framework bundle itself is necessary for e.g. py2app, or if its binary-rewriting logic is sufficient to frameworkify Python to embed it in the app bundles it generates.

Packaging python itself as a framework — independent of app-bundle issues — is also required for platform native development tools (i.e.: Xcode) to make sense of Python as a thing with headers and libraries inside it so that you can embed it into a non-Python application; if uv is to become The Way We Distribute Python, then this use-case probably also needs to be considered.

[cbrnr]

FWIW, I use uv-installed Python for my app bundle made with PyInstaller, and so far it has been working great! https://github.com/cbrnr/mnelab

[freakboy3742]

I'd be interested to hear from @ronaldoussoren about whether the presence of the python framework bundle itself is necessary for e.g. py2app, or if its binary-rewriting logic is sufficient to frameworkify Python to embed it in the app bundles it generates.

I can't speak with any authority on py2app's capabilities specifically - but on a purely technical level, it's entirely possible to take a non-framework build of a dynamic library, and convert it into a framework by doing the necessary binary rewriting and providing missing metadata and directory structures. The complications have much more to do with RPATH-related issues than they do to with whether a "framework build" was the explicit output of the original compilation process.

As for whether it's better for Python to be distributed by a framework, or for tools like py2app to engage in this sort of rewriting - I'd argue it's vastly preferable for Python to be distributed as a framework. As you note, it's a pre-requisite for Xcode to use the library, and it dramatically reduces the likelihood of each individual py2app-alike tool making an error in the conversion process.

(And for the record - the macOS support package that Briefcase uses is a framework; I'm hoping to get this build artefact upstreamed into CPython in the 3.15 timeframe).

[cbrnr]

I just came up with an almost reproducible example (meaning it cannot be reproduced every time, but if you run the example often enough, you will hit it eventually) in https://github.com/matplotlib/matplotlib/issues/30372.

So yes, I think the need for a framework build is real!