Support the use of `pip` in packaged apps

[freakboy3742]

pip is currently deliberately removed from Briefcase-packaged apps. This is on the theory that apps should be shipping with all the code they require, and installation triggered by the end-user shouldn't be necessary.

However, there is a limited subset of cases where it is necessary. #418 is one example where it has been requested; https://github.com/phildini/hera is another.

We should support this use case.

This will require:

• Ensuring that it is possible to include pip in the requires list and get a functioning version of pip
• Reconciling the discrepancy between the default site_packages and Briefcase's app_packages.
• Providing an entry point that can be invoked programatically. By design, pip does not have a stable programmatic API, by design, and PyPA has indicated that the have no intention of adding one (and have actively removed APIs that made pip programatically invocable
• Reconciling how to use pip on platforms where runtime installation may be prohibited by app store guidelines.

[freakboy3742]

@phildini If you have any insight on this from your experiences with Hera, I'd love to hear it.

[tlambert03]

I think you've taken the right approach by not including pip up-front. And it's already super-close to being able to support pip just by adding it to the requires list. In case it's useful for the discussion, I'll provide a little more background on our rationale for wanting to bundle pip, and what i found was necessary to make it work with briefcase.

rationale

We want to include pip because our app (napari) supports plugins, and we'd like to enable end-users to extend the functionality of the bundled app. In fact, it was this exact feature that led us to briefcase in the first place: I had been trying to figure out a good way to support plugins using pyinstaller bundles that was both easy for the developer and the end user. It was tough, to say the least. Even if we restrict ourselves to wheels for plugins, there is the issue of dependencies, and even if we bundle a handful of standard libraries (numpy, scipy, etc...) in our app, the dependency chain and version management is still tough. So we hoped that shipping a non-frozen environment that leverages pip would make all this easier, and in fact it has! It works great! (Side note: would be happy to hear about alternative strategies for plugins, if you have any). Not that this changes anything with regards to safety, but we never intended for the bundle end-user to use pip directly. We have a GUI that shows available plugins and lets the user install/uninstall them with a button... pip then runs in a subprocess.

getting it to work

Adding pip to the app requires list was sufficient to get a functioning version of pip (or at least I haven't found any issues yet). I don't think it is the responsibility of briefcase to provide a pip entry point, and I also don't think a programmatic API is necessary... we've been able to achieve everything we need using something like subprocess.run([sys.executable, '-m', 'pip', 'install', ...]). The main challenges have to do with path and environment wrangling.

macOS

by default, running pip in a subprocess will install apps to appname.app/Contents/Resources/Support/lib/python3x/site-packages. That works fine for us, but we do want to be able to immediately discover newly pip-installed plugins on the first run (without restarting the app), so the trick there was to make sure the (empty) site-packages folder is created (right after running briefcase create) and included in the package. That way, it will automatically be on sys.path from the start. That was it for mac!

windows

same trick as mac applies here: which is to make sure that the site-packages folder at app\src\python\Lib\site-packages. Two additional tricks were required. First, because the msi installer removes empty directories, I created not only the site-packages folder, but also an empty stub file there, right after running briefcase create and before running build. As mentioned in #418, it was also necessary to add the line .\\\\Lib\\\\site-packages\n to the app\src\python\python3x._pth file ... otherwise the sys.executable subprocess running pip install would not see the site-packages folder when determining already installed packages.

linux

because the AppImage format is read-only, we can't pip install directly to the bundle, so here we add a designated path (something like appdirs.user_data_dir()) as a prefix in pip: subprocess.run([sys.executable, '-m, 'pip', 'install', '--no-warn-script-location', '--prefix', 'some/dir', ...]) and then, make sure that the corresponding site-packages directory underneath "some/dir" is added to sys.path in your actual program, and also added to PYTHONPATH environment variable when running pip uninstall.

With those tricks, I'm pretty happy with our bundled plugin installer/uninstaller. So I guess from the perspective of briefcase, if you wanted to, you could make sure that the lib/.../site-packages folder was included in the bundle if pip is in the list of requires (and for windows, update the _pth file). It does mean that newly installed packages don't go into briefase's app_packages, but that seems like it would be harder since that can't currently be used directly as a pip --prefix.

[freakboy3742]

Thanks for those details @tlambert03.

The Linux case in particular makes me wonder if we need to take a slightly different approach, and leverage user-config directories - i.e., make sure the briefcase will look in ~/.somewhere / ~/Library/Application Support / C:\Users\me\Application Data (or whatever the platform appropriate user-configuration folder would be) as an extension of the system path. Although it's only absolutely required for Linux, a "global installation" on MacOS or Windows could also potentially have permissions issues, and having plugins be user-specific might also be necessary/useful.

We could also potentially ship a "briefcase pip" which would be a shim that does the subprocess.run() step, wrapped in a nice API, writing to the platform appropriate directory, etc.

[paulproteus]

Those are basically the same things that @phildini and I worked on with Hera, as I recall. I'm glad you two are having this conversation, and also discovered the same things that @phildini and I did in Hera! (Seemingly not pushed to GitHub yet -- https://github.com/phildini/hera .) We had a similar use-case of letting the user install some extensions.

@tlambert03 my suggestion (up to you! I'm just some rando) is that this would make a great PyPI package called e.g. appdirs-pip and it would have two functions:

configure_sys_path() -- This would add the right directory to sys.path. Perhaps it optionally takes a Briefcase app name, so it picks the right user-config directory for the specific app.

run_pip() -- This would take pip argv, as well as the app name.

It's the kind of thing that's useful for any Linux app running pip, and even if you migrate away from briefcase one day, you could keep using the same code, and even non-briefcase users might want it.

I'm -0 on adding anything specific to briefcase about this, beyond possibly docs, just because IMHO briefcase's scope is already big, but that's for @freakboy3742 and others to decide.

P.S. Anti-dependency extremists might say that this could be a blog post with some copy-pastable code. I like the idea of it living on PyPI, personally.

[freakboy3742]

@paulproteus Interesting; I would have considered it Briefcase's role to set up the path for the executing PYTHONPATH for the app, encompassing platform-appropriate paths for:

• standard library
• app dependencies
• the app code itself
• global plugins
• user plugins

rather than expecting the code to self-modify sys.path at runtime. Do you have any particular reason for for favouring in-process sys.path modification?

[phildini]

Latest hera work has been committed now: https://github.com/phildini/hera/commit/b33be2a4c5b4edec6aea5953785aa3ae25f0b4a2

[phildini]

@freakboy3742 I generally agree that the briefcase / toga philosophy is "make it easy for developers to do the right thing", I think to @paulproteus point, what's the interface that makes that happen?

What interface would make sense for distinguishing between the cases you listed in such a way that the developer had confidence they were installing to the right place?

Unless briefcase is going to proxy pip altogether, it seems like the developer would need to know some interesting details about briefcase's built-in path mucking to know if they were installing a plugin to the app or the global library space or the user library space.

[freakboy3742]

@phildini Agreed; although I think there's also a question about areas of responsibility.

The API that @paulproteus suggested leans towards a generic "pip wrapper" API that make's it the app author's responsibility to ensure that sys.path is correct by manipulating it at runtime.

I'd argue for a different division of responsibilities:

1. Toga (and a user's app code using Toga) shouldn't care at all about PYTHONPATH/sys.path. It should find code on whatever paths it is given at runtime.
2. Briefcase should package apps in an environment that guarantees that the "5 paths" described previously will be on sys.path at runtime (by PYTHONPATH/._pth manipulation, as required on any given platform, to platform appropriate locations)
3. We write a "briefcase plugin installer" wrapper that wraps standard pip, installing to one of the two "user accessible" known briefcase locations (global plugin or user plugin). The API is essentially install_package("mypackage==1.2.3", global=True).
4. If another packaging tool wants to support a different set of plugin paths, they define their own plugin installer.

An alternative framing of 3 and 4 would be: 3. A generic "invocable pip" wrapper that exposes the --target argument as an option, and
4. A "briefcase-compatible appdirs" package that defines the global and local plugin paths. An alternative packaging tool would define their own appdirs.

I have a moderate preference for the former (a "briefcase plugin installer" wrapper) on the basis that plugin handling is intimately linked with the packaging tool, so we shouldn't pretend otherwise. That said, the latter framing is really just pip-api, with the paths being a known path on top of the user_config_dir/site_config_dir of appdirs (or toga.paths, for that matter); so going down the path of yet another package might be overkill.

[phildini]

To point (1): Should "user packages" and "global packages" and "app packages" all be combined? Is there value in wanting to separate? That seems to me like part of the challenge.

But if we assume that the metaphor is "the app is always acting as the current user, and should prefer user packages, then global, then app, in that strict order" then what you're saying makes a ton of sense.

[freakboy3742]

The Linux case demonstrates that user, global and app packages can't be combined. Once a Linux AppImage is packaged, app_packages can't be modified, so we need to have an "external" location to install.

I'd argue global and user packages shouldn't be combined either - just because you want plugin X installed doesn't mean I do; unless the sysadmin has said that all users must have the plugin.

There's an argument to be had about whether app and app_packages could/should be merged. They certainly could be - the only real advantage is that by keeping the two "types" of packages separated, it makes it slightly easier to reset just the app or the app packages, and in the case of the app overloading the name of a package in app_packages it ensures that the app always has primacy (rather than being installation order dependent). However, that's a fairly minor benefit.

There's another argument over whether app and app_packages should be abandoned entirely, in favour of the default site_packages location. The primary argument here is that the default site_packages location is inside the support package, which makes transparent upgrading of the support package without upgrading the app and app packages much more difficult.