packaging-problems
packaging-problems copied to clipboard
pypa
Can wheels install arbitrary files
Problem description
Can a wheel install an arbitrary file? For example, /lib/systemd/system/foo.service on Linux.
No. See the wheel spec. All files in a wheel are installed to one of the distribution-specified purelib/platlib/headers/scripts/data locations.
Imagine I'm writing a new project in Python which wants to install manpages (/usr/share/man/man8/foo.8), init scripts (/etc/init.d/foo), and systemd units (/lib/systemd/system/foo.service).
How would I do that using the modern packaging systems?
As that's integrating with the Unix system, and isn't expected to work on (for example) Windows, I'd say you should be building an RPM or a DEB for it, not a wheel (which is a cross-platform format for Python libraries). I don't know much about Unix packaging, so I can't say much more than that, though, I'm afraid.
Sorry, I should have been clear. distutils is deprecated already and will be removed in 3.12, and setuptool's install command is deprecated too. What is the modern way to install Unix files?
Python's packaging ecosystem isn't designed to install Unix system files, that's what I'm trying to say. It never was - I'm not 100% sure how distutils/setuptools is relevant here (they aren't designed for installing Unix system files either).
Maybe I'm missing something, and historically people somehow used Python tools like distutils to install Unix system files. I don't think that was ever intended/supported use (I've heard a lot of Linux distro maintainers complaining that people should use the Linux management tools, not Python, to alter system-managed files) so there's no "modern way" of doing this, and if it was possible in the past then I guess sorry, it's no longer supported...
Interesting, thanks.
FWIW, there's plenty of software out there which subclasses the install command to write arbitrary files. https://weblog.christoph-egger.org/Installing_a_python_systemd_service_.html is an example from 2016.
Interesting, thanks.
FWIW, there's plenty of software out there which subclasses the
installcommand to write arbitrary files. https://weblog.christoph-egger.org/Installing_a_python_systemd_service_.html is an example from 2016.
Running python setup.py install is officially deprecated. In general, using python setup.py is considered a bad practice unless strictly necessary. Therefore there is not much point in subclassing install anymore...
FWIW, there's plenty of software out there which subclasses the install command to write arbitrary files.
Interesting. Presumably that software would need to be installed using setup.py (or pip) run as root? Which is the sort of thing that I was referring to when I mentioned distro maintainers complaining - running scripts from the internet as root is a hugely unsafe practice, and frankly I suspect that most people don't properly audit the code of packages they install like this.
So yes, I'd strongly recommend switching to building an rpm/deb for applications like this.
Packaging RPM/DEB is orthogonal to this issue. How would one generate the rpm/deb in the first place? The typical process is:
-
setup.py install --root=staging -
dpkg-deb --build staging
If I've a 100% python package that also wants to install an init script, is the only solution to use something like Meson/cmake/autotools and not use the Python packaging infrastructure at all?
To clarify about the root thing: in distribution-land the packager runs setup.py as a normal user, but to a staging directory. This directory is then packaged by the packaging tools. All of this can happen as non-root.
How would one generate the rpm/deb in the first place?
With config files and scripts specific to the target distro. Python packaging tools have taken years to redirect their aim from building all package kinds for all targets to only building python-specific formats. The tools don’t try to build deb or rpm themselves, but other projects (py2rpm, dh+pybuild, etc) are the glue between python formats and OS formats.
Distutils et al. allow you to install system files to /usr on a linux system -- this is fine for most purposes. Indeed, that is then staged and distutils' install command is used as part of the rpm/deb creation process.
You don't need to hack or override install commands for that! The distutils/setuptools data_files kwarg natively supports this. It was a first class distutils/setuptools feature! It was not an accident, it was definitely intended and supported because it took effort to get it to work correctly and document it.
However, IIRC this only works using setup.py install and pip refuses to handle them, but just sticks them in a directory inside site-packages.
Which is the sort of thing that I was referring to when I mentioned distro maintainers complaining
Actually, distro maintainers complain that setup.py install works perfectly for them, including both support for running as not-root, and correctly handling data_files. Meanwhile, distro maintainers hate pip because it does NOT do what distro maintainer want.
Given that setup.py install is deprecated and will not be un-deprecated, developers of Python software that performs Unixy integration are advised to migrate to a non-python build system.
If I've a 100% python package that also wants to install an init script, is the only solution to use something like Meson/cmake/autotools and not use the Python packaging infrastructure at all?
Yes.
For example, you can use Meson, and meson install will install init files to the init files directory, and python files to the python site-packages directory, and everything will work fine. You can also make that pip-installable by using a python PEP 517 backend hooking into Meson, such as https://pypi.org/project/mesonpep517/, but that will allow pip to turn the package into a wheel, then fail to install the init script. Users will still be able to install the program, but for full functionality they might wish to install it as a deb/rpm or via the ./configure && make && make install style methodology (except meson and ninja instead).
The final question is whether you want to register the package as installed to python's language-specific package manager, that is to say registering it to importlib.metadata as an installed distribution so that e.g. pip list can see it. This would require planned future improvements to Meson to have Meson generate a dist-info (rather than delegate that to third-party PEP517 backend bridges) so that it can be created during meson install. Unfortunately, I think we rather need tomllib as a standard library module, and particularly we have a stdlib-only rule that means we may not be able to support it until python 3.11 is broadly available enough to effectively drop support for python 3.10 (it is a terrific shame that a toml library never landed in python 3.10 due to bikeshedding).
If you're just installing a CLI tool then of course you may not care if there isn't a dist-info when installed with meson install.
Python packaging tools have taken years to redirect their aim from building all package kinds for all targets to only building python-specific formats. The tools don’t try to build deb or rpm themselves, but other projects (py2rpm, dh+pybuild, etc) are the glue between python formats and OS formats.
To be honest, neither Meson nor autotools tries to build deb or rpm themselves either, they just try to support generic system installation (and they kind of have to because they are multi-language build systems).
CMake does try to build deb or rpm itself, which personally I think is a bit of madness and isn't desirable from either a generic multi-language buildsystem perspective or a "try to package it for a linux distro" perspective.
The Python packaging tools started off closer to Meson/autotools than to CMake (technically there is bdist_rpm, but has anyone ever used it? how many people know it exists? and there is no bdist_deb), and have moved on to only providing wheels which install either pure python, compiled python, or python data, and don't get involved in anything that isn't strictly python. Python is officially out of the business of doing "integration", not "OS formats".
Ross and I both work on Yocto Project and I recently contributed the move to wheels and pyproject.toml/PEP-517 build-backends.
Canonical and Red Hat have a lot of legacy tools still depending on the distutils behavior (e.g. ufw). Given how much notice they have had to upgrade to setuptools/setup.py we shall see what happens when they are then asked to use meson or other tools to package their tools since Python will no longer do it. There are also legacy tools in the git.kernel.org tree.
For the distro maintainer workflow, we are at the mercy of the new packaging (I understand the reasons and am a fan) and the upstream projects we do not maintain, but need to package.
There probably is no perfect world.
You don't need to hack or override install commands for that! The distutils/setuptools
data_fileskwarg natively supports this. It was a first class distutils/setuptools feature! It was not an accident, it was definitely intended and supported because it took effort to get it to work correctly and document it.However, IIRC this only works using
setup.py installand pip refuses to handle them, but just sticks them in a directory inside site-packages.
Wait, I'm confused. data_files get installed into the install_data in distutils schemes -- which always refers to the base directory of the Python environment:
https://github.com/pypa/distutils/blob/e28e150c036e8b2bd55a37df13e836e098b5f37d/distutils/command/install.py#L40
What exactly are you saying works with setuptools/distutils directly, but not with wheels?
Files with absolute paths, such as /usr/share/applications (sys.prefix/share/applications will not work reliably).
Also extremely useful for /etc, or more precisely the configurable ${sysconfdir} directory.
As mentioned by the author of this ticket:
Imagine I'm writing a new project in Python which wants to install manpages (
/usr/share/man/man8/foo.8), init scripts (/etc/init.d/foo), and systemd units (/lib/systemd/system/foo.service).
There is:
- a
sysconfdirinit script file - a systemd unit that cannot serve any useful purpose installed to anywhere other than /usr regardless of Python environment
- a manual page which actually does work sometimes because the GNU
manprogram checks every entry in $PATH for../share/man/to find manpages corresponding to./prog. (BSD mandoc does not, so for cross platform support you do in the end need to install man pages to the system search path. Oh well!)
None of these reliably work if your view of the installation environment is "python environment only, nothing outside of sys.prefix".
While I am sympathetic to PyPA's desire to "get out of that game" and leave it to packaging tools such as meson, autotools, cmake, or Makefiles, it remains true that users who need this did in fact have something that served their needs in python setup.py install and do in fact need to switch to packaging tools such as meson, autotools, cmake, or Makefiles.
...
Also absolute paths are a hack, because what you actually want is, for example, Meson's ability to do this:
meson setup builddir --prefix='$HOME/.local" -Dpython.install_env=venv
And then you will be able to do a user install of $XDG_DATA_HOME compatible files to ~/.local/share/, while putting python modules into idk, ~/.venvs/app_foo/lib/python3.10/site-packages/, and everything plays nicely together. You'd want to have certain files be built files that apply the configured bindir, datadir, sysconfdir, python.get_path() etc. locations, of course. e.g. the program entry point wants to embed the shebang line of python.full_path(), ~/.local/share/applications/app_foo.desktop should probably specify an Exec key pointing at bindir, (but you may choose to configure that as python.get_path('scripts'), instead, and override the location of installed programs to that instead of bindir. But, then the program cannot be launched from the command line without first activating the venv).
Of course not everyone actually needs desktop files, and get_path('scripts') is a fair enough location for entry points if you start from the "I am in a venv" perspective and aren't using installer tools that support deep system integration.
For example building a wheel with meson, its good support for cross-language extension compiling, and a PEP517 backend, then having the wheel not try to do desktop files, and handle scripts via entry point metadata that adapts to a different install layout than the one it was built for.