Add support for converting pyproject.toml-based Python3 packages.

[amdei]

TL;DR: It works! (With some limitations at the moment though, as this is work in progress currently)

git clone ....
cd fpm
python3 -m pip install importlib_metadata
ruby bin/fpm --verbose --log debug --debug --debug-workspace -s python -t deb --python-install-lib=/usr/local/lib/python3.9/dist-packages/ --python-package-name-prefix python3 --python-bin=python3  typing_extensions

Limitations:

1. Does not support python2 in any way (obviously)

Temporary limitations (TODOs):

1. Do not support automatic dependencies extraction to the moment
2. Do not support license extraction to the moment
3. Checked only on Debian 11 amd64 (arm64 to go) so far
4. Not sure about applicability to 'native' packages
5. A lot of @todo's in code
6. Code quality is poor
7. Code-style is poor

I had no previous experience with ruby, so any suggestions/comments/guidelines are welcome.

@ObjatieGroba - could you please check Python part of code, and give your suggestions on how to improve it?

Fixes #1780 Fixes #1873 Fixes #1860 Fixes magma/magma#13847

[amdei]

After some reflections I've had a second thought.

1. Standard importlib.metadata is better that backported importlib_metadata
2. importlib.metadata is still not good enough to obtain all required metadata of not-yet-installed python package
3. I don't want to build wheel out of downloaded python package
4. But seems there is no solution without building wheel out of python package prior to converting it to some other format
5. It is possible and it is easy to get ALL required metadata via wheel API from pkginfo package

Going to revise my solution accordingly. Stay tuned.

[amdei]

It seems to be working a little bit better now. At least in terms of metadata retrieval.

Things to do:

• [x] 1. Apply proper arch attribute to generated deb-package. At the moment all toml-based packages has Architecture: all;
• [x] 2. Instruct wheel to not download dependencies for build package wheel;
• [x] 3. For some packages fpm now silently dies with zero exit code. Need to get it work again;
• [x] 4. Deal with packages dependencies. Not easy at the first glance.
• [x] 5. Speed-up execution on processing toml packages. Now it 3 times slower that setup.py-based;
• [ ] 6. Perform code de-duplication. As now it contains significant amount of redundant fragments.
• [ ] 7. Cleanup code.
• [ ] 8. File some issues to pip, wheel, importlib, etc.
• [ ] 9. Obey --[no-]python-obey-requirements-txt fpm's option

Any suggestions/comments/guidelines are still welcome. As well as package samples, where things go wrong.

[jordansissel]

Thanks for your work on this! I’ll take a look as soon as I can :)

[amdei]

Bottom line:

1. Please update python wheel to latest version before trying anything: python3 -m pip install -U wheel - there was a bug, affected fpm, fixed only very recently.
2. It is slow. Very slow. Significantly slower that setup.py-based packaging;
3. Wasn't able to made it with architecture - it looks like there is simply no way to get arch neither from *.whl file nor from sdist via any API. But it contains in *.whl. But I'm not brave enough to parse it in that way (yet).
4. There is no alternatives found for setup.py's --install-lib, --install-data, --install-scripts and build_scripts --executable. So all fpm's --python-install-* options will not work (and currently silently ignored).
5. Staging with pip is broken. For example see here: https://github.com/pypa/pip/issues/8799 Or it is intended for something else.
6. Documentation for pip install is rather short: https://docs.python.org/3/installing/index.html#installing-index Significantly shorter than one would expect.

[amdei]

After a while, I hack the Wheel metadata to get information if package is pure or not. https://peps.python.org/pep-0491/ As it seems to be impossible (at least yet) via pkginfo.Wheel API: https://bugs.launchpad.net/pkginfo/+bug/2015657

[amdei]

It reached "works for me" stage. Let's collect feedback before starting to polish things..

[JordanStopford]

Amy update on this? A lot of packages no longer ship setup.py files so if we need to use fpm then we're stuck on old versions

[cwegener]

I don't understand what this does? It still requires a legacy setup.py file to be generated by the python project that is supposed to be converted. However, pretty much all PEP-517 build backends disable the support for generating setup.py legacy files by default, so they will not be able to be used by fpm. (some backends don't even offer an option at all to generate legacy setup.py files)

And if you have a source python project that has the option to generate a legeacy setup.py enabled, then such a project can already be used as a source in fpm without this change.

I am a bit confused what this change is supposed to do.

What is the intention of this change?

[amdei]

It still requires a legacy setup.py file

No, it doesn't.

There is an option to use setup.py file, if it presented. But if there is pyproject.toml file - it will be used instead.

[cwegener]

It still requires a legacy setup.py file

No, it doesn't.

There is an option to use setup.py file, if it presented. But if there is pyproject.toml file - it will be used instead.

Thanks for the clarification. I'll need to read through the patch again then. :grin:

[amdei]

Although in this MR I made attempt to solve the issue via pip wheel, not via build module from PyPA, as suggested in https://github.com/jordansissel/fpm/issues/2040.

[cwegener]

Although in this MR I made attempt to solve the issue via pip wheel, not via build module from PyPA, as suggested in #2040.

That's excellent. pip wheel will actually just invoke the Python build module which takes care of performing the right build process (legacy setup.py vs new PEP-517 build backends)

I will need to test out your PR and report the results.

Specifically, I will need to test with this PEP-517 project ... https://github.com/openai/openai-python

[gmabey]

Just an enthusiastic vote for more eyeballs on this merge request ... I would love :heart: to see the project catch up to this shift in the python community.

[cwegener]

I will need to test out your PR and report the results.

Quick feedback from my initial testing of this patch:

Testing with

• openai package from PyPI (which uses hatchling as the PEP 517 build backend)
• Testing on Ubuntu 22.04 with built-in Python3.10

My main issue that I'm looking into at the moment is the boot-strapping of the PEP 517 build environments for pip wheel (and to a lesser degree pip download ... which is a separate story).

I think the main concern so far is the new dependency on the pkginfo Python library. E.g. on Ubuntu 22.04, the version of python3-pkginfo is a bit older and does not yet have support for version 2.3 Metadata file formats.

But a lot of wheels will now write 2.3 Metadata file formats.

Therefore, injecting or vendoring an up-to-date version of pkginfo is important.

[amdei]

My main issue that I'm looking into at the moment is the boot-strapping of the PEP 517 build environments for pip wheel

@cwegener could you please elaborate, what is your issue with openai package building? Is it difficulty with determining build dependencies? Or something else?

[cwegener]

My main issue that I'm looking into at the moment is the boot-strapping of the PEP 517 build environments for pip wheel

@cwegener could you please elaborate, what is your issue with openai package building? Is it difficulty with determining build dependencies? Or something else?

I think it's actually a different issue that is unrelated to the PEP-517 build support.

The issue is that all dependencies are being built from source due to a change introduced here: https://github.com/jordansissel/fpm/commit/77e0cf7a768bdd883409d58e0549691a7d1d762f

The --no-build :all: option forces all dependencies to be built from source, even though this is unnecessary.

The dependencies will be thrown away at the end. They don't end up in the released package that fpm creates. So there is really no benefit in building them from source.

Nevertheless, I still have not managed to build an openai package, but I think that might just be an issue on my side.

[amdei]

Nevertheless, I still have not managed to build an openai package, but I think that might just be an issue on my side.

Just checked - built seamlessly.

Could you please try to build it with more verbose output and share the result? Something like this: ./bin/fpm --verbose --log debug --debug --debug-workspace -s python -t deb --python-bin=python3 --python-package-name-prefix python3 --python-install-lib=/usr/local/lib/python3.9/dist-packages/ openai

(you may want to adjust --python-install-lib option to reflect your python version)

[cwegener]

I had another look. And I think I figured out where the issue with this approach is. pip download will perform unwanted build steps in order to generate metadata. Those unwanted build steps will fail quite often.

Example package: jiter

python3 -m pip download --no-clean --no-deps --no-binary :all: -d . jiter

This will try to immediately build a wheel from the jiter sdist, which will fail due to the build backend (maturin) not being present on my system.

This is a pip problem as per the following references.

References: https://github.com/pypa/pip/issues/7995 https://github.com/pypa/pip/issues/1884