pythran in Debian

[saraedum]

Currently pythran does not seem to be packaged for Debian. Would you mind me trying to package it for Debian?

[serge-sans-paille]

I 100% don't mind! I used to maintain a private debian repo but lack time, so go ahead :-) I'd be curious of the package rules, do no hesitate to share them!

[saraedum]

I guess #733 is essentially a dependency for this.

[saraedum]

So what's your relation to boost.simd? It is a hard dependency I suppose? It appears to be somewhat unclear where that project is heading and I am not sure whether it makes sense to include it into Debian at the moment.

[serge-sans-paille]

Neither am I. But it is still a hard dependency and there is no short term workaround...

[serge-sans-paille]

So my guess is that you should probably bundle boost.simd within pythran and not as an external dependency?

[saraedum]

Sorry that it took me a bit to look into this again. I think that putting parts or all of third_party/ into a Debian package would not be acceptable. So I would have to drop that entire directory for Debian. I understand that a standard boost would do for all but boost/simd, right? It appears that your boost/simd is not the (old) upstream release but has been patched by you as well? Do you know if somebody is maintaining an unofficial but maintained fork of boost.simd that could be packaged for Debian? It's going to be hard to get something into Debian if it's already abandoned by upstream. And…it really appears to me that upstream has given up on the open-source version of boost.simd.

[serge-sans-paille]

standard boost would do for all but boost/simd, right?

yes

It's going to be hard to get something into Debian if it's already abandoned by upstream

agreed. I could remove that dependency and use xsimd instead from @quantstack, but that would delay the packaging 'tll the end of year.

My boost.simd version is patched, why cannot you consider it as an implementation detail of pythran?

[saraedum]

My boost.simd version is patched, why cannot you consider it as an implementation detail of pythran?

I don't think Debian is going to accept this. From https://wiki.debian.org/UpstreamGuide#No_inclusion_of_third_party_code

Please do not include other code (like libraries) or data that are also shipped separately inside your source archive, or if you do, please make sure they can be reliably ignored. Instead of shipping third party libraries you should rather make sure your program will be link nicely against recent versions of these libraries. If a security issue is found in one of the bundled packages, it is far easier for the package maintainers or the Security Team to patch and rebuild one package than to scan the entire archive for all copies of this code and patch them individually (this happened for zlib, for example). It's also preferable for the end users to receive an update for just one package (e.g. OpenSSL) rather than a large number of applications. It is even worse if you maintain your private forks of third party libraries. This is not only troublesome for distributors but in the long run also to your own project. You should always make sure that the patches you might need for your specific application will be backported to the library upstream - that's simply how Free Software works.

Is there a chance to disable boost.simd or would that render pythran virtually pointless?

[saraedum]

Out of curiosity, @serge-sans-paille are you now actually planning on migrating to xsimd? (Which, apart from the work involved, seems like a great idea.)

[serge-sans-paille]

Out of curiosity, @serge-sans-paille are you now actually planning on migrating to xsimd? (Which, apart from the work involved, seems like a great idea.)

And the answer is... yes :-)

[serge-sans-paille]

@saraedum xsimd is now vendrored with pythran, and we dropped the boost.simd dependency. It's now possible to install pythran without extra deps if boost.math and boost.format and xsimd are already installed, using the appropriate setup.py flags. (master version only, requires the latest version of xsimd)

[serge-sans-paille]

@saraedum up? Pythran is packaged in Fedora now, it would be cool to have it in Debian too.

[saraedum]

I have no time to work on this, sorry. I am happy to assist if you want to give it a try yourself of course.

[diego-plan9]

As a RFP was recently filled, I'm volunteering to retake the debian packaging effort (including the packaging of gast and beniget). :crossed_fingers:

Hopefully thanks to the migration to xsimd (which also entered debian unstable recently) and the good work since the original effort things will be a bit smoother!

[serge-sans-paille]

@diego-plan9 : That's great news!, thanks for sharing it :-) I'm going to do a release by the end of the week, so that's probably too short to sneak some distrib-specific patches in, but feel free to propose any updates / ask questions etc.

[diego-plan9]

Thanks Serge for the heads up on the new version - and for being so open to potential discussions, questions and patches!

The "real" packaging work is likely still some time away, as it needs the dependencies to be packaged first, but I'll make use of your invitation of sharing my findings and asking questions as soon as I have progressed a bit more :+1:

[diego-plan9]

As a brief status update: gast is in the NEW queue, a RFS for beniget will be issued after gast is accepted, and pythran 0.10.0 keeps being polished - work in progress can be checked at salsa.

Taking advantage of the previous invitation, some questions/notes:

1. Is there a subset of the test suite that be used as a "proxy" for ensuring the package works as expected?

   Predictably, I have been running into some practical issues trying to run the full test suite (impressive coverage!). It is still in my TODO list to find out how to best handle long-running processes at the Debian packaging/CI side, but if there were a subset of tests that could run in reasonable time and still have enough confidence about the package being working, that would be very handy. I have been preliminary using test_base.py for that purpose - but any other recommendations would be welcome!

2. Usage of -Wno-absolute-value - is it safe to remove?

   When running the test suite with the PYTHRAN_CXX_FLAGS flags defined in TestEnv, I ran into the following (using gcc-11):

   error: command-line option '-Wno-absolute-value' is valid for C/ObjC but not for C++ [-Werror]
   

   I'm not sure if this points towards a C++ compiler being used instead of a C compiler in my setup, or if it might point to an an issue with the versions of the compilers involved / supported.

3. Support for /etc/pythran.cfg (or a similar system-wide configuration dir)

   Related to how to best deal with the .cfg files - does the current version look for pythran.cfg in the /etc/ folder? It's far from required - mostly a wishlist/question item.

4. Packaging the headers as a separate pythran-dev package

   This one is basically a question out of curiosity - has it been considered to distribute the .hpp files as a separate package? The thinking would be to take advantage of them ending up installed in a standard location, and keep the Python package a bit more "pure" (and biased towards a packaging perspective). I have not looked into the implications fully, but I'm wondering if you would be open to me attempting to tackle it at the debian-packaging level (which might end up being causing more issues than solving, as I don't want to deviate too much from the original sources!), either at this stage or eventually.

[diego-plan9]

As another update - both beniget and gast have just cleared the new queue, and should be in experimental soon.

I've also updated the salsa pythran packaging attempting to make reasonable assumptions:

• for 1 above, the package currently uses test_base.py only.
• 2 is included as a patch
• 3 and 4 are not tackled

Could you let me know if this seems reasonable? Hopefully should make the package ready for review :tada:

[serge-sans-paille]

Thanks @diego-plan9 for the update and sorry for the late reply.

1. You can get a better coverage of pythran functionalities by running test_cases.py only. It's slightly slower than test_base.py though.

2. It's safe to remove -Wno-absolute-value (I'll do it upstreamm too)

3. /etc/pythran.cfg is not necessary at all.

4. The pythran headers have no reason to be bundled as a seperate package. Their content is bound very thighly to the code generated by a given pythran version, and the API is not meant to be used by someone else (yet)

[diego-plan9]

Thanks for the reply Serge - and no rush! On a preliminary run, test_cases.py seems to be considerably slower than test_base.py, although I have run it without xdist. It might still be under the acceptable threshold, though!