Fix handling of flavour changes

[ari]

Prerequisites

• [X] Have you checked for an existing issue describing your problem?
• [X] Are you running the latest version?
• [X] Is your ports tree recent?
• [X] Is your FreeBSD Host on a supported release?

How to reproduce

1. Change make file from DEFAULT_VERSIONS+= python=3.7 to DEFAULT_VERSIONS+= python=3.8
2. poudriere bulk -j 13 -f myports

Expected behavior

I would expect all python ports to be rebuilt with the new flavour and the old pkg to be removed. I could then simply do pkg upgrade on each server to move to the new python.

Instead I've now got a mixture of python 3.7 and 3.8 pkgs.

Ideally poudriere would detect the flavour change in the same way it is able to detect options changes.

[bdrewery]

If you want to trim out the py37 packages use pkgclean. Adding that into a script right after bulk should do what you're wanting.

Root cause and discussion

The root cause of this is poorly (not) thought out package naming schemes in the ports framework for python, ruby, php, pear, and lua.

The problem is how ports names packages. py37-foo is not the same package as py38-foo. That may seem obvious in a way but consider that before flavors misc/py37-foo and misc/py38-foo would obviously be separate unique things. With FLAVORS misc/foo with 2 different FLAVORS are separate unique things. They come from the same origin (misc/py-foo) but because they have separate FLAVORS Poudriere intentionally treats them as the separate things they are. bulk will often not inspect packages it does not care about because people tend to run bulk and testport with partial lists and expect their not-listed packages to remain untouched. This is especially true with people testing ports. #493 may eventually change this.

Another problem is pkg install py37-foo means py38-foo will never automatically replace it. This one has bitten me before many times.

Back to the naming, if all the user cares about is the command foo then our current scheme is confusing and broken. If ports named it foo rather than FLAVOR-foo then there would be no problems. I am suggesting that for default 3.8 py38-foo would be foo and everything else from misc/foo would be PYFLAVOR-foo. Poudriere would rebuild on the flavor change for the same package name before/after update and no orphaned package would remain. It's also simpler for users to pkg install foo rather than having to determine the py38 portion of pkg install py38-foo.

Porters seem to care more about the "modules", or "library", aspect of a package rather than the CLI aspect. For example, py38-foo may include /usr/local/bin/foo but also include /usr/local/lib/python3.8/site-packages/foo which someone may want to include in their own scripts. Another confusing example is arcanist-php80-20210113 with php80 in it when it only provides 1 file, /usr/local/bin/arc.

Potential fixes

• Sub-packages (#741) could potentially be used to create the default foo which depends on py38-foo that contains the library files.
• Until sub-packages, it could be possible to remedy the problem by auto creating metadata packages named foo for the default versions which simply depended on the current default pyXX-foo. Then when the default changes to py39, the py38 flavor would be built as py38-foo and the metadata package would be updated to depend on py39-foo. pkg and Poudriere both support rebuild/upgrade of a changed dependency without PORTVERSION/PORTREVISION bump. So a user installing foo would always get whatever the default is but a user installing py38-foo would always keep that version independently installed. Technically this could be implemented as another flavor named, for example, py. So with flavors py37 py38 py39 py and a default Python 3.8 we would get packages: py37-foo, py38-foo, py39-foo, foo. foo would depend on py38-foo. I'll have to think upon this more and maybe present a patch for review in Ports to achieve this.

[ari]

Version handling is by far the worst part of an otherwise excellent ports system in FreeBSD. We've got ports like https://www.freshports.org/net-mgmt/zabbix52-server/ which are versioned by creating a new port every six months for no particular reason. Others like openjdk which have javavmwrapper to route connections between libraries/apps to the correct runtime. And then flavours like python where libraries are built and installed to specific versions of the runtime even though 99.9% of those libraries are completely portable.

Of course many of these choices are driven by upstream choices, but the FreeBSD ports system doesn't make it any easier. The distinction between origin and pkg names is confusing. What's the point of origin groups (www, devel) if the package name has to be unique anyway?

Back to poudriere... the pkg system can detect options changes. I assume there is some sort of hash of all the options stored somewhere. Could we store the flavour as another option? Would that be a lightweight first step, even if it doesn't solve all the things? At least we could trigger the upgrades to happen when a flavour changes.

Are you suggesting that flavours all become subpackages? That there is then py-foo and py38-foo both installed, with py-foo just an empty wrapper to force the appropriate dependency to be installed?

[bdrewery]

Back to poudriere... the pkg system can detect options changes. I assume there is some sort of hash of all the options stored somewhere. Could we store the flavour as another option? Would that be a lightweight first step, even if it doesn't solve all the things? At least we could trigger the upgrades to happen when a flavour changes.

We do store and compare FLAVORS already for packages with the same origin and PKGNAME in delete_old_pkg Since the PKGNAMEs for py38-foo and py37-foo differ we don't compare them.

• https://github.com/freebsd/poudriere/blob/a3b8720a3d6eadfafee6808421c05950f421e1a5/src/share/poudriere/common.sh#L5517
• https://github.com/freebsd/poudriere/blob/a3b8720a3d6eadfafee6808421c05950f421e1a5/src/share/poudriere/common.sh#L5676-L5683

I am about to push a commit that should address the case in this issue. I'm keeping the issue open though for a reminder to look into the python package naming scheme.

Are you suggesting that flavours all become subpackages? That there is then py-foo and py38-foo both installed, with py-foo just an empty wrapper to force the appropriate dependency to be installed?

I'm suggesting something analogous to Linux "dev" packages. Ones that may have a CLI tool as its potential primary purpose but also provides a library, module, scripts, whatever, that could be the intended primary purpose. So /usr/local/bin/foo vs /usr/local/lib/libfoo.so. For ports they probably should depend on both by default but could gradually move to only depending on what they need. I suppose 1 package may produce 3 different packages. The CLI, the library/module, and any headers and other build-related stuff. So someone could install just the CLI package (and it depends on the lib subpackage). But if they install the dev package they get the library package and headers package. I think by default a user typing pkg install foo should get all 3, or whatever they might get today. So I'm definitely not proposing something too annoying here. Any patch I work towards for the default PKGNAME for Python though would only be a very simple meta package py-foo that depends on the current default pyXX-foo that has the real files. When the default changes py-foo would be automatically rebuilt by Poudriere and pkg upgrade handles it too so the user would lose the no-longer-needed pyOLD-foo and would get the new pyNEW-foo while py-foo remains.

[ari]

Thank you. If I had a wishlist for pkg itself, I'd love to be able to have a switch which disabled flavours entirely. So rather than supporting three versions of python all installed at once in their own paths, I'd like my python libraries to be installed in a common location.

For many people, supporting multiple versions of python (or php, etc) at the same time is not required and adds complexity. I prefer to move all my python applications to a new version in one go and trust upstream python compatibility enough that I can go from python 3.7.2 to 3.8.3 without fear, just as I today might go from 3.7.2 to 3.7.3. Its unlikely the major 2->3 change will ever happen again.

But I know I'm now well outside the scope of this bug and poudriere in general.

[vishwin]

This discussion seems to be limited to Python and other scripting language package flavours. It's not necessarily about supporting multiple versions at the same time, but rather not every package supports or runs properly on every version we have. Thus, we cannot simply trust upstream Python compatibility, even in 3.x; there have been more instances than I would like to admit during the 3.8 and now the 3.10 cycles that flavours have had to be marked BROKEN for the newest release. Combined with how we offer quarterly packages, some folks want to strictly manage their DEFAULT_VERSIONS to older versions whilst allowing the flexibility to run say one or two packages under a newer Python. Thus the perceived/real complexity here is warranted.

Flavours in other ports, ie devel/git, are most definitely warranted, else we go back to the slave port maintenance nightmare. So a switch to disable flavours should not happen.

I am one of those partial list bulk people, because CPU cycles are very limited and unnecessarily rebuilding consumers of consumers is not a good use of CPU time.

[ari]

Hi @vishwin . I'm not sure I understand your points here. The problem is that it is rarely possible to run one or two packages under a different Python (or PHP, etc). If two ports each rely on a very common port (for example www/py-requests) then you have to upgrade everything to the new flavour in one go. You can't have py38-requests and py37-requests installed at the same time, even though they install to the same place.

Anyhow, perhaps we should just focus on poudriere here. It is hard to change flavour in poudriere without a clean rebuild of the whole tree. That's step 1.

[vishwin]

The problem is that it is rarely possible to run one or two packages under a different Python

This is patently false on those ports specifying USE_PYTHON=concurrent, as the case with www/py-requests. Every file that port installs is contained within lib/<PYTHON_VERSION>/site-packages, meaning that they do not install to the same place, so py37- and py38- can most certainly coexist. For packages containing CLI commands, the non-default version flavour is installed as bin/<foo>-<PYTHON_SUFFIX> rather than bin/<foo>. Additionally, flavours is designed such that when building ports in non-default flavours, you should be specifying the flavour directly, ie www/py-requests@py37.

When it comes to poudriere (and FreeBSD packaging in general), given the flexibility allowed, not automatically removing non-default flavoured packages is intended.

[bdrewery]

This discussion seems to be limited to Python and other scripting language package flavours. It's not necessarily about supporting multiple versions at the same time, but rather not every package supports or runs properly on every version we have. Thus, we cannot simply trust upstream Python compatibility,

I'm not sure what you're responding to but this issue isn't about compatibility. It's about users getting a sane default and default upgrades to the new default. Currently people have to jump through hoops on new