Package transitions at conda-forge

[hmaarrfk]

Your question:

As more features and libraries have been added to the conda-forge ecosystem, some maintainers have expressed the desire to reduce the total number of dependencies imposed on their users.

While a few proposals have been raised to improve creating split packages, standardized naming, catch all for files, one challenge in our ecosystem remains in terms of how to deal with the package transition.

Specifically, these package splits typically occur in phases as:

1. Maintainers gain interest in splitting off certain features.
2. Maintainers learn about different flags proposed upstream.
3. Upstream exposes new capabilities in splitting a package.

The use case I'm particularly concerned with, is with large "mega packages" such as vtk, ffmpeg, opencv where plugin mechanisms may exist that enable discovery of extra features at runtime.

It may be desirable for example:

1. On one day, to split off opencv-hdf5 to a new package. Creating opencv-base and opencv-hdf5.
2. It may be desirable the next day, to also split off libopencv_highgui,

But since we have already used our name "base", we are now stuck with the current tools I know of at conda-forge.

External references

• Debian's transition guide: https://wiki.debian.org/PackageTransition
• Fedora's transition guide: https://fedoraproject.org/wiki/Upgrade_paths_%E2%80%94_renaming_or_splitting_packages https://github.com/conda-forge/conda-forge.github.io/issues/544

Packages that are considering splits for various reasons:

• Removal of optional dependency (ffmpeg) https://github.com/conda-forge/vtk-feedstock/pull/282
• Removal of heavy dependency (qt) https://github.com/conda-forge/opencv-feedstock/pull/337
• Removal of heavy dependency (qt) https://github.com/conda-forge/matplotlib-feedstock/pull/178

Packages where splitting was (seemingly) not possible:

• https://github.com/conda-forge/ffmpeg-feedstock/pull/138

Packages where creating a split package stalled due to unclear (realistic) transition path:

• Splitting qt-base into finer grained packages https://github.com/conda-forge/qt-main-feedstock/issues/78

[h-vetinari]

I'm proposing a linter-rule to add messages for specific dependencies (e.g. if obsoleted by some transition or renaming) in https://github.com/conda-forge/conda-smithy/pull/1734, perhaps this would help.

[jakirkham]

Curious if we have thought about a migrator to handle package renames. This is a common enough issue tooling may help

[h-vetinari]

There is functionality like that checked into regro/cf-scripts, but I've never seen it used and I don't know how it works, or indeed if it still works...

[jakirkham]

cc @beckermr (in case you have thoughts on the migrator angle)

[beckermr]

I can show folks how to setup a replacement migrator that does renaming.

I'd say unless we are willing to apply version epochs widely, we are stuck with this.