Drop torchvision 0.2.2 from conda-forge as it's often a silent indicator of major package conflicts

[vadimkantorov]

🐛 Describe the bug

Running the following triggers install of torchvision 0.2.2.

We need a stable conda install recipes and suggestions that deal with common bugged dependencies such as OpenCV and ffmpeg given all the mess with glibc / python versions etc. At least some known working commands (even if not the latest python).

I understand that it can't be the job of pytorch maintainers to test all conda packages, but the combination of torch / opencv / ffmpeg (also maybe some packages for 3d vision) is super-common one. If someone succeeds in installing them together, it is super-valuable to collect and publish these experiences. Or maybe some mamba recipes?

cc @malfet

PREFIX=/path/lab/vadim/prefix
mkdir -p $PREFIX

curl -L -so $PREFIX/miniconda.sh https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh && chmod +x $PREFIX/miniconda.sh && $PREFIX/miniconda.sh -b -p $PREFIX/miniconda && rm $PREFIX/miniconda.sh

export PATH=$PREFIX/miniconda/bin:$PATH

conda install -y opencv pytorch torchvision cudatoolkit=11.1 -c pytorch -c conda-forge

Versions

N/A

[vadimkantorov]

If it's again a problem with Python 3.10 released and conda trying to use it, can we delete torchvision 0.2.2 from conda altogether? Better have an error than a crippled install

[vadimkantorov]

Currently it seems that torchvision is not compatible with OpenCV conda package: maybe because of libpng / libjpeg others

[vadimkantorov]

Strangely enough, installing opencv+pytorch from conda and then installing torchvision from pip seems to work:

conda install -y opencv pytorch cudatoolkit=11.1 -c pytorch -c conda-forge
python -m pip install torchvision

A question: why installing torchvision from conda doesn't work?

[fmassa]

@vadimkantorov thanks for the report.

I believe we are tracking this issue in https://github.com/pytorch/vision/issues/4648

[vadimkantorov]

I'll then rename this issue to propose dropping torchvision 0.2.2 from conda algotether (I think error is more preferred than so outdated installation)

[vadimkantorov]

Another instance is at https://github.com/pytorch/vision/issues/5374

I think it would help a lot of debugging time to simply remove version 0.2.2 from all package managers, it is massively outdated and hides other dependency issues. I think a installation error is preferable to silent installation of such outdated versions

[hmaarrfk]

Im just browsing around and stumbled onto this.

It would be great to have somebody from the pttorch team helping maintain the packages at conda forge.

We can patch up old packages to give the the appropriate constraints to ensure a new package is pulled

[vadimkantorov]

IMO that antique 0.2.2 package should just be deleted from conda-forge regardless of other general better maintentership practices...

[hmaarrfk]

That is an opinion. But people are moving toward making science more reproducible, and "pulling" packages is somewhat counter to that.

~We~ conda-forge (we) have guidelines on how to deal with broken packages, and generally conda-forge has solutions, for this, outlined https://conda-forge.org/docs/orga/guidelines.html?highlight=broken#fixing-broken-packages

I guess ultimately, conda-forge and pytorch are two different community organizations. I was mostly trying to point to maybe a more sustainable collaborations. I'm not sure what your affiliation is vadimkantorov but if you are speaking about the conda-forge channel, I think you want to cross reference your issue on https://github.com/conda-forge/pytorch-cpu-feedstock/

[vadimkantorov]

I have no affiliation with torchvision / FAIR etc so I'm not speaking for pytorch/torchvision maintainers or the team :) So my messages are no more than just an opinion, as you correctly guessed :) I have found on numerous occasions that very antique 0.2.2 package gets used only in case of severe package conflicts and is always an indication/cover-up/hiding of larger problems

[hmaarrfk]

severe package conflicts and is always an indication/cover-up/hiding of larger problems

Correct.

I am sympathetic with your challenges. I wanted to bring attention that you are likely asking the wrong community for help.

In either case, you did get the attention of the conda-forge community (myself). If you do raise an issue at conda-forge, my first ask would be to check that it still occurs today (I patched up a few releases) and likely submit a patch to mentionned repository in the link i sent you asking me to review.

Ultimately, I do think it is important that Meta/Facebook/Pytorch be aware that users are using conda-forge and that it may be useful for them to help maintain it.

[h-vetinari]

I have found on numerous occasions that very antique 0.2.2 package gets used only in case of severe package conflicts and is always an indication/cover-up/hiding of larger problems

A less severe form of fixing that (rather than marking it as broken) is to write a repodata-patch that "corrects" the metadata of this package in a way that it does not get considered for contemporary environments anymore. For that, it would be important to have an example where such a wrong solver result occurs.

[vadimkantorov]

At the point of creating this issue, the example in OP was repro (and in issues there were other cases like this, but I don't have them at hand). But I think, 0.2.2 is truly antique to the point that deleting it completely will not do any harm and just prevent unnoticed conflicted installations.

[vadimkantorov]

cc @atalman

[vadimkantorov]

Another instance of this: https://github.com/pytorch/vision/issues/6842

[vadimkantorov]

Another related issue: https://github.com/pytorch/vision/issues/7038

Keeping 0.2.2 on pip/conda just conceals dependency resolving issues

[pmeier]

We yanked the torchvision < 0.3 releases from PyPI. Meaning, you can still install them explicitly with pip install torchvision==0.2.2, but they are excluded from range installs like pip install torchvision.

As for the packages on in the conda-forge channel, this is not actionable here, since we don't maintain them. The maintainers for the package are listed in the README of the feedstock: https://github.com/conda-forge/torchvision-feedstock. I suggest you take it up with them.

[vadimkantorov]

@pmeier Actually, the maintainer of conda-forge/torchvision is @hmaarrfk who commented in this very thread :) https://github.com/pytorch/vision/issues/4665#issuecomment-1189375716

So if you have a message to pass from the PyTorch team - it's conveniently the right place :)

[vadimkantorov]

Are there any similar problems with other domain libraries? like torchaudio / torchtext?

[h-vetinari]

As for the packages on in the conda-forge channel, this is not actionable here, since we don't maintain them. The maintainers for the package are listed in the README of the feedstock: https://github.com/conda-forge/torchvision-feedstock. I suggest you take it up with them.

I happen to co-maintain torchvision in conda-forge, though 0.2.2 was before my time. It would be strictly speaking possible to mark those builds as broken (meaning they don't get taken into account for the resolver, and never installed unless explicitly asked with a special incantation), but this is not great for many reasons:

• It breaks reproducibility of existing environments (yes, it can be argued that they are broken, but maybe they work just well enough for someone to have built something on top).
  • This is especially relevant the longer packages have been available, which in this case, is a looooong time.
• Removing torchvision from the solver is not guaranteed to avoid the silent package conflicts, which may just move on to some other package. It can be a game of whack-a-mole.
• Finally, those conflicts do not (to my knowledge) appear in the only setup that conda-forge supports[^1], which is "all packages from conda-forge", or more specifically "strict channel priority". If you have a recent example to the contrary, please let us know on the feedstock.

[^1]: This is a function of the impossibility of taking care of the combinatorial explosion of possibilities once different packages may get installed from different channels (including PyPI), multiplied by possible ABI incompatibilities, and the fact that we have zero control over what other channels do even if we were to diagnose a problem. We have enough on our plate just to keep just our corner of the ecosystem healthy.

[hmaarrfk]

vadim, I understand why pmeler closed the issue. it really isn't related to this specific project.

Generally, this is a hard issue for us: (see others trying to discuss it https://github.com/conda-forge/conda-forge.github.io/issues/1867)

As a volunteer to the conda-forge community, I've asked for help and pointed you and others to the best reference i knew at the time https://github.com/pytorch/vision/issues/4665#issuecomment-1189361999

As a company Meta is in their right to close issues that they deem out of scope.

If you need help installing conda-forge packages, please open an issue on the relevant feedstocks. We ask for alot of specific information to help us, help you, troubleshoot.

h-vetinari's comment hold.

[pmeier]

@vadimkantorov

So if you have a message to pass from the PyTorch team - it's conveniently the right place :)

There is no new information otherwise I would have done so. I'm not aware of any decision in favor or against officially supporting the conda-forge build. I know about at least one instance were the conda-forge maintainers approached us and I think we responded in a timely manner (I can't find the issue / PR).

That being said, I'm personally in favor of doing so, but that is not my decision to make. Plus, we (torchvision team) are currently understaffed, so large scale changes to our packages are pretty far down on our priority list. In that sense, a big thank you from me personally to @h-vetinari and @hmaarrfk for maintaing the conda-forge package.

Are there any similar problems with other domain libraries? like torchaudio / torchtext?

I'm not aware of any. Skimming the release history on PyPI, it seems for both packages the very first release already included compiled code and thus there is never any wheel that pip falls back to.

@h-vetinari

It breaks reproducibility of existing environments

I'm not too knowledgeable on conda packaging. Are you saying that marking them as "broken" means even if someone specifies torchvision==0.2.2 in their environment, they can't be installed? If yes, I agree this is not a good way forward.

If no, could you explain how marking them as broken is different from yanking releases from PyPI as defined in PEP 529? As explained above, if someone has an unpinned torchvision requirement, < 0.3 will no longer be picked up. We are ok with that kind of behavior, because if the version is not pinned, one cannot expect to have reproducible environments.

[h-vetinari]

I'm not too knowledgeable on conda packaging. Are you saying that marking them as "broken" means even if someone specifies torchvision==0.2.2 in their environment, they can't be installed? If yes, I agree this is not a good way forward.

Yes, conda works differently here than yanked packages. If a package is marked broken, it cannot be installed even with a hard pin, unless specifying an additional channel that enables installation of packages that have been marked broken, i.e.

# normal case, package not broken
conda install -c conda-forge some_package=some_version

# installing a package that has been marked broken
conda install -c conda-forge -c conda-forge/label/broken some_package=some_version

This is an escape hatch to enable reproducibility even in cases of a package having been marked broken, but it's not widely known (my impression), and so - de facto - marking builds that have been around a while as broken is equivalent to breaking the reproducibility of any environments that make use of those builds.

[skroon]

I'm trying to install torchvision 0.2.2 to build a reproduction of an environment we have been using, but I now get a PackageNotFoundError. The 0.2.2 binaries do not appear to be on conda-forge : they are not listed at https://anaconda.org/conda-forge/torchvision/files?page=10 (they do not appear when I choose the broken label either), and the PackageNotFoundError persists when I include conda-forge/label/broken in the channels for the environment.yml.

I don't see anything in the above indicating that the 0.2.2 files have been removed, or even that the package has been marked as broken on conda-forge. Does anyone here have more information on this?

[pmeier]

@skroon We are not maintaining the conda forge package. Please raise an error to the team that is: https://github.com/conda-forge/torchvision-feedstock

I can confirm however that 0.2.2 seems to be missing.

❯ conda search -c conda-forge torchvision | grep '0\.2'
torchvision                    0.2.0          py27_0  pkgs/main           
torchvision                    0.2.0          py35_0  pkgs/main           
torchvision                    0.2.0          py36_0  pkgs/main           
torchvision                    0.2.1          py27_0  pkgs/main           
torchvision                    0.2.1          py35_0  pkgs/main           
torchvision                    0.2.1          py36_0  pkgs/main           
torchvision                    0.2.1          py37_0  pkgs/main

[skroon]

After creating and discussing a question about this on https://github.com/conda-forge/torchvision-feedstock, I have verified on an earlier image that torchvision 0.2.2 was actually installed from the pytorch channel, and not the conda-forge channel. As a result, I've returned to this issue to follow up on the problem. The pytorch channel is also missing torchvision 0.2.2:

$ conda search -c pytorch torchvision | grep '0\.2'
torchvision                    0.2.0          py27_0  pkgs/main           
torchvision                    0.2.0          py35_0  pkgs/main           
torchvision                    0.2.0          py36_0  pkgs/main           
torchvision                    0.2.1          py27_0  pkgs/main           
torchvision                    0.2.1          py35_0  pkgs/main           
torchvision                    0.2.1          py36_0  pkgs/main           
torchvision                    0.2.1          py37_0  pkgs/main 

[pmeier]

Let's open a new issue for this.