PyPI release is not possible with direct references

[pjbull]

We use direct references for pytorchvideo, densepose, and yolox in our requirements:

https://github.com/drivendataorg/zamba/blob/b4b706a1be2a91879e8d9a38d6658e72fbae7a58/setup.cfg#L37 https://github.com/drivendataorg/zamba/blob/b4b706a1be2a91879e8d9a38d6658e72fbae7a58/setup.cfg#L47 https://github.com/drivendataorg/zamba/blob/b4b706a1be2a91879e8d9a38d6658e72fbae7a58/setup.cfg#L64

The official PyPI does not support direct references. Per PEP-440:

Public index servers SHOULD NOT allow the use of direct references in uploaded distributions. Direct references are intended as a tool for software integrators rather than publishers.

This means that we can't publish to PyPI until our dependencies are free of direct references. The best way to do so is to have maintainers of those packages publish on PyPI. Here's the current status:

• pytorchvideo has a recent release with the features we use. We can start using that immediately.
• yolox has a PR open to make it specify dependencies properly. Once that is merged, there will need to be a separate request to publish as a package on PyPI.
• densepose depdends on detectron2, which has a closed issue requesting it be made available on PyPI.

Plan for now is to have documentation install from github, and to try to push the dependency packages forward to a happy resolution.

When fixing:

• We have a publish to PyPI workflow in our release.yml that is commented out until we can resolve this issue
• We have a PyPI badge that in the README.md that is commented out as well

[ejm714]

yolox is now pip-installable 🎉 as of release on 2022/01/18: https://github.com/Megvii-BaseDetection/YOLOX/releases

[jayqi]

I have a somewhat 🤨 idea regarding detectron2: What if we publish detectron2 to PyPI?

We could either:

• Just claim the detectron2 name, which is open.
• Or, name it something else like detectron2-drivendata. Searching for detectron2 on PyPI, it seems like this was the solution adopted by several other people. (1, 2, 3).

This will take a little work to maintain but should be mostly automateable with CI.

If we do claim the detectron2 name, we could just be nice about releasing it back to FAIR if they ever change their mind about publishing to PyPI.

[ejm714]

yolox is now pip-installable 🎉 as of release on 2022/01/18: https://github.com/Megvii-BaseDetection/YOLOX/releases

I tried pip installing yolox and it seems like they still have an improperly handled dependency.

Collecting yolox
  Downloading yolox-0.3.0.tar.gz (79 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 80.0/80.0 kB 2.6 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [9 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/ft/5tzjjc1d0ysf14360lvrp3d80000gn/T/pip-install-ydc9nz72/yolox_84fe5f27c10f4653a9d63b48b65c31ff/setup.py", line 77, in <module>
          ext_modules=get_ext_modules(),
        File "/private/var/folders/ft/5tzjjc1d0ysf14360lvrp3d80000gn/T/pip-install-ydc9nz72/yolox_84fe5f27c10f4653a9d63b48b65c31ff/setup.py", line 50, in get_ext_modules
          assert TORCH_AVAILABLE, "torch is required for pre-compiling ops, please install it first."
      AssertionError: torch is required for pre-compiling ops, please install it first.
      [WARNING] Unable to import torch, pre-compiling ops will be disabled.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

[jayqi]

It looks like yolox 0.2.0 (2022-01-17) had manylinux wheels on PyPI, which meant you could pip install without torch on Linux.

However, yolox 0.3.0 (2022-04-22) went back to only having source distributions, so you have to build it when you install.