Mismatched protobuf versions in sub-dependencies

[briansemrau]

When installing hivemind (as a dependency of petals) using pipenv, pipenv failed to resolve a valid version for protobuf. Could not find a version that matches protobuf<4.0.0,<4.0dev,<5.0dev,>=3.12.2,>=3.20.3,>=4.21.6

Here's the trimmed dependency graph for hivemind to show the conflicts:

- hivemind [required: ==1.1.3, installed: 1.1.3]
  - grpcio-tools [required: >=1.33.2, installed: 1.51.1]
    - protobuf [required: <5.0dev,>=4.21.6, installed: 3.20.3]
  - protobuf [required: <4.0.0,>=3.12.2, installed: 3.20.3]

I haven't tested if this causes any actual issues, but it looks risky.

[justheuristic]

Thanks! I can attest that 3.20.x works fine, but I agree that this will eventually blow in our faces. One way we could handle that is by choosing an older grpcio version version that supports protobuf 3.20, but i'll first try to make it work with the newer protobuf.

[mryab]

I've investigated this issue a bit, and my findings are as follows:

• Newer Protobuf versions introduce breaking changes (https://github.com/onnx/onnx/pull/4629#issuecomment-1304821038) with downgrade being the most popular solution for some. In general, I'd avoid changing anything related to Protobuf compilation/handling unless strictly necessary
• Our current constraints on grpcio-tools should technically work: grpcio-tools==1.33.2 requires protobuf 3.5.0, which is small enough for other constraints

Thus, I'm inclined to believe that the reported error is due to pipenv's dependency resolver and not a mismatch of our dependencies. @briansemrau what's your take on this, have I missed anything? Have you encountered similar issues with Poetry or pip's new resolver?

[briansemrau]

Interesting. Sounds like an issue with pipenv then. It looks like I should be able to constrain grpcio-tools<1.49.0 to give pipenv some help.

I can't speak to other tools, as I haven't used them.