mediapipe icon indicating copy to clipboard operation
mediapipe copied to clipboard

Migrate to protobuf >4.0.0

Open adrianloy opened this issue 1 year ago • 7 comments

Is there anything planned towards a migration to a newer protobuf version? Most of the python ecoversion have already adopted v4, and it makes integration of mediapose with other libraries difficult. In my case, I want to run a flyte pipeline which needs a newer protobuf version.

adrianloy avatar Jan 18 '24 09:01 adrianloy

Among many others, this also blocks usage with recent versions of gRPC

therc avatar Jan 30 '24 14:01 therc

Hello @adrianloy and @therc , we dont have a plan in place for upgrading Protobuf version. However, if this turns out to be a major issue affecting the adoption of MediaPipe in larger applications, we surely will add this to our roadmap.

Hence, to further understand the requirements and how it affects your application needs, we would request you to elaborate your use cases in detail, with the blocker, the error you get right now, and any relevant details that you think might help us understand your pain points better.

ayushgdev avatar Jan 30 '24 18:01 ayushgdev

We're on an old version of kubeflow pipelines (kfp 1.8.22 from last May), because mediapipe doesn't like protobuf v4. Google announced v2, which has lots of improvements, including in security: https://cloud.google.com/blog/products/ai-machine-learning/whats-new-in-kubeflow-pipelines-v2 but that will crash and burn if you have mediapipe: kfp 2.6.0 requires protobuf<5,>=4.21.1, but you have protobuf 3.20.3 which is incompatible

See the upstream issue: https://github.com/kubeflow/pipelines/issues/8370 and also the comment that Python protobuf v3 has been unsupported since July https://protobuf.dev/support/version-support/#python

As for gRPC, you're stuck with 1.54. There have been six releases since.

More in general, mediapipe is not very Conda-friendly and we've gently nudged our users to remove it as a dependency...

therc avatar Jan 30 '24 19:01 therc

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

github-actions[bot] avatar Feb 07 '24 01:02 github-actions[bot]

Another example: recent versions of Dask started warning about using pyarrow 11, because a switch to 14.0.1 is imminent. But pyarrow-14 requires libarrow-flight-14, which requires libgrpc 1.59, or libgoogle-cloud 2.12.0, which requires libgrpc 1.60. And both of those...

therc avatar Feb 13 '24 19:02 therc

Hi! I have the same problem, I need to install mediapipe along with weaviate-client >= 4.0 but I can not because of the mentioned conflict with protobuf:

8.751 ERROR: Cannot install -r requirements.txt (line 22) and weaviate-client because these package versions have conflicting dependencies. 8.751 8.751 The conflict is caused by: 8.751 mediapipe 0.10.9 depends on protobuf<4 and >=3.11 8.751 grpcio-health-checking 1.60.1 depends on protobuf>=4.21.6

bastiansg avatar Feb 14 '24 02:02 bastiansg

Actually, Dask's error about pyarrow is due to a vulnerability

https://github.com/dask/dask/pull/10622

therc avatar Feb 14 '24 23:02 therc

Thank you for the feedback. To summarize the list of reported issues related to the potential upgrade to Protobuf 4:

  1. Flyte ML pipeline: Requires Protobuf 4 for compatibility.
  2. gRPC: Integration with the latest gRPC is blocked due to the pending Protobuf 4 upgrade.
  3. Weaviate vector database: The v4 client depends on Protobuf 4.0, hindering its use with MediaPipe until the upgrade occurs.

We'll keep this issue open to encourage further reports of any similar incompatibilities. With this information, we'll conduct internal discussions and re-evaluate the prioritization of the Protobuf 4 upgrade.

ayushgdev avatar Feb 27 '24 06:02 ayushgdev