pipelines
pipelines copied to clipboard
kubeflow
[sdk] Support for Protobuf 5
Environment
The latest kfp version pins protobuf < 5.0:
https://github.com/kubeflow/pipelines/blob/2c91fb797ed5e95bb51ae80c4daa2c6b9334b51b/sdk/python/requirements.in#L21
Other packages such as grpcio are starting to require protobuf 5. What would it take for KFP to support protobuf 5?
Impacted by this bug? Give it a 👍.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
@HumairAK, @gregsheremeta, if I understand correctly, you plan to work on this? If so, any timeline? Thanks!
When this is worked on, it would be great if it could be done in a way that allows simultaneous support for Protobuf 4 and Protobuf 5. That way, users upgrading their kfp version won't be forced to track down and upgrade all packages depending on Protobuf 4 at the same time.
When support for Protobuf 4 was added to kfp in #10305 and #10307, support for Protobuf 3 was dropped in the same commits/releases. That was not ideal for users.
All of the GCP SDK packages support Protobuf 3, Protobuf 4, and Protobuf 5 together (example: google-cloud-pubsub). Hopefully something similar can be done for kfp for Protobuf 4/5.
The GCP SDK packages seem to achieve their simultaneous support by using the proto-plus package as a compatibility layer. It's possible that a similar approach would work for kfp. And even if that wouldn't work, that proto-plus package itself supports multiple versions of Protobuf, so perhaps what is done there would work for kfp.
When this is worked on, it would be great if it could be done in a way that allows simultaneous support for Protobuf 4 and Protobuf 5. That way, users upgrading their kfp version won't be forced to track down and upgrade all packages depending on Protobuf 4 at the same time.
There's now a release candidate for Protobuf 6. It would be nice to include support for that in this, too. Edit/update: Protobuf 6 was released.
The same goes for KFP 1.8.22 which is now incompatible with the kserve SDK. So 1.8.23 just with updated dependencies is needed.
We hit the same dilemma recently as we run DBT in Vertex runtime which uses kfp to build the pipeline. DBT has iterated to its major support efforts for 1.9/1.10 with dependency on protobuf >=5.
So the dependency deadlock causes kfp no co-existent with DBT under community support in the same project.
Appreciate for any efforts to fix this issue.
ref: https://docs.getdbt.com/docs/dbt-versions/core
worked on in https://github.com/kubeflow/pipelines/issues/12038 and the PR https://github.com/kubeflow/pipelines/pull/12042
This is resolved for v2, v1 proto upgrade work to address cves is also happening here, if you would like to track it
We opted to not support ranges and do a hard switch to protobuf 6 (and protoc 31.x) , as the burden for maintaining older versions was becoming too great. If this continues to be an ask from the community, please create a new github issue so we can track it separately and gauge interest and prioritize accordingly.