[Task]: Update the minor version of protobuf library in the upper bound prior to Beam release.

[AnandInguva]

What needs to happen?

If a Beam dependency has a flexible upper bound, users will download the most recent compatible version of a dependency at sdk installation time. Overtime, the version used at job submission may become newer than the version installed in a released Beam container. Given that forwards-compatiblity of a protobuf library is not guaranteed, the pipeline may fail.

To mitigate, protobuf library should be specified in install_requires with tight upper bound limiting to last recently released minor version. But if we depend on an old version of a library, it will cause inconveniences users,. Therefore, we should periodically update the upper bound we set, at least once per release cycle.

Issue Priority

Priority: 3 (nice-to-have improvement)

Issue Components

• [X] Component: Python SDK
• [ ] Component: Java SDK
• [ ] Component: Go SDK
• [ ] Component: Typescript SDK
• [ ] Component: IO connector
• [ ] Component: Beam examples
• [ ] Component: Beam playground
• [ ] Component: Beam katas
• [ ] Component: Website
• [ ] Component: Spark Runner
• [ ] Component: Flink Runner
• [ ] Component: Samza Runner
• [ ] Component: Twister2 Runner
• [ ] Component: Hazelcast Jet Runner
• [ ] Component: Google Cloud Dataflow Runner

[AnandInguva]

@damccorm can you add the milestone 2.48.0 to catch this for the next release?

[damccorm]

The next release is 2.47.0 (the current one is 2.46.0)

[tvalentyn]

cc: @tvalentyn

[damccorm]

This is still up to date, moving to 2.48

[Abacn]

This is still up to date (4.23.3<4.24.0), moving to 2.50

[lostluck]

2.50 release manager here. This issue is currently tagged for the 2.50.0 release, which cuts in a week on August 9th.

Please complete work and get it into the main branch in that time, or move this issue to the 2.51 Milestone: https://github.com/apache/beam/milestone/15

[riteshghorse]

Already on latest version 4.23.4. Moving to 2.51.0 Milestone

[damccorm]

We're up to date as of now looking at https://pypi.org/project/protobuf/ and https://github.com/apache/beam/blob/master/sdks/python/setup.py#L300

This might change if protobuf gets a fix in for #28246 - but I'll at least move this blocker forward and we can address that in the separate issue

[jrmccluskey]

It looks like we're up to date with the upper bound at <4.26.0 and the latest release is 4.25.1.

[lostluck]

There's one week until the 2.54.0 cut and this issue is tagged for that release, if possible/necessary please complete the necessary work before then, or move this to the 2.55.0 Release Milestone.

[Abacn]

4.25.3 is the latest 4.x release as of 2.55.0 cut day. and it is <4.26.0. Next release needs to aware that the upcoming release bumped the major version, 4.25 -> 5.26, indicating breaking change

note: protobuf currently encode its version in minor version number for all language impls; and the major version is reserved for language specific breaking change

[riteshghorse]

+1. I'm working with @tvalentyn on workaround for that

[damccorm]

We still need to wait for google-api-core to release new version. The latest version of google-api-core has a dependency on protobuf<5 which would result in dependency conflict if we update the protobuf version for beam.

apache-beam[gcp,test] 2.55.0.dev0 depends on protobuf==5.26.0rc3
    google-api-core 2.17.1 depends on protobuf!=3.20.0, !=3.20.1, !=4.21.0, !=4.21.1, !=4.21.2, !=4.21.3, !=4.21.4, !=4.21.5, <5.0.0.dev0 and >=3.19.5

[kennknowles]

Is there any outstanding work on the protobuf major version bump?

[liferoad]

https://pypi.org/pypi/google-api-core/2.19.0/json

requires_dist": [
      "googleapis-common-protos<2.0.dev0,>=1.56.2",
      "protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0.dev0,>=3.19.5",
      "proto-plus<2.0.0dev,>=1.22.3",
      "google-auth<3.0.dev0,>=2.14.1",
      "requests<3.0.0.dev0,>=2.18.0",
      "grpcio<2.0dev,>=1.33.2; extra == \"grpc\"",
      "grpcio-status<2.0.dev0,>=1.33.2; extra == \"grpc\"",
      "grpcio<2.0dev,>=1.49.1; python_version >= \"3.11\" and extra == \"grpc\"",
      "grpcio-status<2.0.dev0,>=1.49.1; python_version >= \"3.11\" and extra == \"grpc\"",
      "grpcio-gcp<1.0.dev0,>=0.2.2; extra == \"grpcgcp\"",
      "grpcio-gcp<1.0.dev0,>=0.2.2; extra == \"grpcio-gcp\""
    ],

The new version requires protobuf < 5.0.0.dev0

[liferoad]

https://github.com/apache/beam/pull/30556 more work needed in the future.

[jrmccluskey]

https://pypi.org/pypi/google-api-core/2.19.1/json

Looks like the google-api-core package has been updated requiring < 6.0.0.dev0 for protobuf now. Do we want to prioritize this upgrade for 2.58.0, targeting the same upper bound?

[jrmccluskey]

Followed up, google-api-core 2.19.1 supports protobuf 5 but other GCP dependencies (specifically google-cloud-aiplatform) do not. The workaround for the breaking timestamp change introduced in 5.26.0 explored in #30556 can go in before we formally support protobuf 5.x, but will need a little bit more effort to test as a result. I'll be coming back to work on this after the 2.58 release

[lostluck]

Do we need to do anything further for the 2.59.0 release for this, or should I push it to 2.60.0 ?

[damccorm]

It is fine to push to 2.60

[Abacn]

protobuf 4.25.x still receives update, the latest one being 4.25.5 on Sept 18, 2024. Moving to 2.61.0

[damccorm]

We now support arbitrary 5.x versions (as of https://github.com/apache/beam/pull/32679) so we can close this