my-protobuf-3.4.0 breaks backward compatibility by silently upgrading to protobuf-4

[alonbl]

Hello,

protobuf-3 is still alive, especially since it is the protoc version that is available in stable ubuntu and other distros and providing the native libraries.

Starting from mypy-protobuf-3.4.0 the protobuf major was bumped, without the library major bump, hence breaking protobuf-3 installations.

protobuf-4 is a new major which is incompatible with protobuf-3, a new major per semver.

As far as I know, it is not a good practice to bump major version of breaking dependencies, without bumping the major of the package, hence instead of my-protobuf-3.4.0 it should have been my-protobuf-4.0.0, and the two y-streams should be maintained.

What do you think? Alon

Traceback (most recent call last):
  File "xxx/.tox/test-py310/bin/protoc-gen-mypy", line 5, in <module>
    from mypy_protobuf.main import main
  File "xxx/.tox/test-py310/lib/python3.10/site-packages/mypy_protobuf/main.py", line 25, in <module>
    from . import extensions_pb2
  File "xxx/.tox/test-py310/lib/python3.10/site-packages/mypy_protobuf/extensions_pb2.py", line 5, in <module>
    from google.protobuf.internal import builder as _builder
ImportError: cannot import name 'builder' from 'google.protobuf.internal' (xxx/.tox/test-py310/lib/python3.10/site-packages/google/protobuf/internal/__init__.py)
--mypy_out: protoc-gen-mypy: Plugin failed with status code 1.
Traceback (most recent call last):

[nipunn1313]

Thanks for the report!

Interestingly, protoc https://github.com/protocolbuffers/protobuf/releases - is still calling itself 3.21.x - but the python package bumped major to 4.21.x

There's some information about this in the release notes here https://github.com/protocolbuffers/protobuf/releases/tag/v21.0

I'll look into this some more.

My thinking is that mypy-protobuf can depend on whichever version of protobuf it wants as long as it generates compatible code. I'm a bit surprised by this crash, so can look into it a bit more to understand. My original thought is that mypy-protobuf depends on its version of python-protobuf, and thus internally should work properly. I'll spend some time trying to repro running a newer mypy-protobuf with an older protoc.

Would appreciate some information from your side if you can provide

protoc --version
protoc-gen-mypy --version

[alonbl]

Hi,

Thanks, here are the versions:

protobuf-compiler-3.12.4-1ubuntu7
$ protoc --version
libprotoc 3.12.4
$ protoc-gen-mypy --version
mypy-protobuf 3.4.0

Regards,

[nipunn1313]

Hi. I've tried to repro and I'm unable. I did this:

mkdir testdir
cd testdir

python3 -m venv env
source env/bin/activate

pip3 install mypy-protobuf==3.4.0
env) ➜ git:(main) ✗ protoc-gen-mypy --version
mypy-protobuf 3.4.0

# Download protoc 3.12.4 from source
(env) ➜ git:(main) ✗ ~/Downloads/protobuf-3.12.4/src/protoc --version
libprotoc 3.12.4

mkdir testout/

(env) ➜ git:(main) ✗ ~/Downloads/protobuf-3.12.4/src/protoc --python_out=testout/ --mypy_out=testout/ ./duration.proto
Writing mypy to duration_pb2.pyi
# success

perhaps something else tricky is going on here? Do you have repro steps?

You mentioned semver incompatibility, however semver allows you to bump a dependency version by a major bump as long as you internalize the associated effects in your library, which is what I believe is going on here.

[alonbl]

Hi,

Thank you for testing, it took me a while to doubt my sanity... took me a while to find the root cause.

We have dependency hell...

Prepare

Environment: ubuntu-22.04

$ sudo apt install protobuf-compiler
$ python3 -m venv /tmp/env.test1
$ . /tmp/env.test1/bin/activate
$ cat > test.proto << __EOF__
syntax = "proto3";

message test {
  int64 n = 1;
}
__EOF__

Default mypy-protobuf installation

We cannot use the default protobuf-4.*, as it is incopatible with ubuntu-22.04 protoc:

(env.test1) $ pip install mypy-protobuf
Collecting mypy-protobuf
  Using cached mypy_protobuf-3.4.0-py3-none-any.whl (15 kB)
Collecting protobuf>=4.21.8
  Using cached protobuf-4.21.9-cp37-abi3-manylinux2014_x86_64.whl (408 kB)
Collecting types-protobuf>=3.20.4
  Using cached types_protobuf-3.20.4.2-py3-none-any.whl (60 kB)
Installing collected packages: types-protobuf, protobuf, mypy-protobuf
Successfully installed mypy-protobuf-3.4.0 protobuf-4.21.9 types-protobuf-3.20.4.2
(env.test1) $ protoc --python_out . --mypy_out . test.proto 
Writing mypy to test_pb2.pyi
$ python3 test_pb2.py 
Traceback (most recent call last):
  File "/home/alonbl/exodigo/exodigo-python-skel/test_project/x/test_pb2.py", line 36, in <module>
    _descriptor.FieldDescriptor(
  File "/tmp/env.test1/lib/python3.10/site-packages/google/protobuf/descriptor.py", line 560, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
$ protoc --version
libprotoc 3.12.4

Force protobuf==3.*

With protobuf==3.* the output is supported by native libraries of ubuntu-22.04, and for some reason it also works with mypy-protobuf-3.4.0.

(env.test1) $ pip uninstall protobuf
(env.test1) $ pip install 'protobuf==3.*'
Collecting protobuf==3.*
  Using cached protobuf-3.20.3-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.1 MB)
Installing collected packages: protobuf
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
mypy-protobuf 3.4.0 requires protobuf>=4.21.8, but you have protobuf 3.20.3 which is incompatible.
Successfully installed protobuf-3.20.3
(env.test1) $ protoc --python_out . --mypy_out . test.proto 
Writing mypy to test_pb2.pyi
$ python3 test_pb2.py
<success>

Force protobuf<3.20

There are dependeices which force this such as tensorboard [https://pypi.org/project/tensorboard/] which has:

Requires-Dist: protobuf (<3.20,>=3.9.2)

When tox pulls everything it installs: mypy-protobuf==3.4.0,...,protobuf==3.19.6,...,tensorboard==2.10.1, there is obviosly a bug in pip as well, in tox log I see:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
mypy-protobuf 3.4.0 requires protobuf>=4.21.8, but you have protobuf 3.19.6 which is incompatible.

When invoking manually to reproduce:

(env.test1) $ pip uninstall protobuf
$ pip3 install 'protobuf<3.20'
Collecting protobuf<3.20
  Using cached protobuf-3.19.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)
Installing collected packages: protobuf
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
mypy-protobuf 3.4.0 requires protobuf>=4.21.8, but you have protobuf 3.19.6 which is incompatible.
Successfully installed protobuf-3.19.6
(env.test1) $ protoc --python_out . --mypy_out . test.proto 
Traceback (most recent call last):
  File "/tmp/env.test1/bin/protoc-gen-mypy", line 5, in <module>
    from mypy_protobuf.main import main
  File "/tmp/env.test1/lib/python3.10/site-packages/mypy_protobuf/main.py", line 25, in <module>
    from . import extensions_pb2
  File "/tmp/env.test1/lib/python3.10/site-packages/mypy_protobuf/extensions_pb2.py", line 5, in <module>
    from google.protobuf.internal import builder as _builder
ImportError: cannot import name 'builder' from 'google.protobuf.internal' (/tmp/env.test1/lib/python3.10/site-packages/google/protobuf/internal/__init__.py)
--mypy_out: protoc-gen-mypy: Plugin failed with status code 1.

[nipunn1313]

Hi. Does it work if you run them in their own environments? To test my hypothesis.

In your system environment.

protoc --python_out .

In your virtualenv with mypy-protobuf installed

protoc --mypy_out .

Then see if your python out runs, and your pyi files typecheck.

My hypothesis is that protoc --python_out requires that the installed version of python-protobuf exactly matches that of protoc (which is reasonable).

mypy-protobuf, which releases much less often, requires its own version of python-protobuf, however the generated pyi files would still be valid.

First I want to see if this hypothesis is valid (with your help). If so, then we can determine what to do from there.

[aucampia]

A bit of a pile on, but it would be really nice to still have support for older versions of protobuf as tensorflow still only supports protobuf <4 [ref].

Not sure how difficult this would be though.

[nipunn1313]

Hi @aucampia - can you further describe what issue you're seeing?

I believe that mypy_protobuf should work by installing mypy_protobuf in a separate venv.

I think it's somewhat impossible to have mypy_protobuf work in the same venv with all possible versions of protobuf. mypy_protobuf has an internal dependency on protobuf in order to run.

The generated pyi files should be compatible with a broad variety of protobuf versions - you just need to generate them inside a venv that's more specific. Does that make sense?

If you have specific ideas on how to improve the status quo, I am interested.

[aucampia]

I believe that mypy_protobuf should work by installing mypy_protobuf in a separate venv.

That is an option, at the moment I'm using poetry to manage my dev tools though and I would really like to keep doing that but I will look at some other options.

[aucampia]

bit of a hack I guess, but I created a tox environment for mypy-protobuf now:

# https://tox.wiki/en/latest/config.html
# https://tox.wiki/en/latest/user_guide.html
[tox]
envlist =
isolated_build = True


[testenv:tool-mypy-protobuf]
skip_install = True
base = void
deps =
    -rdevtools/requirements-mypy-protobuf.txt

And wrote two wrapper scripts:

$ cat devtools/protoc-gen-mypy
#!/usr/bin/env bash

exec "$(tox -qq -e tool-mypy-protobuf exec -- python -c 'import shutil; print(shutil.which("protoc-gen-mypy"))')" "${@}"
$ cat devtools/protoc-gen-mypy_grpc 
#!/usr/bin/env bash

exec "$(tox -qq -e tool-mypy-protobuf exec -- python -c 'import shutil; print(shutil.which("protoc-gen-mypy_grpc"))')" "${@}"

And specified the paths to these wrapper scripts in buf.gen.yaml:

version: v1
managed:
  enabled: true
plugins:
  - name: python
    out: generated/proto
  - name: mypy
    path: devtools/protoc-gen-mypy
    out: generated/proto

[nipunn1313]

If there are specific versions of protobuf that we want to work with - I can try to document exactly which versions of mypy-protobuf work properly in the same venv with those versions of mypy-protobuf and try to have the mapping be sensible.

Does that seem reasonable?

It will always work in a venv, but maybe for these special case versions of protobuf that are in LTSes - we can make sure there's a known good setup.