TypeError: Couldn't build proto file into descriptor pool: duplicate file name

[pangyoki]

What version of protobuf and what language are you using? Version: v3.19.0 Language: C++

What operating system (Linux, Windows, ...) and version? Linux

What runtime / compiler are you using (e.g., python version or gcc version) python3.7 gcc8.2

What did you do? compile proto file using make

What did you expect to see code runs normally

What did you see instead? After generating _pb2.py file, the method DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile() in py file reports an error.

Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs).

Anything else we should know about your project / environment works well in v3.20.0 and something is wrong in v4.21.1.

[ericsalo]

Impossible to determine whether this is a bug without a test case, would it be possible for you to post a simple one that reproduces the error?

[jmichalicek]

Same problem here, but the "duplicate" file is openapiv2.proto

Python 3.10.4 and python-protobuf 4.21.6 fails. Python 3.10.4 and python-protobuf 3.20.1 works

The relevant part of a stack trace seems to be the following, which makes me think maybe the problem (for me at least) lies in a 3rd party package (https://github.com/hack-edu/py_protoc_gen_openapiv2) due to being built with an older version of python-protobuf or older .proto files.

  File "/app/.venv/lib/python3.10/site-packages/protoc_gen_openapiv2/options/annotations_pb2.py", line 15, in <module>
    from protoc_gen_openapiv2.options import openapiv2_pb2 as protoc__gen__openapiv2_dot_options_dot_openapiv2__pb2
  File "/app/.venv/lib/python3.10/site-packages/protoc_gen_openapiv2/options/openapiv2_pb2.py", line 18, in <module>
    DESCRIPTOR = _descriptor.FileDescriptor(
  File "/app/.venv/lib/python3.10/site-packages/google/protobuf/descriptor.py", line 1028, in __new__
    return _message.default_pool.AddSerializedFile(serialized_pb)
TypeError: Couldn't build proto file into descriptor pool: duplicate file name (protoc-gen-openapiv2/options/openapiv2.proto)

[haberman]

If you try to load the same proto file twice (eg. protoc-gen-openapiv2/options/openapiv2.proto), the second one should succeed as long as it is identical to the previous file with that same name.

Since you are getting this error, it would suggest that the two copies of protoc-gen-openapiv2/options/openapiv2.proto that you are trying to load are slightly different.

If this worked with previous versions of the library, it might be because of slightly different logic when comparing the two files. Can you create a minimal repro for this? If so we could look into how the logic might have changed.

[gmazzola]

I created a minimal repro for this issue. It's located in this repository: gmazzola/protocolbuffers-issue-10158

The Dockerfile in that repository depends on the following pip freeze packages:

grpcio==1.50.0
grpcio-tools==1.50.0
protobuf==4.21.9

The repository contains two protobuf files:

• common.proto: messages used by multiple applications.
• store.proto: message used by a specific application. It imports and uses common.proto.

common.proto

syntax = "proto3";

package Example;

message StoreInfo {
}

message BrokenMessage {
    string first_field = 1;

    reserved 2;
    reserved "is_deprecated";
}

store.proto

syntax = "proto3";

import "common.proto";

package Example.STORE;

message QueryStore{
    Example.StoreInfo store_info = 1;
}

Note that store.proto does not use BrokenMessage — and yet BrokenMessage contributes to the failure somehow. If I drop BrokenMessage field 2, or drop BrokenMessage entirely, the build succeeds.

[eshikafe]

I am experiencing the same issue when I try to build some proto files.

python -m grpc_tools.protoc --version
libprotoc 3.21.6

python --version
Python 3.8.10

OS: Ubuntu (wsl)

~/bess-rs$ pip install --user protobuf grpcio scapy grpcio-tools
Requirement already satisfied: protobuf in /home/austin/.local/lib/python3.8/site-packages (4.21.7)
Requirement already satisfied: grpcio in /home/austin/.local/lib/python3.8/site-packages (1.49.1)
Requirement already satisfied: scapy in /home/austin/.local/lib/python3.8/site-packages (2.4.5)
Requirement already satisfied: grpcio-tools in /home/austin/.local/lib/python3.8/site-packages (1.49.1)
Requirement already satisfied: six>=1.5.2 in /home/austin/.local/lib/python3.8/site-packages (from grpcio) (1.16.0)
Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (from grpcio-tools) (45.2.0)

I am using the following command template to compile the .proto files: python -m grpc_tools.protoc --proto_path={srcdir} --python_out={outdir} --grpc_python_out={outdir} {proto_file}

> bessctl/bessctl
Traceback (most recent call last):
  File "bessctl/bessctl", line 51, in <module>
    from pybess.bess import *
  File "/home/austin/bess-rs/bessctl/../pybess/bess.py", line 61, in <module>
    from builtin_pb import bess_msg_pb2 as bess_msg
  File "/home/austin/bess-rs/pybess/builtin_pb/bess_msg_pb2.py", line 18, in <module>
    DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\x0e\x62\x65ss_msg.proto\x12\x07\x62\x65ss.pb\x1a\x19google/protobuf/any.proto\x1a\x0b\x65rror.proto\"\x0e\n\x0c\x45mptyRequest\".\n\rEmptyResponse\x12\x1d\n\x05\x65rror\x18\x01 \x01(\x0b\x32\x0e.bess.pb.Error\"A\n\x0fVersionResponse\x12\x1d\n\x05\x65rror\x18\x01 \x01(\x0b\x32\x0e.bess.pb.Error\x12\x0f\n\x07version\x18\x02 
...
\x01(\x0b\x32\x14.google.protobuf.Any\"!\n\x12PauseWorkerRequest\x12\x0b\n\x03wid\x18\x01 \x01(\x03\"\"\n\x13ResumeWorkerRequest\x12\x0b\n\x03wid\x18\x01 \x01(\x03\x62\x06proto3')
TypeError: Couldn't build proto file into descriptor pool: duplicate file name (bess_msg.proto)

There is no duplicate file name anywhere.

The .proto files are located here: https://github.com/eshikafe/bess-rs/tree/master/protobuf

[eshikafe]

Interestingly, when I downgrade the pip python packages, it gives me the expected result.

austin:~/bess-rs$ pip install --user protobuf==3.20.1 grpcio==1.46.0 grpcio-tools==1.46.0
Requirement already satisfied: protobuf==3.20.1 in /home/austin/.local/lib/python3.8/site-packages (3.20.1)
Requirement already satisfied: grpcio==1.46.0 in /home/austin/.local/lib/python3.8/site-packages (1.46.0)
Requirement already satisfied: grpcio-tools==1.46.0 in /home/austin/.local/lib/python3.8/site-packages (1.46.0)
Requirement already satisfied: six>=1.5.2 in /home/austin/.local/lib/python3.8/site-packages (from grpcio==1.46.0) (1.16.0)
Requirement already satisfied: setuptools in /home/austin/.local/lib/python3.8/site-packages (from grpcio-tools==1.46.0) (65.5.1)
austin:~/bess-rs$ bessctl/bessctl
Type "help" for more information.
Connection to localhost:10514 failed
Perhaps bessd daemon is not running locally? Try "daemon start".
<disconnected> $
<disconnected> $
<disconnected> $

[gnossen]

@haberman Friendly ping. This is impacting users of gRPC.

[haberman]

Minimal repro:

from google.protobuf import descriptor_pool
from google.protobuf import descriptor_pb2
from google.protobuf import text_format

text = """
name: "foo.proto"
message_type {
  name: "BrokenMessage"
  reserved_name: "is_deprecated"
}
"""

fdp = text_format.Parse(text, descriptor_pb2.FileDescriptorProto())
serialized = fdp.SerializeToString()

desc1 = descriptor_pool.Default().AddSerializedFile(serialized)
desc2 = descriptor_pool.Default().AddSerializedFile(serialized)

I think the problem is that we are not copying reserved names. There is even a TODO for it: https://github.com/protocolbuffers/upb/blob/f6307877d35b863ddfac1210494eedf25b136d75/upb/util/def_to_proto.c#L368 Implementing that will fix this issue.

[ericsalo]

A fix for this has been written, should be pushed tomorrow unless something unexpected happens.

[ericsalo]

Pushed. Closing.

[reetika-roy]

Which version of protobuf do we use to get this fix?

[haberman]

This fix is not released yet. We are planning a release for the week of Dec 5 (we're not pushing this week on account of Thanksgiving).

[ryan-thielke-ent]

Hello - has this fix been released yet? I think I'm seeing this behavior on 4.21.10.

[haberman]

No, the release work is currently in progress.

[haberman]

The fix has been released in Protobuf 4.21.11, which is now on PyPI.

[ryan-thielke-ent]

Thanks @haberman the fix is working for me!

[pascohen]

Hi I have a similar problem with timestamp.proto and the update to 4.21.11 does not seem to have solved the problem (Python 3.10/Linux WSL) If this is not the right place to ask a question please apologise and could you tell me where I should ask ? Thanks

[haberman]

@pascohen if you have a repro for a problem that occurs on 4.21.11, please open a new issue for it.