protobuf
protobuf copied to clipboard
protocolbuffers
MergeFrom error when using pure python message implementation
What version of protobuf and what language are you using? Version: 3.6.1 Language: Python 3.7
What operating system (Linux, Windows, ...) and version?
- Ubuntu 19.04 (GNU/Linux 5.0.0-13-generic aarch64)
- macOS 10.15.4
What runtime / compiler are you using (e.g., python version or gcc version) Python 3.7
What did you do? I have two proto file. In our actual project, these two files have a lot of content, only two message are pasted to illustrate the problem.
One is route.proto:
syntax = "proto3";
package a.b.c;
message Route {
string id = 1; // uuid, generated by minion
string dst = 2;
string via = 3;
string link_name = 4;
int32 table = 5;
}
Another is network.proto:
syntax = "proto3";
package a.b.c;
message GetDefaultGatewayResponse {
Route route = 1;
}
I run the following command in Interactive Python:
from rpc.network.network_pb2 import *
from rpc.network.route_pb2 import *
GetDefaultGatewayResponse(route=Route())
Then the following exception was thrown:
~/venv/lib/python3.7/site-packages/google/protobuf/internal/python_message.py in MergeFrom(self, msg)
1229 raise TypeError(
1230 "Parameter to MergeFrom() must be instance of same class: "
-> 1231 'expected %s got %s.' % (cls.__name__, msg.__class__.__name__))
1232
1233 assert msg is not self
But I add print(type(msg), cls) got same value is <class 'route_pb2.Route'> but id of that different.
However, there is no such problem with cpp message implementation. What did you expect to see
In [1]: from rpc.network.network_pb2 import *
...: from rpc.network.route_pb2 import *
...: GetDefaultGatewayResponse(route=Route())
Out[1]:
route {
}
What did you see instead?
In [1]: from rpc.network.network_pb2 import *
...: from rpc.network.route_pb2 import *
...: GetDefaultGatewayResponse(route=Route())
<class 'route_pb2.Route'> <class 'route_pb2.Route'>
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
~/venv/lib/python3.7/site-packages/google/protobuf/internal/python_message.py in init(self, **kwargs)
517 try:
--> 518 copy.MergeFrom(new_val)
519 except TypeError:
~/venv/lib/python3.7/site-packages/google/protobuf/internal/python_message.py in MergeFrom(self, msg)
1230 "Parameter to MergeFrom() must be instance of same class: "
-> 1231 'expected %s got %s.' % (cls.__name__, msg.__class__.__name__))
1232
TypeError: Parameter to MergeFrom() must be instance of same class: expected Route got Route.
During handling of the above exception, another exception occurred:
TypeError Traceback (most recent call last)
<ipython-input-1-5aaaa755c5d> in <module>
1 from rpc.network.network_pb2 import *
2 from rpc.network.route_pb2 import *
----> 3 GetDefaultGatewayResponse(route=Route())
~/venv/lib/python3.7/site-packages/google/protobuf/internal/python_message.py in init(self, **kwargs)
518 copy.MergeFrom(new_val)
519 except TypeError:
--> 520 _ReraiseTypeErrorWithFieldName(message_descriptor.name, field_name)
521 self._fields[field] = copy
522 else:
~/venv/lib/python3.7/site-packages/google/protobuf/internal/python_message.py in _ReraiseTypeErrorWithFieldName(message_name, field_name)
446
447 # re-raise possibly-amended exception with original traceback:
--> 448 six.reraise(type(exc), exc, sys.exc_info()[2])
449
450
~/venv/lib/python3.7/site-packages/six.py in reraise(tp, value, tb)
700 value = tp()
701 if value.__traceback__ is not tb:
--> 702 raise value.with_traceback(tb)
703 raise value
704 finally:
~/venv/lib/python3.7/site-packages/google/protobuf/internal/python_message.py in init(self, **kwargs)
516 new_val = field.message_type._concrete_class(**field_value)
517 try:
--> 518 copy.MergeFrom(new_val)
519 except TypeError:
520 _ReraiseTypeErrorWithFieldName(message_descriptor.name, field_name)
~/venv/lib/python3.7/site-packages/google/protobuf/internal/python_message.py in MergeFrom(self, msg)
1229 raise TypeError(
1230 "Parameter to MergeFrom() must be instance of same class: "
-> 1231 'expected %s got %s.' % (cls.__name__, msg.__class__.__name__))
1232
1233 assert msg is not self
TypeError: Parameter to MergeFrom() must be instance of same class: expected Route got Route. for field GetDefaultGatewayResponse.route
Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs).
Anything else we should know about your project / environment It is similar to another issue #5272 ?
I have the exact same issue on Raspberry Pi with python 3.7.3 and python 3.7.5. Somehow, it is does not happen on my Ubuntu laptop with Python 3.7.5. I use protobuf 3.12.2.
I run the exact same code on both but the isinstance check fails on Raspberry Pi.
The class type name is the same on both Raspberry Pi and my Ubuntu PC, but the class object id is different on Raspberry Pi.
I know python3 is compiled on ARM for Raspberry Pi and compiled for AMD64 on my laptop. Maybe there is a difference there.
met same problems too. The weird thing i met is that, when running the same code on x86 platform, everything works fine. While when running code on aarch platform like jetson tx2 or xavier nx, the problem occurs.
I had the exact same problem. In my case is simple, I used Linux Apline, the error occured. And if I use Ubuntu, everythin is fine. Python: 3.7 Protobuf: 3.11.3
There appears to be an assumption baked into the generator code for python at https://github.com/protocolbuffers/protobuf/blob/397d34ca0eb71e2af31881cccb6d8ffcdc3a0ee6/src/google/protobuf/compiler/python/python_generator.cc#L879
This explicitly references the module it expects the generated code to be available under. Combined with how the message classes are constructed dynamically via descriptors, this can result in 2 different object id's for the same message class definition appearing under two different modules.
Taking the example above for route.proto:
package a.b.c;
message Route {
string id = 1; // uuid, generated by minion
string dst = 2;
string via = 3;
string link_name = 4;
int32 table = 5;
}
and network.proto:
syntax = "proto3";
package a.b.c;
message GetDefaultGatewayResponse {
Route route = 1;
}
The generated python code will look something like: file: a/b/c/route_pb2.py
Route = _reflection.GeneratedProtocolMessageType('Route', (_message.Message,), {
'DESCRIPTOR' : _DEPLOYMENTINFO,
'__module__' : 'a.b.c.route_pb2'
# @@protoc_insertion_point(class_scope:a.b.c.Route)
})
_sym_db.RegisterMessage(Route)
file: a/b/c/network_pb2.py
from a.b.c import route_pb2 as a_dot_b_dot_c_dot_route__pb2
....
....
GetDefaultGatewayResponse = _reflection.GeneratedProtocolMessageType('GetDefaultGatewayResponse', (_message.Message,), {
'DESCRIPTOR' : _GETDEFAULTGATEWAYRESPONSE,
'__module__' : 'a.b.c.network_pb2'
# @@protoc_insertion_point(class_scope:a.b.c.GetDefaultGatewayResponse)
})
_sym_db.RegisterMessage(GetDefaultGatewayResponse)
Now this all works fine if the files are actually structured such as a is the project name and import a.b.c.network_pb2 is how it is imported.
However if you are placing the files elsewhere, or changing the prefix so that the import path is rpc.network.network_pb2 rather than what is defined as __module__ in the generated code, it appears you end up with two identical classes but different object id's from python's perspective.
Part of what happens is that when from rpc.network.network_pb2 import * is done, it imports a.b.c.route_pb2 rather than rpc.network.route_pb2 and this appears to result in one definition of the class being registered for the module a.b.c.route_pb2. When the subsequent import of rpc.network.route_pb2 occurs in the project code, the Route descriptor ends up being evaluated as a different python class even though they are identical. Because of the hard coding of the module attribute for the classes if you try to print __class__.__module__ you'll see the same, so it looks like you are importing the same module, but they are from different modules.
I uncovered this by a snippet a bit like the following to a unit test that was failing on trying to upgrade to python 3.9.1:
import sys
for mod in sys.modules:
if "route_pb2" in mod:
print(f"{mod}: {sys.modules[mod]}")
print(id(sys.modules[mod].Route))
combined with adding the line print(f"expected id: {id(cls)}, got id: {id(msg.__class__)}") to where the exception gets thrown from:
https://github.com/protocolbuffers/protobuf/blob/d16bf914bc5ba569d2b70376051d15f68ce4322d/python/google/protobuf/internal/python_message.py#L1318-L1322
I don't know enough of what is happening, I could see that there were 2 different imports appearing in sys.modules for the library I was working with, and the class id was different for each one. That allowed me to work out that if I imported the file that would register and then import using the module path hardcoded in the files would result in using the same ones in all locations.
import rpc.network.route_pb2
from a.b.c.route_pb2 import *
import rpc.network.network_pb2
from a.b.c.network_pb2 import *
It appears once rpc.network.route_pb2 is imported, there is an instance added to sys.modules as 'a.b.c.route' which is used by the other files that are part of the protobuf definition, so importing from a.b.c.route ensures I end up using the exact same and any isinstance checks will pass.
#1491, #4614 and #7470 appear to suggest there is a whole bunch of fun around import paths and python generated code. I suspect the divergence between what the generated code is assuming layout to be absolute and how usage of the resulting python code may assume that it can be placed under a different root to suit application layout or even be able to provide the generated protobuf code as a library is at the core of this issue. Having popped up various times and disappearing when import load orders end up switching slightly in python versions resulting in getting the needed class rather than the alternative definition.
I did some more digging and it turned out our local code had injected additional paths into sys.path. This is why the equivalent of from a.b.c.route_pb2 import * worked.
Given the important of paths in python module imports, and the assumptions about the base package name made by the generated code, which isn't made by the generated golang code, because all files in the same directory (ignoring test files for now) are in the same package which is defined as a simple path component, I think supporting relative path imports will make this class of error go away for use in python. Might also be nice if the __module__ value is correct, but I don't think based on my further digging, that the value stored there is anything more than cosmetic?
I have been facing the same issue, where the CopyFrom or MergeFrom works fine on ubuntu system by has the same error as metioned by OP on a raspberry pi. A workaround i found to work was something like this
default_response = GetDefaultGatewayResponse()
route = Route()
default_response.route.ParseFromString(route.SerializeToString())
Got the same issue on jetson-NX, The mergeFrom function is work on my x86 laptop, but run failed on getson-NX. Any solution?
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNKNOWN details = "Exception calling application: Parameter to MergeFrom() must be instance of same class: expected Value got Value."
The problem is related to isinstance inbuilt function of Python version <3.7 .
check what does isinstance function returns for the types of the objects.
Today I faced same issue on ARM 64 based hardware (Strange things was its working fine in Ubuntu X86 hardware) but @electrofelix workaround worked perfectly. Workaround was to just import these pb2 , pb2_grpc file from * instead of from a.b.*
Had the same issue, in my case, VSCode added automatically an import, while our codebase usually adds the path to sys paths, so full path is not required.
This resulted in code importing same proto file in 2 modes:
from blabla_pb2 import DefinedProtoObj
while in another file:
from module1.module2.blabla_pb2 import DefinedProtoObj
