protobuf.js
protobuf.js copied to clipboard
protobufjs
Unable to use Google well known types
protobuf.js version: 6.8.6
When I attempt to encode a message from a Google well known type (like google.protobuf.Struct) I get an empty field in the decoded message IF my path includes google/protobuf.
Is this a bug or do we need to provide the path in some other way. I assume the problem starts when loading the commons wrapper here.
const protobuf = require('protobufjs');
let root = new protobuf.Root();
const protoFilePath = '<path to a file which imports google.protobuf.Struct>';
const protoRoot = .............;
root.resolvePath = function (origin, target) {
// ignore the same file
if (target == protoFilePath) {
return protoFilePath;
}
// Resolved target path for the import files
return protoRoot + target;
};
const struct = {
fields: {
label: {
struct_value: {
fields: {
another_label: {
number_value: 10
}
}
}
}
}
};
root = root.loadSync(protoFilePath);
const MyType = root.lookupType('MyType');
const msg = MyType.create({
struct_message: struct
});
const buffer = MyType.encode(msg).finish();
console.log('Decoded message', MyType.decode(buffer));
Decoded message: Struct { fields: { label: Value {} } } }
Note: this was working with protobuf.js version 5.
Bump: @dcodeIO, is this a bug or is there another way of using these types?
This might be due to protobuf.js implicitly converting field names from underscore_notation to camelCaseNotation since v6 (like google's js reference js impl does). You can try
root.loadSync(protoFilePath, { keepCase: true })
to retain underscore notation or change the struct contents accordingly.
What happens if you do:
const struct = {
fields: {
label: {
kind: "struct_value",
struct_value: {
fields: {
another_label: {
kind: "number_value",
number_value: 10
}
}
}
}
}
};
Just edited this to use the string field names, can you try again? I believe it looks for the virtual oneof field there to determine which type of value is provided.
Hmm, what happens if you also do
const msg = MyType.fromObject({
struct_message: struct
});
If that also doesn't work, can you provide the .proto definition for MyType?
I am using these protos: https://github.com/restorecommerce/protos. The proto definition is in io/restorecommerce/resource_base.proto (ReadRequest). It has a filter field which is of type google.protobuf.Struct. It has always worked with grpc-node which fallsback to protobufjs v5 when loading proto definitions but not if we specifically try v6.
Thanks. Seems that there is an issue with keepCase in conjunction with the struct type, because src/common.js explicitly defines it in camelCase notation, which isn't converted back when keepCase = true.
This appears to work:
var root = new protobuf.Root();
root.load("google/protobuf/struct.proto", function() {
protobuf.parse(`syntax = "proto3"; message MyType { google.protobuf.Struct filter = 1; }`, root);
var MyType = root.MyType;
var myType = MyType.fromObject({
filter: {
fields: {
number: {
numberValue: 10
},
struct: {
structValue: {
fields: {
number: {
numberValue: 20
}
}
}
}
}
}
});
console.log(myType);
});
Yes, this is a bug affecting keepCase. A proper fix could be to move field renaming out of the parser into construction of the reflection structure, in turn defining all of common.js in underscore notation.
Btw, what is the use of those custom wrappers for Google well known types? Wouldn't they be parsed correctly in the "standard" way?
Custom wrappes (from wrappers.js) provide a way to work with instances of Value, for example, in a more natural way. For example, if you provide an object that has a Value value of number to fromObject, it will automatically "wrap" it in a proper Value type. Similarly it unwraps to a single value from a Value instance in toObject. At least that's the intention.
Any workaround or update here? I'm blocked not being able to load protos importing google/protobuf/empty.proto. I believe this is the same issue.
This is an issue for us trying to use the standard wrappers, they are not valid JSON for other JSON parsers that handle the standard wrappers according to the JSON documentation here: https://developers.google.com/protocol-buffers/docs/proto3#json
I'm still experiencing this issue
Error: no such type: google.protobuf.Empty
Isn't this fixed with f61b4bc517b10c587ada6bd8e0799cccfef0dc51? If so, could you please release a new minor version?
Maybe related/duplicates of this issue:
- #1061
- #1098
- #1266
I'm currently patching frontend code to manually parse to JSON and back. :(
And to adjust type typescript typings for the message containing messages of type Struct, I'd create a new interface which extends the generated pb message type and replace the field that I'm manually mapping.
// This is how the generated TS type looks like
export namespace MyMessage {
export type AsObject = {
foo: string,
bar: ptypes_struct_struct_pb.Struct.AsObject, // :(
}
}
// Patch the generated types by creating a new type that extends the generated type and override field(s) which require manual mapping.
export interface MyMessageAsObject extends Omit <MyMessage.AsObject, 'bar'> {
bar: /* Your custom type here */
}
The methods below also appear to be incompatible.
https://www.npmjs.com/package/google-protobuf https://github.com/protocolbuffers/protobuf/tree/master/js
import { ListValue, Struct, Value } from 'google-protobuf/google/protobuf/struct_pb';
ListValue.fromJavaScript(['foo', 'bar']).toObject();
Struct.fromJavaScript({ foo: 'bar' }).toObject();
Value.fromJavaScript('foobar').toObject();
protobufjs: 6.9.0
We checked the issue again with latest protobufjs version 6.9.0, but still we are not able to encode google.protobuf.Struct messages.
We are still using grpc.load which falls back to protobufjs version 5 and grpc.load is deprecated.
This issue is holding us back to use grpc\protoloader. Any updates on this issue ?