Provide custom options at runtime

[hugebdu]

@protobuf-ts/runtime-rpc/reflection-info#MethodInfo has options. Any plans to have it in protobuf-es as well?

Same applies to ServiceInfo

Thanks

[timostamm]

enum, enum fields, messages, message fields, and files can also have options, but I'd agree that service and rpc are some of the most interesting ones.

We've punted on this feature so far because some users don't want to ship their options in a bundle, so this can become complicated. Can I ask what you want to use it for?

[hugebdu]

we use our custom options for various needs.

• message fields annotated with validation options (both for API documentation and runtime validations)
• methods and services annotated with some runtime-related options (permissions etc)

and then there are google http annotations which we also use.

[hugebdu]

Also backtracking reference (from MethodInfo to ServiceInfo) is missing and it is really convenient, especially in protobuf-ts/runtime-rpc interceptors.

I'm trying to migrate gradually to protobuf-es + @grpc/grpc-js from protobut-ts stack.

[smaye81]

Thanks for the info @hugebdu. Appreciate the input and this is definitely something we have on our minds. As @timostamm mentioned though, it becomes complicated with some users who don't want their option values shipped as part of the bundle. So, making both camps happy is tricky and probably involves some sort of plugin option.

It's something we'd like to think through and get right, so I don't see us implementing this anytime soon but it's on our radar. In the meantime, though, you could implement your own plugin using Protobuf-ES' plugin framework. Custom options are available to you during code generation time (docs). Maybe that solution would work for you?

[hugebdu]

@smaye81 I have a workaround for now - I generate a binary descriptor as well and use it at runtime for all the option-bound logic.

Thanks

[timostamm]

Protocol Buffers v22.0 adds the feature option retention. This could potentially help us here to control which options should be included in generated code.

[hugebdu]

@smaye81 @timostamm

Custom options are available to you during code generation time (docs).

Is there any easy generic way to convert custom options to JSON (as appears in protobuf-ts generated code) without being aware of the extension number?

10x

[timostamm]

Is there any easy generic way to convert custom options to JSON (as appears in protobuf-ts generated code) without being aware of the extension number?

@hugebdu, support for extensions was just released in v1.7.0, making this very feasible.

Assuming you have a binary google.protobuf.FileDescriptorSet in a file set.binpb, the following script prints the options for every RPC as JSON:

import { createDescriptorSet, createRegistryFromDescriptors } from "@bufbuild/protobuf";
import { readFileSync } from "node:fs";

const set = createDescriptorSet(readFileSync("./set.binpb"));
const reg = createRegistryFromDescriptors(set);
for (const file of set.files) {
  for (const service of file.services) {
    for (const method of service.methods) {
      const options = method.proto.options;
      if (!options) {
        continue;
      }
      const json = options.toJsonString({ typeRegistry: reg, prettySpaces: 2 });
      console.log(`Options on ${method}: ${json}`);
    }
  }
}

For the test data from protobuf-ts, the output is:

Options on rpc spec.AnnotatedService.Get: {
  "idempotencyLevel": "NO_SIDE_EFFECTS",
  "[spec.rpc_bay]": 10,
  "[spec.rpc_foo]": true,
  "[spec.rpc_bar]": "hello",
  "[spec.rpc_baz]": 9,
  "[google.api.http]": {
    "get": "/v1/{name=messages/*}",
    "additionalBindings": [
      {
        "get": "xxx"
      },
      {
        "get": "yyy"
      }
    ]
  }
}

As you can see, the field idempotency_level is included, and since custom options are extensions, they are serialized with square brackets surrounding the name. But I think it should be pretty easy to iterate over the JSON object and remove the brackets and idempotencyLevel if you need exactly the same output that protobuf-ts generates.

We'll provide a mechanism to retrieve custom options at runtime as well. We'll keep this issue updated.

[hugebdu]

@timostamm king! 👑

[timostamm]

This is up next for us.

I've update the title of this issue to better reflect that we intend to support (custom) options at runtime for all Protobuf types.

[timostamm]

This is implemented in version 2 with the functions getOption and hasOption. See the documentation for details.