Allow plain JS object repeated fields to use toArray() method

[glasser]

We have several use cases for wanting to build up complex nested protobuf objects incrementally over time before serializing where representing an array with a repeated field is not helpful during the connstruction process:

• Protobuf maps only support integral and string as keys, so we sometimes have fields that are repeated message objects that represent a complex key and a value. In order to build up this field over time in a non-quadratic way, it would be nice to be able to use an object containing (say) a Map, which can be converted to a list of messages once at encoding time.

• We have a repeated int64 field that represents a histogram (each number represents a number of events that fell in a certain duration bucket). This histogram generally contains large stretches of 0s so we have a compressed encoding where we use negative numbers to encode stretches of zeroes. We want to only convert from the uncompressed to compressed encoding when we're ready to encode.

These challenges can be solved by creating a hierarchy of classes very similar to your protobuf hierarchy and converting to protobufs manually when it's time to encode, but this is a lot of effort to create and maintain. One could also use JS proxies to trap indexing and trigger some sort of just-in-time conversion to array, but that's annoying to write and probably not particularly efficient.)

Instead, this change allows plain JS objects to contain non-Arrays for repeated fields as long as they have a toArray method. Implementations of encode, fromObject, and verify are updated to call toArray if necessary. The jsdoc in static and static-module targets reference this in field typings where appropriate, which is processed correctly by pbts.

This change only looks for toArray on fields that explicitly contain a [(js_use_toArray) = true] option. This is because of unbenchmarked guesses that maybe leaving the toArray check out will be faster for fields that don't need it. This feature would be simpler to use if toArray was just hardcoded, though.

Note that protobufjs does not require you to declare options, but you'll want to declare the option if you want to use your proto file with the official protobuf compiler. Something like this should do the trick:

import "google/protobuf/descriptor.proto";

extend google.protobuf.FieldOptions {
  bool js_use_toArray = 50505;
}

Note that this import will cause protobuf.js to include thousands of lines of generated code in your output; see #1300.

Example TypeScript usage:

# report.proto
syntax = "proto3";
message Report {
  repeated Entry entries = 1 [(js_use_toArray)=true];
}
message Entry {
  string key1 = 1;
  string key2 = 2;
  string value = 3;
}

# run.ts
import {IReport, IEntry, Report, Entry} from "./report";

class MyEntries {
    private readonly entries = new Map<string, Entry>();
    public set(key1: string, key2: string, value: string) {
        const entry = new Entry({key1, key2, value});
        const key = JSON.stringify({key1, key2});
        this.entries.set(key, entry);
    }
    public toArray(): IEntry[] {
        return Array.from(this.entries.values());
    }
}

const r: IReport = new Report({entries: new MyEntries()});
const nestedEntries = r.entries as MyEntries;
nestedEntries.set("x", "y", "v");
nestedEntries.set("x", "y", "w");
nestedEntries.set("a", "b", "c");

const roundTrip = Report.decode(Report.encode(r).finish());
console.log(roundTrip);

[glasser]

I'm noticing that this repository seems to be back under development! Is there any interest in taking this PR (if I rebase it)? We've been using it in our fork.

[glasser]

4.5 years later I'm happy to reiterate my offer to rebase the PR if there's any chance of it being merged. The Apollo Server project had to fork protobuf.js many years ago in order to have this performance benefit but we'd certainly be happy to unfork.