feature: Let users provide their own buffer to encode into

[benluddy]

Is your feature request related to a problem? Please describe.

The application calling Marshal has more information about the object it wants to marshal than this library does (e.g. what type is it, what is the observed 95th-percentile serialized size of similar objects, etc.). It can use this information to increase the utilization of encode buffers beyond what is possible with a shared buffer pool for all calls to Marshal.

Separately, an application may want to marshal an object directly to a buffer that it already has, without allocating a new byte slice. Encoder can be used today to encode to any io.Writer, but the object is first encoded to a temporary buffer (from the shared buffer pool) before being copied to the destination writer. The final copy from the temporary buffer to the destination writer is unnecessary, although it is probably cheaper than the many interface method calls it'd take to use io.Writer in place of concrete *encoderBuffer method calls throughout.

Describe the solution you'd like

A new MarshalToBuffer(v interface{}, *bytes.Buffer) error method on *encMode that is identical to the existing Marshal method, except that it encodes to the provided buffer instead of encoding to a buffer from the internal pool and copying to a freshly-allocated byte slice.

Expanding the method set of the EncMode interface is not backwards-compatible, so I propose that this method be added directly to *encMode who no change to that interface. Callers can reach the method by type asserting on an EncMode value. A new interface interface{ MarshalToBuffer(interface{}, *bytes.Buffer) error } could be declared for convenience. The interface might initially be documented as subject to change.

Describe alternatives you've considered

• User provides a buffer pool. This didn't seem helpful to me because the library can't offer many hints to the pool. It doesn't have enough knowledge about the thing it is marshaling, and it also can't compute the required buffer size in advance without an additional pass over the object.
• User provides a destination byte slice. Marshaling would either have to fail due to insufficient space (a pretty significant change, and hard to support as a second mode) or allocate a fresh byte slice as needed (might as well continue using a *bytes.Buffer if it will allocate to resize anyway).
• User provides an allocator. The user would be able to do things like reclaim bytes when the buffer is resized, but otherwise this has similar problems to having a user-provided buffer pool. I also suspect that only a small fraction of users would have a need to do this.

Additional context

[benluddy]

I wrote up an implementation of what I am proposing in https://github.com/fxamacker/cbor/pull/521.

[fxamacker]

Thanks Ben! :tada: Closed by #521.