storage: pool grpc writer buffers for small objects

[fpetkovski]

The gRPCWriter allocates a new byte slice for each new object, which causes a lot of GC pressure for use cases where many small objects are uploaded to a bucket.

This commit adds a bytes pool so that allocations can be amortized for objects smaller than 256K, which is the minimum chunk size.

Benchmark results

name          main time/op    branch time/op    delta
GRPCWrite-11    45.5µs ±14%    36.7µs ± 2%  -19.44%  (p=0.000 n=9+9)

name          main alloc/op   branch alloc/op   delta
GRPCWrite-11     282kB ± 0%      20kB ± 0%  -92.94%  (p=0.000 n=10+9)

name          main allocs/op  branch allocs/op  delta
GRPCWrite-11       264 ± 0%       263 ± 0%   -0.38%  (p=0.000 n=10+9)

[google-cla[bot]]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[quartzmo]

/cc @BrennaEpp

[fpetkovski]

Thanks for the review @tritone. Here are my thoughts:

Using 256k chunk size leads to larger uploads being slower. Should we think about pooling for larger buffers as well?

The chunk size is configurable and 256K is the smallest size the library allows. This change only pools those buffers, and everything larger will get directly allocated as before. So there shouldn't be a performance regression for larger chunk sizes.

Perhaps a more flexible approach would be to allow the user to pass in a buffer to the Writer? That way the end user could handle the pooling based on their workload profile.

This would be a great API since it allows the client to control allocations.

We'd also need to run benchmarking against the real service to see how this does in practice.

I wasn't able to do this because we use this library as a transitive dependency, and it wasn't trivial to use a replace in go.mod due to the difference between the module name and github repository url.