google-cloud-go
google-cloud-go copied to clipboard
spanner: reuse decoding buffers from row to row
The following statement is from a discussion:
It is nice that spanner package has Decoder interface, but we don't use it because it is hard to reuse buffers (needed for conversion) from row to row. We have https://pkg.go.dev/go.chromium.org/luci/resultdb/internal/spanutil#Buffer which we allocate before a loop-over-rows and use for many rows (usage). Example:
var b spanutil.Buffer
err := spanutil.Query(ctx, st, func(row *spanner.Row) error {
var key testVariantKey
if err := b.FromSpanner(row, &key.testID, &key.variantHash); err != nil {
return err
}
tvs[key] = struct{}{}
return nil
})
Here is a better one: https://chromium.googlesource.com/infra/luci/luci-go/+/HEAD/resultdb/internal/testresults/query.go#159 We compress some fields with zstd encoder, and reuse decoding buffer from row to row. Compression is implemented here: https://chromium.googlesource.com/infra/luci/luci-go/+/HEAD/resultdb/internal/spanutil/compression.go
This is just for records. I am still not quite sure about the benefit of reusing the buffer. Need a further investigation.
@rahul2393 @harshachinta : Please take a look at this.