google-cloud-go icon indicating copy to clipboard operation
google-cloud-go copied to clipboard

spanner: reuse decoding buffers from row to row

Open hengfengli opened this issue 3 years ago • 2 comments

The following statement is from a discussion:

It is nice that spanner package has Decoder interface, but we don't use it because it is hard to reuse buffers (needed for conversion) from row to row. We have https://pkg.go.dev/go.chromium.org/luci/resultdb/internal/spanutil#Buffer which we allocate before a loop-over-rows and use for many rows (usage). Example:

	var b spanutil.Buffer
	err := spanutil.Query(ctx, st, func(row *spanner.Row) error {
		var key testVariantKey
		if err := b.FromSpanner(row, &key.testID, &key.variantHash); err != nil {
			return err
		}
		tvs[key] = struct{}{}
		return nil
	})

Here is a better one: https://chromium.googlesource.com/infra/luci/luci-go/+/HEAD/resultdb/internal/testresults/query.go#159 We compress some fields with zstd encoder, and reuse decoding buffer from row to row. Compression is implemented here: https://chromium.googlesource.com/infra/luci/luci-go/+/HEAD/resultdb/internal/spanutil/compression.go

hengfengli avatar May 04 '21 01:05 hengfengli

This is just for records. I am still not quite sure about the benefit of reusing the buffer. Need a further investigation.

hengfengli avatar May 04 '21 01:05 hengfengli

@rahul2393 @harshachinta : Please take a look at this.

ansh0l avatar Sep 22 '22 05:09 ansh0l