[zstd] test low memory options

[mostynb]

Let's see how this affects memory usage and performance.

Related to #473.

[mostynb]

@BoyangTian-Robinhood if you get a chance to try this out, please let me know how it goes.

[BoyangTian-Robinhood]

I did some tests, run some bazel test for same packages by cherry-pick this changes on top of [disk] drain any remaining data before Put returns and without cherry-pick this changes. (The numbers shows below is a rough number, I ran 2 times for each cases and just use round number here)

Summary

With this commit. Compare zstd with uncompressed:

zstd highest memory is 3 times higher than without zstd, and when it stable zstd memory is 1.74 times higher. For disk saving it is around 3.29 times saving with zstd.

Without this commit. Compare zstd with uncompressed:

zstd highest memory is 3.8 times higher than without zstd, and when it stable zstd memory is 2.3 times higher. For disk saving it is around 3.29 times saving with zstd.

Compare with this commit and without this commit zstd saving:

memory saving on peak is 1.04 times. memory saving on stable is 1.12 times. And disk has no change.

But it seems zstd also has memory leak issue even with grpc goroutines fix I can still see goroutines number stay increase by bazel clean each time

goroutine 112377 [chan receive, 4 minutes]:
github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder(0xc00f7b0e60, 0xc00a6f07b8)
	external/com_github_klauspost_compress/zstd/decoder.go:484 +0x230
created by github.com/klauspost/compress/zstd.(*Decoder).Reset
	external/com_github_klauspost_compress/zstd/decoder.go:202 +0x1ef

Detail

Start new service without anything there, memory is around 6MB.

Head is this commit (647f07b34095cd9c1dd87f32398f4c855c633b47)

With zstd options

memory peak is 1.54GB, stable is 268MB, disk is: 677MB

Still this commit by using --storage_mode=uncompressed

memory peak is 513MB, stable is 154MB, disk is: 2.23GB

Head is without this commit but with grpc goroutine fix (cc0a82a765e9c4d1cf1c35569dfae6a76118544c)

With using options :

memory peak is 1.6GB, stable is 300MB, disk is: 677MB

Still this commit by using --storage_mode=uncompressed

memory peak is 420MB, stable is 129MB, disk is: 2.23GB