elastic icon indicating copy to clipboard operation
elastic copied to clipboard

Use Pool to store and reuse GZIPWriters

Open trams opened this issue 5 years ago • 0 comments

GZIPWriter are very expensive to create and leads to allocating a lot of memory. They are perfect for reusing (they have .reset method for this)

I profiled a web service inside my company and I noticed that creation of GZipWriter is responsible for 20% of memory allocation (in bytes).

Inspired by https://www.akshaydeo.com/blog/2017/12/23/How-did-I-improve-latency-by-700-percent-using-syncPool/ https://dzone.com/articles/memory-pooling-in-go-where-why-and-how

Could you help me to run unit tests?

I noticed there is a shell script ./run-es.sh which worked after I did sudo sysctl -w vm.max_map_count=262144 (my workstation is linux)

But go test . still fails for a lot of tests with

--- FAIL: TestSearchFilterPath (0.01s)
    setup_test.go:348: elastic: Error 400 (Bad Request): request [/elastic-test] contains unrecognized parameter: [include_type_name] [type=illegal_argument_exception]

I am quite puzzled

I also learned that I probably misuse defer (cause it is in the loop) Current proposal with additional parameter looks a bit ugly too. I will appreciate your feedback. This is my first pull request in go project (I am new to go)

trams avatar May 13 '20 08:05 trams