zed icon indicating copy to clipboard operation
zed copied to clipboard

zed load: +36% mem use, -15% load time (commit 7dfef14)

Open philrz opened this issue 3 years ago • 0 comments

Through a set of iterative runs as described at https://github.com/brimdata/zed-perf-testing/tree/main/memory-load, I've observed that at Zed commit commit 7dfef14 (associated with #3482), the zed load of the fdns-a.zeek.gz data set saw an increase of 36% in reported go_memstats_sys_bytes in exchange for a 15% reduction in the time for the load to complete.

Here's a macro-level plot of the trend across many commits:

image

In a team discussion, @nwt was unsurprised that the perf improvement came with some memory cost. This issue has been opened to log the observation and consider if the trade is acceptable or if there's ways to potentially maintain the better perf while using less memory.

philrz avatar Oct 04 '22 22:10 philrz