luceneutil icon indicating copy to clipboard operation
luceneutil copied to clipboard

Run nightly benchmarks in memory-constrained fashion?

Open jpountz opened this issue 1 year ago • 2 comments

We're currently working on better parallelizing I/O for Lucene queries. To be able to track improvements (and regressions!), it would be nice if we could run nightly benchmarks with a page cache size that is only a fraction of the total index size.

jpountz avatar May 22 '24 12:05 jpountz

For now we can use ramhog for this if needed. In the future we could specify cgroupv2 limits in the systemd service/timer for all the benchmarks, or run particular benchmarks with systemd-run, specifying lower limits.

rmuir avatar May 22 '24 15:05 rmuir

@rmuir also suggested using fincore --output-all /path/to/index/* to monitor how many hot/cold pages we see in the index while/after benchmarking.

mikemccand avatar May 22 '24 16:05 mikemccand