noobaa-core icon indicating copy to clipboard operation
noobaa-core copied to clipboard

NSFS buffer pool need to support small files efficiently

Open guymguym opened this issue 1 year ago • 0 comments

Environment info

  • NooBaa Version: 5.13
  • Platform: NA

Actual behavior

  1. buffer pool has buffers of a single size config.NSFS_BUF_SIZE which is default 8MB
  2. for small files this is very wastful. and allows rather low concurrency of requests being handled.

Expected behavior

  1. in most cases we know before the buffer allocation what is the object size.
  2. we should have several pools per different sizes and - I think of these limits:
  • 4 KB buffers * 1024 = 4MB total
  • 64 KB buffers * 256 = 12.8 MB total
  • 1 MB buffers * 128 = 128 MB total
  • 8 MB buffers * 128 = 1 GB total
  1. we should allocate from the smallest pool, and keep their sizes limited.
  2. we should also define per pool the min/max memory size, and set an expiry time on buffers that are unused for long time so that pools that don't get used very often will return back some space (up to the minimum size, which can be 1/2 of the max for example).

Steps to reproduce

  1. use high concurrency of small object operations.

More information - Screenshots / Logs / Other output

guymguym avatar Aug 20 '23 21:08 guymguym