bplib icon indicating copy to clipboard operation
bplib copied to clipboard

bpcat should not stop/abort when running out of memory

Open jphickey opened this issue 2 years ago • 0 comments

The "bpcat" test program by default reserves 1MB of memory for temporary/ephemeral bundle storage. It will also send data as fast as it can read() from the file, which is generally faster than the rate at which it will be stored in persistent storage.

When sending a large file (greater than 1MB) this causes the temporary/ephemeral storage to be used up, which in turn causes bplib_send() to return an error at some point. This is a temporary error, but bpcat does not currently handle this and resume, it just aborts the transfer.

  • consider making bplib_send() also employ the timeout when obtaining ephemeral buffers (this may be a nice to have but not really critical)
  • use a separate error code (other than generic BP_ERROR) to indicate the temporary out of memory condition. This would indicate that the caller should not abort, but try again later.
  • Have bpcat throttle back and resume when getting the out of memory error, do not abort.

jphickey avatar Jul 20 '22 17:07 jphickey