opteryx icon indicating copy to clipboard operation
opteryx copied to clipboard

✨ Emit large files in blocks from readers

Open joocer opened this issue 1 year ago • 0 comments
trafficstars

As large files are being processed, they cause slowdowns

this may be because large amounts of memory are being allocated and deallocated at each step.

We should split large files into smaller blocks when processing to help reduce overheads associated with large blocks of memory.

joocer avatar May 18 '24 21:05 joocer