opteryx
opteryx copied to clipboard
✨ Emit large files in blocks from readers
trafficstars
As large files are being processed, they cause slowdowns
this may be because large amounts of memory are being allocated and deallocated at each step.
We should split large files into smaller blocks when processing to help reduce overheads associated with large blocks of memory.