bmc-tools
bmc-tools copied to clipboard
Fix memory usage for huge speed improvement
Multiple copies of the input file data were being created, due to data being copied in a loop. In Python, the slice of a bytes buffer does not create a reference but a copy. Hence this was slow and inefficient.
By adding a memoryview(), this eliminates the repeated copying of data making it much faster. On my system, a 100MB file, previously took 142 seconds to process, now only takes 21 seconds. So this is almost 7 times faster now, the actual speed variation though will depend on the file size, larger the file, better the performance (compared to earlier!). This is now better on memory and speed.
Looks fine to me.