bmc-tools icon indicating copy to clipboard operation
bmc-tools copied to clipboard

Fix memory usage for huge speed improvement

Open ydkhatri opened this issue 2 years ago • 1 comments

Multiple copies of the input file data were being created, due to data being copied in a loop. In Python, the slice of a bytes buffer does not create a reference but a copy. Hence this was slow and inefficient.

By adding a memoryview(), this eliminates the repeated copying of data making it much faster. On my system, a 100MB file, previously took 142 seconds to process, now only takes 21 seconds. So this is almost 7 times faster now, the actual speed variation though will depend on the file size, larger the file, better the performance (compared to earlier!). This is now better on memory and speed.

ydkhatri avatar Dec 01 '23 22:12 ydkhatri

Looks fine to me.

Schramp avatar Dec 08 '23 22:12 Schramp