ion-python icon indicating copy to clipboard operation
ion-python copied to clipboard

C extension should allows user to config read bytes size for performance reason

Open cheqianh opened this issue 2 years ago • 0 comments

Refer to https://github.com/amzn/ion-python/pull/216

The performance depends on the size of files we are going to read. On the one hand, if the file is really small but we maintain a large buffer, it’s not efficient and slower in performance. On the other hand, if the file is really large but we only read a small number of bytes, say 32 bytes, each time it’s also very slow. I use 32*1024 first and we should eventually allow user to configure this in the future.

If we can control the size of read buffer, users will see around 5-10 % performance improvement (A brief benchmark data for sample file I tested).

cheqianh avatar Aug 18 '22 23:08 cheqianh