ion-python
ion-python copied to clipboard
C extension should allows user to config read bytes size for performance reason
Refer to https://github.com/amzn/ion-python/pull/216
The performance depends on the size of files we are going to read. On the one hand, if the file is really small but we maintain a large buffer, it’s not efficient and slower in performance. On the other hand, if the file is really large but we only read a small number of bytes, say 32 bytes, each time it’s also very slow. I use 32*1024 first and we should eventually allow user to configure this in the future.
If we can control the size of read buffer, users will see around 5-10 % performance improvement (A brief benchmark data for sample file I tested).