Support for multi-threading conversion

Open stkw0 opened this issue 1 year ago • 2 comments

Currently it seems it only uses 1 thread to parse the files. It would be awesome if it could use all the available cores.

In my use case I have a single file of over 300M and with my 16-core CPU takes ~5m to convert the file. The only core is used is constantly at 100%, so it seems there is a lot of room for improvement.

Dec 16 '24 16:12 stkw0

Does arrow-rs support that? Is parsing CPU or IO bound?

Dec 16 '24 20:12 domoritz

I don't know if it supports it. The program overall is CPU bound

Dec 17 '24 10:12 stkw0