clickhouse-connect icon indicating copy to clipboard operation
clickhouse-connect copied to clipboard

Improve Insert Performance

Open genzgd opened this issue 1 year ago • 4 comments

Insert data transformation is currently all in Python. This should be moved to C/Cython where expensive. It should also be possible to optimize Numpy/Pandas inserts by reading the Numpy buffer directly for int/float datatypes.

### Tasks
- [x] https://github.com/ClickHouse/clickhouse-connect/issues/201
- [ ] Direct copy of numpy arrays to output buffer
- [ ] Buffered bytearray destination for all writes to reduce allocations
- [ ] Datetime optimizations in C
- [ ] Decimal optimizations in C
- [x] Dynamically calculate block size

genzgd avatar Feb 01 '23 15:02 genzgd