framework
framework copied to clipboard
Optimize data/time performance
Overview
Currently, for data parsing and validation, we use native Python datetime objects (so during the validation we create them). It's a really big hit at performance as these objects are extremely slow. Tables having date/times might be X times slower than ones without them.
We need to investigate another options:
- fast libraries
- parsing default formats into plain dicts (this object will have
dt.to_native()function); this operation is super fast for strings with known positions structure likeyyyy-mm-dd