framework icon indicating copy to clipboard operation
framework copied to clipboard

Optimize data/time performance

Open roll opened this issue 3 years ago • 0 comments

Overview

Currently, for data parsing and validation, we use native Python datetime objects (so during the validation we create them). It's a really big hit at performance as these objects are extremely slow. Tables having date/times might be X times slower than ones without them.

We need to investigate another options:

  • fast libraries
  • parsing default formats into plain dicts (this object will have dt.to_native() function); this operation is super fast for strings with known positions structure like yyyy-mm-dd

roll avatar Nov 23 '22 08:11 roll