PowerSystemDataModel icon indicating copy to clipboard operation
PowerSystemDataModel copied to clipboard

CSV time series source with window buffer

Open ckittl opened this issue 3 years ago • 1 comments

The current implementation of csv time series source loads the full time series at initialization. This can be very resource consuming, if the time series is long or you'd like to load a lot of time series at the same time.

The targeted implementation shall use a window buffer with configurable look ahead interval. This means, only chunks of the time series are read in and held in storage at the same time. However, this has the drawback, that this source only supports linear querying resp. no possibility to jump back in time.

ckittl avatar Sep 02 '21 10:09 ckittl

Regarding your PR I have a question: does this require linear time discrete equidistant time series? What happens with non-equidistant time series? Do we support them atm? Do we support them after your change?

Don't get me wrong I'm fine with both, but maybe it makes sense to have a time series requirement documentation in the docs that state the assumptions and requirements of time series processing.

In any way I think time series are the best argument for a database or similar that allows range queries ... 😅

johanneshiry avatar Jan 15 '22 08:01 johanneshiry