PowerSystemDataModel
PowerSystemDataModel copied to clipboard
CSV time series source with window buffer
The current implementation of csv time series source loads the full time series at initialization. This can be very resource consuming, if the time series is long or you'd like to load a lot of time series at the same time.
The targeted implementation shall use a window buffer with configurable look ahead interval. This means, only chunks of the time series are read in and held in storage at the same time. However, this has the drawback, that this source only supports linear querying resp. no possibility to jump back in time.
Regarding your PR I have a question: does this require linear time discrete equidistant time series? What happens with non-equidistant time series? Do we support them atm? Do we support them after your change?
Don't get me wrong I'm fine with both, but maybe it makes sense to have a time series requirement documentation in the docs that state the assumptions and requirements of time series processing.
In any way I think time series are the best argument for a database or similar that allows range queries ... 😅