GARD icon indicating copy to clipboard operation
GARD copied to clipboard

Break up processing into chunks

Open gutmann opened this issue 6 years ago • 0 comments

To permit larger runs in limited memory it would be nice to be able to specify a chunk size in time (or space) over which GARD would process datasets.

Performing this split in time might be the most obvious (and goes along with splitting up the output files #11 )

Performing the split in space could be more efficient because GARD needs a complete time series for each grid point in the training period regardless of how much time is being processed. Would this end up writing separate files for each subset that need to be recombined later or could it write just the subset it processed to an output file that contains the entire grid?

Ideally both should be options, but do they need to be?

Tradeoffs in implementation complexity and benefit should be assessed briefly.

gutmann avatar Jun 08 '18 17:06 gutmann