lazy_dataset
lazy_dataset copied to clipboard
fgnt
lazy_dataset: Process large datasets as if it was an iterable.
TileDataset should be more efficient than concatenating the input dataset for large `repetitions` (in my case in the 1000s)
In [1] it is demonstrated to address the issue of the memory consumption, when multiprocessing is used. Although, we don't use multiprocessing (it's implemented, but threads are usually faster for...
The discussion in pull request #10 lead to this issue: The API should support method and function calls in the style of numpy, i.e. def op(self, (d1, d2, ,...), *args):...
The current version is a bit opaque in the sense that I do not know how to use `.shuffle()` in combination with `.prefetch()` or `.map(..., num_workers=10)`. Please provide some documentation...
