mapchete icon indicating copy to clipboard operation
mapchete copied to clipboard

optimize dask graph computation

Open ungarj opened this issue 3 years ago • 0 comments

collect multiple tasks into larger tasks in order not to overwhelm the scheduler:

  • [ ] tile batches, no overviews (concurrent.futures, dask) no dependencies
  • [ ] tile batches, with overviews (dask) dependencies between tile batches; use tile outputs within one tile batch
  • [ ] tile batches with preprocessing tasks (dask) dependencies between all tasks; use preprocessing output for follow up (tile) tasks

ungarj avatar Feb 18 '22 16:02 ungarj