mapchete
mapchete copied to clipboard
optimize dask graph computation
collect multiple tasks into larger tasks in order not to overwhelm the scheduler:
- [ ] tile batches, no overviews (
concurrent.futures
,dask
) no dependencies - [ ] tile batches, with overviews (
dask
) dependencies between tile batches; use tile outputs within one tile batch - [ ] tile batches with preprocessing tasks (
dask
) dependencies between all tasks; use preprocessing output for follow up (tile) tasks