Matthew Bryan

> Tests are now finally green. Well done! > One remaining question is: do we want to have the `atexit` at `Context`, or at executor level? There can be cases...

Just giving this a quick test, and in the environment I'm running (3.10.8 / distributed 2022.10.2) calling `DaskJobExecutor.close()` will cause a bunch of tracebacks on normal script / interpreter `exit()`....

From `dask.distributed` we see they register their own `atexit` handler for all `SpecClusters`: https://github.com/dask/distributed/blob/1d0701b51c1dc354fb22af9b27b8f722f02daf63/distributed/deploy/spec.py#L671-L677 ``` @atexit.register def close_clusters(): for cluster in list(SpecCluster._instances): if getattr(cluster, "shutdown_on_close", False): with suppress(gen.TimeoutError, TimeoutError): if...

/azp run libertem.libertem-data

One case to consider would be the output when there are many parallel workers; it would be too much to display 20+ progress bars at once. An option would be...

> As an alternative to multiple progress bars, it could be possible to have a single progress bar, but instead of dividing it into partitions, using the number of navigation...

> As an alternative to multiple progress bars, it could be possible to have a single progress bar, but instead of dividing it into partitions, using the number of navigation...

Some more comments on the memmap + memory usage warnings when running in a `DaskJobExecutor`. - Dask considers the sum of all accesses to a memmap object as part of...

So I did a bit of benchmarking to understand the behaviour of the custom reader, and the results confirm what we expect: the fewer passes we can do through the...

A (growing) list of Zenodo datasets which could be used to test this PR in CI: - https://zenodo.org/record/4307783 - provides a 1GiB C-ordered 4D-STEM file - https://zenodo.org/record/7041997 - Al_irradiated.dm4 -...