Tommaso Comparin

Hi there, quick question: Is this issue considered as closed? The corresponding PR https://github.com/dask/dask/pull/7379 was not brought forward and -if I understand correctly- it would force the whole execution before...

Thanks for your comment. I confirm that adding `gc.collect()` here and there (both within `run_cellpose` function, and especially right after each call to this function within the loop) does not...

For your info: We are planning to implement something slightly related, that is, making a new `SlurmExecutor.map()` method that executes N tasks by only submitting M

I fully agree about keeping clusterfutures simple (we are working on this feature due to a specific constraint), and this likely leads to the one-task-one-job approach. > If a job...

The standard `Executor.map` method has an important difference from `Executor.submit`, in that at the end of `map` the list of Futures that were created (one for each item of the...

What follows is the first workaround we could come up with. We modify the `_completion` method as follows: ```python def _completion(self, jobid): """Called whenever a job finishes.""" with self.jobs_lock: fut,...

Thanks for the details. Using `set_running_or_notify_cancel` looks like a nice solution to this problem, indeed. I will try and implement it, and report back here.

My bad, I had not read your message carefully. After some more thoughts, I agree that just skipping the `_completion` call-back function if the future is cancelled does work as...

(I can obviously prepare a PR with the small change I implemented, or something similar, if that's useful)

I think that the warning in the previous comment (by @bert2002) is unrelated to python 3.12; it's rather a `passlib` issue with recent `bcrypt` versions (see https://foss.heptapod.net/python-libs/passlib/-/issues/190). --- > Unfortunately,...