lithops icon indicating copy to clipboard operation
lithops copied to clipboard

Why doesn't the API follow that of concurrent.futures?

Open TomNicholas opened this issue 9 months ago • 3 comments

Why doesn't the API follow that of concurrent.futures? The lithops executors are clearly modelled after the executors in that standard library module, but they don't actually follow the API, nor do they inherit from the concurrent.futures.Executor ABC. The lithops future object doesn't inherit from concurrent.futures.Future either.

This matters - in particular the lithops .map() method returns a future whereas the concurrent.futures .map executes eagerly. This means downstream code cannot treat other executors such as concurrent.futures.ThreadPoolExecutor and lithops executors interchangably.

I can understand why you extended the API to add new methods (e.g. .map_reduce()), but why not follow the convention concurrent.futures set for the methods it does define? That standard library module predates this package by many years.

If this difference is intended, it would be nice to note why on the docs page on lithops futures, to avoid other people going down this rabbit hole too.

TomNicholas avatar Mar 25 '25 22:03 TomNicholas

There's also no .submit() method on lithops executors.

TomNicholas avatar Mar 25 '25 22:03 TomNicholas

Another difference: lithops wait() is not a top-level function, it's a staticmethod on the lithops executor class, unlike concurrent.futures.wait.

TomNicholas avatar Mar 25 '25 22:03 TomNicholas

To be more constructive, could we imagine adding a compliant executor interface to lithops, like I did here?

https://github.com/zarr-developers/VirtualiZarr/pull/349#discussion_r2013060304

TomNicholas avatar Mar 25 '25 23:03 TomNicholas