Cubed

Note: this is a proof-of-concept, and many things are incomplete or don't work.

Bounded-memory serverless distributed N-dimensional array processing

Cubed is a distributed N-dimensional array library implemented in Python using bounded-memory serverless processing and Zarr for storage.

• Implements the Python Array API standard (see coverage status)
• Guaranteed maximum memory usage for standard array functions
• Follows Dask Array's chunked array API (map_blocks, rechunk, apply_gufunc, etc)
• Zarr for persistent and intermediate storage
• Multiple serverless runtimes: Python (in-process), Lithops, Modal, Apache Beam
• Integration with Xarray via cubed-xarray

Documentation

Articles

Cubed: Bounded-memory serverless array processing in xarray

Optimizing Cubed