cubed
cubed copied to clipboard
cubed-dev
Bounded-memory serverless distributed N-dimensional array processing
There are a couple of parameters we can set: * `split_every`, which controls the number of blocks in each stage of tree reduce (defaults to 4, but set to 10...
> We should probably make Cubed store its intermediate data in a directory named `{CONTEXT_ID}/{compute_id}`, but that's a bit more work. _Originally posted by @TomNicholas in https://github.com/cubed-dev/cubed-benchmarks/pull/10#discussion_r1513284448_
The way [`stack` is currently implemented](https://github.com/cubed-dev/cubed/blob/be962a43befa024d51be2c378651c55d3e330f91/cubed/array_api/manipulation_functions.py#L277-L307) is slightly different to the other functions in that the number of arguments passed to [the blockwise function `_read_stack_chunk`](https://github.com/cubed-dev/cubed/blob/be962a43befa024d51be2c378651c55d3e330f91/cubed/array_api/manipulation_functions.py#L310-L311) is just one (the chunk...
This fixes `array_api_tests/test_linalg.py::test_vecdot` that was failing.
The logic in [index](https://github.com/cubed-dev/cubed/blob/fda2f1e7bedec5389cae00ee751a54076ccf28e3/cubed/core/ops.py#L400-L504) for manipulating an index selection is fairly complicated, particularly the part that removes any ellipsis. (See #400, #401, #402.) We could use [ndindex](https://quansight-labs.github.io/ndindex/index.html) to help with...
See #381. This relies on #396 to make it easier to cross-refer operation IDs in the debug output to the visualizations.
Lithops [configures Python logging](https://github.com/lithops-cloud/lithops/blob/55392b0f875651568542e339985866903555d550/lithops/utils.py#L176-L227) itself, which can be handy, but equally can make it hard to configure logging for other (non-Lithops) modules. For example, using `logging.basicConfig` in your code doesn't...
I got the first example to run :champagne: `python examples/lithops-add-asarray.py "s3://cubed-$USER-temp" cubed-runtime` But to get it to run on AWS (I don't have a Modal account) I did have to...
The algorithm for [launching backups](https://github.com/tomwhite/cubed/blob/929f52cad0642d7adb11fe6c2ff07e9b8741387f/cubed/runtime/backup.py) has not been tested very much. We should try it on a few runtimes and tune the parameters so it works reasonably well without user...
From @TomNicholas's code to generate the chart at https://xarray.dev/blog/cubed-xarray#cubed-results