banyan-julia
banyan-julia copied to clipboard
banyan-team
A suite of familiar Julia APIs for bigger datasets with less cloud and lower costs.
When waiting for a job to get ready, we should: - [ ] Send/receive single-byte messages on all the SQS queues - [ ] Use SSM - [ ] Write/read...
Here is the output (when job is run with `return_logs=true`): ``` slurmstepd: error: *** JOB 3737 ON compute-dy-t3large-2 CANCELLED AT 2021-08-03T16:28:28 *** slurmstepd: error: *** STEP 3737.0 ON compute-dy-t3large-2 CANCELLED...
Right now we have a fixed maximum memory usage per worker based on the instance type - and because this isn't accurate we end up assuming that only 30% of...
Currently we have a set of configuration parameters which are stored in `banyanconfig.toml`, environment variables, or parameters to the `configure` function. - [ ] Make `configure` also store and maintain...
Currently, deleting a cluster deletes the S3 bucket associated with it. This is because deleting a cluster should clean up all resources associated with the cluster. However, since the S3...
We should really only end a running job if the program crashes on the executor or the user explicitly calls `destroy_job`. ## When scheduling fails On a call to a...
Banyan.jl should provide an `@return` macro so that job results are included in the jobs table on the dashboard.
There are several aspects of this: - `ScaleBy` constraint on a task level, PT level (for `Grouped`/`Blocked`), and for each split/merge/cast function - `Owned` constraint (for mutation) and also ensuring...
Sometimes S3FS takes time to sync up files or file metadata. On one occassion with the Black Scholes tests, we would have `HDF5.ishdf5` cause an error about the file that...
The returned error message: ``` Basic usage of BanyanDataFrames: Error During Test at /home/calebwin/Projects/banyan-julia/BanyanDataFrames/test/test_groupby_agg.jl:1 Got exception outside of a @test IOError(Base.IOError("read: connection timed out (ETIMEDOUT)", -110) during request(https://hcohsbhhzf.execute-api.us-west-2.amazonaws.com/dev/evaluate)) Stacktrace: [1]...
Metadata
Owner
Metadata
A suite of familiar Julia APIs for bigger datasets with less cloud and lower costs.