Caleb Winston

Job sometimes crashes with Open-MPI-related message

7

Here is the output (when job is run with `return_logs=true`): ``` slurmstepd: error: *** JOB 3737 ON compute-dy-t3large-2 CANCELLED AT 2021-08-03T16:28:28 *** slurmstepd: error: *** STEP 3737.0 ON compute-dy-t3large-2 CANCELLED...

Accurately estimate the memory available on the executor

1

Right now we have a fixed maximum memory usage per worker based on the instance type - and because this isn't accurate we end up assuming that only 30% of...

Add current job ID and current cluster name to banyanconfig.toml

2

Currently we have a set of configuration parameters which are stored in `banyanconfig.toml`, environment variables, or parameters to the `configure` function. - [ ] Make `configure` also store and maintain...

We should really only end a running job if the program crashes on the executor or the user explicitly calls `destroy_job`. ## When scheduling fails On a call to a...

Return job results with @return

2

Banyan.jl should provide an `@return` macro so that job results are included in the jobs table on the dashboard.

There are several aspects of this: - `ScaleBy` constraint on a task level, PT level (for `Grouped`/`Blocked`), and for each split/merge/cast function - `Owned` constraint (for mutation) and also ensuring...

Sometimes S3FS takes time to sync up files or file metadata. On one occassion with the Black Scholes tests, we would have `HDF5.ishdf5` cause an error about the file that...

HTTP POST request to compute times out

9

The returned error message: ``` Basic usage of BanyanDataFrames: Error During Test at /home/calebwin/Projects/banyan-julia/BanyanDataFrames/test/test_groupby_agg.jl:1 Got exception outside of a @test IOError(Base.IOError("read: connection timed out (ETIMEDOUT)", -110) during request(https://hcohsbhhzf.execute-api.us-west-2.amazonaws.com/dev/evaluate)) Stacktrace: [1]...

When a call to a `read_*` or `write_*` (e.g., `read_hdf5` or `write_parquet`) function is made, we are simply assigning a `Location` to a `Future`. As part of constructing a `Location`...

This could be happening because of several reasons: - Running out of memory because GC.gc() calls not placed strategically (not really an issue any more) - Running out of memory...