Harsha

Setting `block_shape = None` runs successfully on the test data (whose shape is `(256, 256, 256)`. However, the process gets killed while running the same on the kwyk tfrecords despite...

Note: The successful run was on 2 A100s.

Could we not simply run `export CUDA_VISIBLE_DEVICES=0,1...` or `CUDA_VISIBLE_DEVICES=0,1.. python script.py` at the command line?

Verified suggestion- requested two GPUs, but running on only 1 using `CUDA_VISIBLE_DEVICES=1 python {script}.py`

Probably a combination of `du -hL /path/to/data` and [this](https://www.tensorflow.org/tutorials/load_data/tfrecord#:~:text=Note%3A%20In,O%20prefetching%20benefits.) might do?

Update: [More info](https://github.com/tensorflow/probability/issues/1774). Hopefully, this should be resolved in a few days once tf-probability is updated. As soon as the drivers on openmind are updated, there's no need to version...

Tried 2.16 and been running into a lot of other issues with the current code base. Will close this for now to be worked on at a later date.

Note: 1. tf=2.16 + tf-probability=0.24 + tf-keras 2. metrics should be provided as list 3. checkpoint filepath should have a `.keras` extension (this doesn't fail in nobrainer==1.2.0) 4. and so...

won't fail. all good. https://github.com/neuronets/nobrainer/blob/ed0d609333f9aea724c5fd45962c36dfd8a13d88/nobrainer/dataset.py#L241-L243

fails if `n_volumes` is not set which is the case when the `from_tfrecords` function is called.