Topaz Train/Cross-Validation failed with AssertionError: Subprocess exited with status -9.

[PeterXTH]

Here is the error message. Topaz Cross Validation: Traceback (most recent call last): File “cryosparc_worker/cryosparc_compute/run.py”, line 93, in cryosparc_compute.run.main File “/data/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/run_topaz.py”, line 794, in run_topaz_wrapper_cross_validation assert len(tables) > 0, “All subsidiary training jobs failed or were killed.” AssertionError: All subsidiary training jobs failed or were killed. Topaz Train: Traceback (most recent call last): File “cryosparc_worker/cryosparc_compute/run.py”, line 93, in cryosparc_compute.run.main File “/data/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/run_topaz.py”, line 360, in run_topaz_wrapper_train utils.run_process(train_command) File “/data/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/topaz_utils.py”, line 98, in run_process assert process.returncode == 0, f"Subprocess exited with status {process.returncode} ({str_command})" AssertionError: Subprocess exited with status 1 (/primary/vari/software/topaz/default-cryosparc/topaz train --num-particles 200 --k-fold 2 --fold 0 --learning-rate 0.0002 --minibatch-size 128 --num-epochs 10 --method GE-binomial --slack -1 --autoencoder 0 --l2 0.0 --minibatch-balance 0.0625 --epoch-size …)

In the full log, it shows: MemoryError: Unable to allocate 144. GiB for an array with shape (19273333732,) and data type [('image', '<u4'), ('coord', '<u4')] Does anyone know how to solve this problem? Thank you so much. Tinghai