schmitbp

I will add to this: I've tried the suggested OOM error (reducing batch size from default, which is 8 for 4x GPUs, to 4)

When I type --log_level debug I get the following: ######Isonet starts refining###### 2023-01-07 10:45:03.212576: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1 10:45:06, DEBUG [tpu_cluster_resolver.py:35] Falling back to TensorFlow client; we...

tensorflow.python.framework.errors_impl.ResourceExhaustedError: 5 root error(s) found. (0) Resource exhausted: OOM when allocating tensor with shape[2,32,32,32,128] and type bool on /job:localhost/replica:0/task:0/device:GPU:1 by allocator GPU_1_bfc [[node replica_1/model/dropout_14/dropout/GreaterEqual (defined at /threading.py:973) ]] Hint: If...

Hello, thank you for your help, I've attached here how the program was installed (as a .png) as well as the .yml file (saved as a .pdf) which contains all...

It's a consistent problem, anytime I start a reconstruction it starts the endless loop