Background workers not alive / EOF error / no CUDA GPU available

[stang29]

Hi there,

I saw a few threads previously about some of these errors but none have seemed to work, was wondering if I could get some assistance with these errors! I've attached my requirements.txt file, the specs of the server with gpu I have to use (four 32Gb Tesla V100 = 128Gb total) and pasted the script I use to run nnunet. My images are 256^3 and I'm working in a python virtual environment. I've tried different combinations of setting those environment variables listed below but depending on which I use, I get one of either Background workers not alive error, no CUDA GPU available error (even though I check before I run the script that it can see / use cuda), EOF error or a strange keyboard interrrupt error. I'm not using a docker container, although if that is recommended please let me know.

I've been stuck on this for a while, any suggestions is appreciated!

cudainfo.txt

requirements.txt

Script:

export OMP_NUM_THREADS=1

export CUDA_VISIBLE_DEVICES=1

export TORCHDYNAMO_DISABLE=1

#export OPENBLAS_NUM_THREADS=1 #ulimit -u 1540468 #export TORCHDYNAMO_VERBOSE=1

export nnUNet_raw="/m/Researchers/usr/deeppvs/for_nnunet/nnUNet_raw" export nnUNet_preprocessed="/m/Researchers/usr/deeppvs/for_nnunet/nnUNet_preprocessed" export nnUNet_results="/m/Researchers/usr/deeppvs/for_nnunet/nnUNet_results"

nnUNetv2_plan_and_preprocess -d 001 -pl nnUNetPlannerResEncL --verify_dataset_integrity

export nnUNet_n_proc_DA=24

nnUNetv2_train 001 3d_fullres 5 -p nnUNetResEncUNetLPlans --npz

[vmiller987]

export nnUNet_n_proc_DA=0

Try setting this to zero. This solves it for me sometimes.

[vmiller987]

export nnUNet_n_proc_DA=0

Try setting this to zero. This solves it for me sometimes.

This is only an issue on Python 3.11 for me. Switching to Python 3.12 and 3.13 resolved this issue for me.

[stang29]

Thanks for the suggestion! I did see that as a possible solution, although some said it doesn't fully solve the issue (?). What env vars would you suggest using with that? Also, I am currently using python 3.12.