MIC-DKFZ
nnUNetv2_find_best_configuration doesn't see spacing .json
First of all nnUNetv2 is amazing!
I have slight issue. I train on .tif images and per documentation provide both labels and images with individual .json files describing spacing.
nnUNetv2 "sees" this info and uses that info for preporcessing, training and validation. These spacing jsons appear in validation folder and are used for predicting individual folds at the end of training.
However they are not transfered when I run: nnUNetv2_find_best_configuration 1 -p nnUNetResEncUNetMPlans -c 3d_fullres -t nnUNetTrainer_100epochs
WARNING no spacing file found for segmentation /home/jbogomolovas/nnUNet_trained_models/Dataset001_YourDataset/nnUNetTrainer_100epochs__nnUNetResEncUNetMPlans__3d_fullres/crossval_results_folds_0_1_2_3_4/Task001_003.tif Assuming spacing (1, 1, 1).
Is there a way to fix this? I want the best ensemble and spacing varies a bit. I tried to copy spacing .jsons after crossval_results_folds_0_1_2_3_4 is created but it does not seem to "see" them.
+1 I have the same issue. The spacing .json located in ...nnUNet_raw/Dataset005_pooled/labelsTr is correctly recognized during training, but during validation at the end of training I get the warning:
WARNING no spacing file found for segmentation /path/ Assuming spacing (1, 1, 1).
It is using the segmentations found in ...nnUNet_preprocessed/Dataset005_pooled/gt_segmentations, where no spacing files have been generated or migrated over from ...nnUNet_raw/Dataset005_pooled/labelsTr. The JSON ...nnUNet_preprocessed/Dataset005_pooled/dataset_fingerprint.json does contain spacing information but seems it is not being parsed by nnUnet.
Is there a way to manually add spacing information for the validation?
Full log
2025-04-07 11:45:06.506231: Training done. 2025-04-07 11:45:06.798388: Using splits from existing split file: /path/nnUNet_preprocessed/Dataset005_pooled/splits_final.json 2025-04-07 11:45:06.804993: The split file contains 5 splits. 2025-04-07 11:45:06.806742: Desired fold for training: 0 2025-04-07 11:45:06.809000: This split has 13 training and 4 validation cases. 2025-04-07 11:45:06.810970: predicting 10 2025-04-07 11:45:06.825957: 10, shape torch.Size([1, 146, 212, 117]), rank 0 2025-04-07 11:45:43.181865: predicting 59 2025-04-07 11:45:43.215698: 59, shape torch.Size([1, 165, 126, 239]), rank 0 2025-04-07 11:45:49.055432: predicting 7 2025-04-07 11:45:49.459948: 7, shape torch.Size([1, 757, 212, 637]), rank 0 2025-04-07 11:48:46.920358: predicting 72 2025-04-07 11:48:46.985169: 72, shape torch.Size([1, 239, 126, 198]), rank 0 WARNING no spacing file found for segmentation /path/nnUNet_preprocessed/Dataset005_pooled/gt_segmentations/10.tif Assuming spacing (1, 1, 1). WARNING no spacing file found for segmentation /path/nnUNet_preprocessed/Dataset005_pooled/gt_segmentations/59.tif Assuming spacing (1, 1, 1). WARNING no spacing file found for segmentation /path/nnUNet_preprocessed/Dataset005_pooled/gt_segmentations/72.tif Assuming spacing (1, 1, 1). WARNING no spacing file found for segmentation /path/nnUNet_preprocessed/Dataset005_pooled/gt_segmentations/7.tif Assuming spacing (1, 1, 1). 2025-04-07 11:49:59.936994: Validation complete 2025-04-07 11:49:59.938455: Mean Validation Dice: 0.5448516308576765
Environment (mamba) info:
Name Version Build Channel _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 2_gnu conda-forge acvl-utils 0.2.5 pypi_0 pypi anyio 4.9.0 pypi_0 pypi aom 3.9.1 hac33072_0 conda-forge argparse 1.4.0 pypi_0 pypi attr 2.5.1 h166bdaf_1 conda-forge batchgenerators 0.25.1 pypi_0 pypi batchgeneratorsv2 0.2.3 pypi_0 pypi blosc2 3.0.0b4 pypi_0 pypi bzip2 1.0.8 h4bc722e_7 conda-forge ca-certificates 2025.1.31 hbcca054_0 conda-forge certifi 2025.1.31 pypi_0 pypi charset-normalizer 3.4.1 pypi_0 pypi connected-components-3d 3.23.0 pypi_0 pypi contourpy 1.3.1 pypi_0 pypi cuda-crt-tools 12.8.93 ha770c72_1 conda-forge cuda-cudart 12.8.90 h5888daf_1 conda-forge cuda-cudart_linux-64 12.8.90 h3f2d84a_1 conda-forge cuda-cuobjdump 12.8.90 hbd13f7d_1 conda-forge cuda-cupti 12.8.90 hbd13f7d_0 conda-forge cuda-nvcc-tools 12.8.93 he02047a_1 conda-forge cuda-nvdisasm 12.8.90 hbd13f7d_1 conda-forge cuda-nvrtc 12.8.93 h5888daf_1 conda-forge cuda-nvtx 12.8.90 hbd13f7d_0 conda-forge cuda-nvvm-tools 12.8.93 he02047a_1 conda-forge cuda-version 12.8 h5d125a7_3 conda-forge cudnn 9.8.0.87 h81d5506_0 conda-forge cusparselt 0.7.0.0 hcd2ec93_0 conda-forge cycler 0.12.1 pypi_0 pypi dav1d 1.2.1 hd590300_0 conda-forge dicom2nifti 2.6.0 pypi_0 pypi dynamic-network-architectures 0.3.1 pypi_0 pypi einops 0.8.1 pypi_0 pypi exceptiongroup 1.2.2 pypi_0 pypi fft-conv-pytorch 1.2.0 pypi_0 pypi filelock 3.18.0 pyhd8ed1ab_0 conda-forge fonttools 4.56.0 pypi_0 pypi freetype 2.13.3 h48d6fc4_0 conda-forge fsspec 2025.3.2 pyhd8ed1ab_0 conda-forge future 1.0.0 pypi_0 pypi giflib 5.2.2 hd590300_0 conda-forge gmp 6.3.0 hac33072_2 conda-forge gmpy2 2.1.5 py310he8512ff_3 conda-forge h11 0.14.0 pypi_0 pypi httpcore 1.0.7 pypi_0 pypi httpx 0.28.1 pypi_0 pypi idna 3.10 pypi_0 pypi imagecodecs 2025.3.30 pypi_0 pypi imageio 2.37.0 pypi_0 pypi importlib-resources 6.5.2 pypi_0 pypi jinja2 3.1.6 pyhd8ed1ab_0 conda-forge joblib 1.4.2 pypi_0 pypi kiwisolver 1.4.8 pypi_0 pypi lazy-loader 0.4 pypi_0 pypi lcms2 2.17 h717163a_0 conda-forge ld_impl_linux-64 2.43 h712a8e2_4 conda-forge lerc 4.0.0 h27087fc_0 conda-forge libabseil 20240722.0 cxx17_hbbce691_4 conda-forge libavif16 1.2.1 hbb36593_2 conda-forge libblas 3.9.0 31_h59b9bed_openblas conda-forge libcap 2.75 h39aace5_0 conda-forge libcblas 3.9.0 31_he106b2a_openblas conda-forge libcublas 12.8.4.1 h9ab20c4_1 conda-forge libcudss0 0.4.0.2 he55f5cd_2 conda-forge libcufft 11.3.3.83 h5888daf_1 conda-forge libcufile 1.13.1.3 h12f29b5_0 conda-forge libcurand 10.3.9.90 h9ab20c4_1 conda-forge libcusolver 11.7.3.90 h9ab20c4_1 conda-forge libcusparse 12.5.8.93 hbd13f7d_0 conda-forge libde265 1.0.15 h00ab1b0_0 conda-forge libdeflate 1.23 h4ddbbb0_0 conda-forge libffi 3.4.6 h2dba641_1 conda-forge libgcc 14.2.0 h767d61c_2 conda-forge libgcc-ng 14.2.0 h69a702a_2 conda-forge libgcrypt-lib 1.11.0 hb9d3cd8_2 conda-forge libgfortran 14.2.0 h69a702a_2 conda-forge libgfortran5 14.2.0 hf1ad2bd_2 conda-forge libgomp 14.2.0 h767d61c_2 conda-forge libgpg-error 1.51 hbd13f7d_1 conda-forge libheif 1.19.7 gpl_hc18d805_100 conda-forge libiconv 1.18 h4ce23a2_1 conda-forge libjpeg-turbo 3.0.0 hd590300_1 conda-forge liblapack 3.9.0 31_h7ac8fdf_openblas conda-forge libllvm20 20.1.1 ha7bfdaf_0 conda-forge liblzma 5.6.4 hb9d3cd8_0 conda-forge libmagma 2.8.0 h566cb83_2 conda-forge libnl 3.11.0 hb9d3cd8_0 conda-forge libnsl 2.0.1 hd590300_0 conda-forge libnvjitlink 12.8.93 h5888daf_1 conda-forge libnvjpeg 12.3.5.92 h97fd463_0 conda-forge libopenblas 0.3.29 pthreads_h94d23a6_0 conda-forge libpng 1.6.47 h943b412_0 conda-forge libprotobuf 5.28.3 h6128344_1 conda-forge libsqlite 3.49.1 hee588c1_2 conda-forge libstdcxx 14.2.0 h8f9b012_2 conda-forge libstdcxx-ng 14.2.0 h4852527_2 conda-forge libsystemd0 257.4 h4e0b6ca_1 conda-forge libtiff 4.7.0 hd9ff511_3 conda-forge libtorch 2.6.0 cuda126_generic_h4a15719_200 conda-forge libudev1 257.4 hbe16f8c_1 conda-forge libuuid 2.38.1 h0b41bf4_0 conda-forge libuv 1.50.0 hb9d3cd8_0 conda-forge libwebp-base 1.5.0 h851e524_0 conda-forge libxcb 1.17.0 h8a09558_0 conda-forge libxcrypt 4.4.36 hd590300_1 conda-forge libxml2 2.13.7 h0d44e9d_0 conda-forge libzlib 1.3.1 hb9d3cd8_2 conda-forge linecache2 1.0.0 pypi_0 pypi lz4-c 1.10.0 h5888daf_1 conda-forge markupsafe 3.0.2 py310h89163eb_1 conda-forge matplotlib 3.10.1 pypi_0 pypi mpc 1.3.1 h24ddda3_1 conda-forge mpfr 4.2.1 h90cbb55_3 conda-forge mpmath 1.3.0 pyhd8ed1ab_1 conda-forge msgpack 1.1.0 pypi_0 pypi nccl 2.26.2.1 ha44e49d_0 conda-forge ncurses 6.5 h2d0b736_3 conda-forge ndindex 1.9.2 pypi_0 pypi networkx 3.4.2 pyh267e887_2 conda-forge nibabel 5.3.2 pypi_0 pypi nnunetv2 2.6.0 pypi_0 pypi nomkl 1.0 h5ca1d4c_0 conda-forge numexpr 2.10.2 pypi_0 pypi numpy 2.2.4 py310hefbff90_0 conda-forge openjpeg 2.5.3 h5fbd93e_0 conda-forge openssl 3.4.1 h7b32b05_0 conda-forge optree 0.14.1 py310h3788b33_1 conda-forge packaging 24.2 pypi_0 pypi pandas 2.2.3 pypi_0 pypi pillow 11.1.0 py310h7e6dc6c_0 conda-forge pip 25.0.1 pyh8b19718_0 conda-forge pthread-stubs 0.4 hb9d3cd8_1002 conda-forge py-cpuinfo 9.0.0 pypi_0 pypi pybind11 2.13.6 pyh1ec8472_2 conda-forge pybind11-global 2.13.6 pyh415d2e4_2 conda-forge pydicom 3.0.1 pypi_0 pypi pyparsing 3.2.3 pypi_0 pypi python 3.10.16 he725a3c_1_cpython conda-forge python-dateutil 2.9.0.post0 pypi_0 pypi python-gdcm 3.0.24.1 pypi_0 pypi python-graphviz 0.20.3 pypi_0 pypi python_abi 3.10 6_cp310 conda-forge pytorch 2.6.0 cuda126_generic_py310_h9bb2754_200 conda-forge pytz 2025.2 pypi_0 pypi pyyaml 6.0.2 pypi_0 pypi rav1e 0.6.6 he8a937b_2 conda-forge rdma-core 56.0 h5888daf_0 conda-forge readline 8.2 h8c095d6_2 conda-forge requests 2.32.3 pypi_0 pypi scikit-image 0.25.2 pypi_0 pypi scikit-learn 1.6.1 pypi_0 pypi scipy 1.15.2 pypi_0 pypi seaborn 0.13.2 pypi_0 pypi setuptools 75.8.2 pyhff2d567_0 conda-forge simpleitk 2.4.1 pypi_0 pypi six 1.17.0 pypi_0 pypi sleef 3.8 h1b44611_0 conda-forge sniffio 1.3.1 pypi_0 pypi svt-av1 3.0.2 h5888daf_0 conda-forge sympy 1.13.3 pypyh2585a3b_103 conda-forge threadpoolctl 3.6.0 pypi_0 pypi tifffile 2025.3.30 pypi_0 pypi tk 8.6.13 noxft_h4845f30_101 conda-forge torchvision 0.21.0 cuda126_py310_h4459643_1 conda-forge torchvision-extra-decoders 0.0.2 py310h9a3ef1b_2 conda-forge tqdm 4.67.1 pypi_0 pypi traceback2 1.4.0 pypi_0 pypi triton 3.2.0 cuda126py310h50ec074_1 conda-forge typing-extensions 4.13.0 h9fa5a19_1 conda-forge typing_extensions 4.13.0 pyh29332c3_1 conda-forge tzdata 2025.2 pypi_0 pypi unittest2 1.1.0 pypi_0 pypi urllib3 2.3.0 pypi_0 pypi wheel 0.45.1 pyhd8ed1ab_1 conda-forge x265 3.5 h924138e_3 conda-forge xorg-libxau 1.0.12 hb9d3cd8_0 conda-forge xorg-libxdmcp 1.1.5 hb9d3cd8_0 conda-forge yacs 0.1.8 pypi_0 pypi zstd 1.5.7 hb8e6e7a_2 conda-forge
Hi, sorry for the late response. .tif images are not really intended for 3D medical data. Concider converting your data to niftis or nrrd. Other than that you need to add the spacing json custom...
You are absolutely right @constantinulrich. I convert my preprocessed lightsheet microscopy files to .nrrd and move on.