omnipose
omnipose copied to clipboard
Published
20 hours ago •
kevinjohncutler
kevinjohncutler
3D training with ValueError(msg)
Hi @kevinjohncutler,
I'm trying to train a new model using your new 3D implementation, but I'm having the follow error:
python -m omnipose --train --use_gpu --dir ./omni/omnipose_training --mask_filter _masks --n_epochs 4000 --pretrained_model None --learning_rate 0.1 --save_every 50 --save_each --verbose --dim 3 --RAdam --batch_size 4 --diameter 0
2022-11-29 13:26:02,817 [INFO] WRITING LOG OUTPUT TO /Users/mcruz/.cellpose/run.log
log file /Users/mcruz/.cellpose/run.log
2022-11-29 13:26:02,820 [INFO] ** TORCH GPU version installed and working. **
2022-11-29 13:26:02,820 [INFO] >>>> using GPU
Omnipose enabled. See Omnipose repo for licencing details.
2022-11-29 13:26:02,820 [INFO] Training omni model. Setting nclasses=4, RAdam=True
2022-11-29 13:26:02,824 [INFO] not all flows are present, will run flow generation for all images
2022-11-29 13:26:02,826 [INFO] training from scratch
2022-11-29 13:26:02,826 [INFO] median diameter set to 0 => no rescaling during training
2022-11-29 13:26:02,996 [INFO] No precomuting flows with Omnipose. Computed during training.
2022-11-29 13:26:03,021 [INFO] >>> Using RAdam optimizer
2022-11-29 13:26:03,021 [INFO] >>>> training network with 120 channel input <<<<
2022-11-29 13:26:03,021 [INFO] >>>> LR: 0.10000, batch_size: 4, weight_decay: 0.00001
2022-11-29 13:26:03,021 [INFO] >>>> ntrain = 5
2022-11-29 13:26:03,021 [INFO] >>>> nimg_per_epoch = 5
Traceback (most recent call last):
File "/Users/mcruz/opt/anaconda3/envs/omnipose/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/Users/mcruz/opt/anaconda3/envs/omnipose/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/Users/mcruz/opt/anaconda3/envs/omnipose/lib/python3.10/site-packages/omnipose/__main__.py", line 3, in <module>
main(omni_CLI=True)
File "/Users/mcruz/opt/anaconda3/envs/omnipose/lib/python3.10/site-packages/cellpose/__main__.py", line 476, in main
cpmodel_path = model.train(images, labels, train_files=image_names,
File "/Users/mcruz/opt/anaconda3/envs/omnipose/lib/python3.10/site-packages/cellpose/models.py", line 1045, in train
model_path = self._train_net(train_data, train_labels,
File "/Users/mcruz/opt/anaconda3/envs/omnipose/lib/python3.10/site-packages/cellpose/core.py", line 1001, in _train_net
imgi, lbl, scale = transforms.random_rotate_and_resize(
File "/Users/mcruz/opt/anaconda3/envs/omnipose/lib/python3.10/site-packages/cellpose/transforms.py", line 839, in random_rotate_and_resize
return omnipose.core.random_rotate_and_resize(X, Y=Y, scale_range=scale_range, gamma_range=gamma_range,
File "/Users/mcruz/opt/anaconda3/envs/omnipose/lib/python3.10/site-packages/omnipose/core.py", line 1387, in random_rotate_and_resize
imgi[n], lbl[n], scale[n] = random_crop_warp(img, y, nt, tyx, nchan, scale[n],
File "/Users/mcruz/opt/anaconda3/envs/omnipose/lib/python3.10/site-packages/omnipose/core.py", line 1543, in random_crop_warp
I = do_warp(img[k], M, tyx, offset=offset, mode=mode)
File "/Users/mcruz/opt/anaconda3/envs/omnipose/lib/python3.10/site-packages/omnipose/core.py", line 1606, in do_warp
return scipy.ndimage.affine_transform(A, np.linalg.inv(M), offset=offset, output_shape=tyx, order=order, mode=mode)
File "/Users/mcruz/opt/anaconda3/envs/omnipose/lib/python3.10/site-packages/scipy/ndimage/_interpolation.py", line 586, in affine_transform
raise ValueError(msg)
ValueError: Expected homogeneous transformation matrix with shape (3, 3) for image shape (1, 288), but bottom row was not equal to [0, 0, 1]
@mccruz07 I think I have seen others with this error in the past, it came down to incompatible dependencies. Did you install in a fresh conda environment? Can you post your conda list and pip list?
Sure, here is the conda list:
> conda list
# packages in environment at /Users/mcruz/opt/anaconda3/envs/omnipose:
#
# Name Version Build Channel
aom 3.5.0 h7ea286d_0 conda-forge
appnope 0.1.3 pypi_0 pypi
asttokens 2.2.0 pypi_0 pypi
backcall 0.2.0 pypi_0 pypi
blosc 1.21.1 hd414afc_3 conda-forge
brotli 1.0.9 h1a8c8d9_8 conda-forge
brotli-bin 1.0.9 h1a8c8d9_8 conda-forge
brunsli 0.1 h9f76cd9_0 conda-forge
bzip2 1.0.8 h3422bc3_4 conda-forge
c-ares 1.18.1 h3422bc3_0 conda-forge
c-blosc2 2.4.3 h303ed30_0 conda-forge
ca-certificates 2022.9.24 h4653dfc_0 conda-forge
cachetools 5.2.0 pypi_0 pypi
cellpose-omni 0.7.2 pypi_0 pypi
certifi 2022.9.24 pypi_0 pypi
cfitsio 4.1.0 hd4f5c17_0 conda-forge
charls 2.3.4 hbdafb3b_0 conda-forge
charset-normalizer 2.1.1 pypi_0 pypi
colour 0.1.5 pypi_0 pypi
comm 0.1.1 pypi_0 pypi
contourpy 1.0.6 pypi_0 pypi
cycler 0.11.0 pypi_0 pypi
darkdetect 0.7.1 pypi_0 pypi
dav1d 1.0.0 he4db4b2_1 conda-forge
debugpy 1.6.4 pypi_0 pypi
decorator 5.1.1 pypi_0 pypi
edt 2.3.0 pypi_0 pypi
entrypoints 0.4 pypi_0 pypi
executing 1.2.0 pypi_0 pypi
fastremap 1.13.3 pypi_0 pypi
fonttools 4.38.0 pypi_0 pypi
giflib 5.2.1 h27ca646_2 conda-forge
google-api-core 2.10.2 pypi_0 pypi
google-auth 2.14.1 pypi_0 pypi
google-cloud-core 2.3.2 pypi_0 pypi
google-cloud-storage 2.6.0 pypi_0 pypi
google-crc32c 1.5.0 pypi_0 pypi
google-resumable-media 2.4.0 pypi_0 pypi
googleapis-common-protos 1.57.0 pypi_0 pypi
idna 3.4 pypi_0 pypi
imagecodecs 2022.9.26 py310h24fba19_3 conda-forge
imageio 2.22.4 pypi_0 pypi
ipykernel 6.18.2 pypi_0 pypi
ipython 8.7.0 pypi_0 pypi
jedi 0.18.2 pypi_0 pypi
joblib 1.2.0 pypi_0 pypi
jpeg 9e he4db4b2_2 conda-forge
jupyter-client 7.4.7 pypi_0 pypi
jupyter-core 5.1.0 pypi_0 pypi
jxrlib 1.1 h27ca646_2 conda-forge
kiwisolver 1.4.4 pypi_0 pypi
krb5 1.19.3 he492e65_0 conda-forge
lcms2 2.14 h8193b64_0 conda-forge
lerc 4.0.0 h9a09cb3_0 conda-forge
libaec 1.0.6 hbdafb3b_0 conda-forge
libavif 0.11.1 h3d80962_0 conda-forge
libblas 3.9.0 16_osxarm64_openblas conda-forge
libbrotlicommon 1.0.9 h1a8c8d9_8 conda-forge
libbrotlidec 1.0.9 h1a8c8d9_8 conda-forge
libbrotlienc 1.0.9 h1a8c8d9_8 conda-forge
libcblas 3.9.0 16_osxarm64_openblas conda-forge
libcurl 7.86.0 h1c293e1_1 conda-forge
libcxx 14.0.6 h2692d47_0 conda-forge
libdeflate 1.14 h1a8c8d9_0 conda-forge
libedit 3.1.20191231 hc8eb9b7_2 conda-forge
libev 4.33 h642e427_1 conda-forge
libffi 3.4.2 h3422bc3_5 conda-forge
libgfortran 5.0.0 11_3_0_hd922786_26 conda-forge
libgfortran5 11.3.0 hdaf2cc0_26 conda-forge
liblapack 3.9.0 16_osxarm64_openblas conda-forge
libnghttp2 1.47.0 h519802c_1 conda-forge
libopenblas 0.3.21 openmp_hc731615_3 conda-forge
libpng 1.6.38 h76d750c_0 conda-forge
libsqlite 3.39.4 h76d750c_0 conda-forge
libssh2 1.10.0 h7a5bd25_3 conda-forge
libtiff 4.4.0 hfa0b094_4 conda-forge
libwebp-base 1.2.4 h57fd34a_0 conda-forge
libzlib 1.2.13 h03a7124_4 conda-forge
libzopfli 1.0.3 h9f76cd9_0 conda-forge
llvm-openmp 15.0.4 h7cfbb63_0 conda-forge
llvmlite 0.39.1 pypi_0 pypi
lz4-c 1.9.3 hbdafb3b_1 conda-forge
mahotas 1.4.13 pypi_0 pypi
matplotlib 3.6.2 pypi_0 pypi
matplotlib-inline 0.1.6 pypi_0 pypi
mgen 1.2.0 pypi_0 pypi
mpmath 1.2.1 pypi_0 pypi
natsort 8.2.0 pypi_0 pypi
ncolor 1.1.8 pypi_0 pypi
ncurses 6.3 h07bb92c_1 conda-forge
nest-asyncio 1.5.6 pypi_0 pypi
networkx 2.8.8 pypi_0 pypi
numba 0.56.4 pypi_0 pypi
numpy 1.23.4 py310h5d7c261_1 conda-forge
omnipose 0.3.4.dev9+g88ccc8d pypi_0 pypi
omnipose-theme 1.1.2 pypi_0 pypi
opencv-python-headless 4.6.0.66 pypi_0 pypi
openjpeg 2.5.0 h5d4e404_1 conda-forge
openssl 3.0.7 h03a7124_0 conda-forge
packaging 21.3 pypi_0 pypi
parso 0.8.3 pypi_0 pypi
pbr 5.11.0 pypi_0 pypi
pexpect 4.8.0 pypi_0 pypi
pickleshare 0.7.5 pypi_0 pypi
pillow 9.3.0 pypi_0 pypi
pip 22.3.1 pyhd8ed1ab_0 conda-forge
platformdirs 2.5.4 pypi_0 pypi
prompt-toolkit 3.0.33 pypi_0 pypi
protobuf 4.21.9 pypi_0 pypi
psutil 5.9.4 pypi_0 pypi
ptyprocess 0.7.0 pypi_0 pypi
pure-eval 0.2.2 pypi_0 pypi
pyasn1 0.4.8 pypi_0 pypi
pyasn1-modules 0.2.8 pypi_0 pypi
pygments 2.13.0 pypi_0 pypi
pyparsing 3.0.9 pypi_0 pypi
pyqt6 6.4.0 pypi_0 pypi
pyqt6-qt6 6.4.0 pypi_0 pypi
pyqt6-sip 13.4.0 pypi_0 pypi
pyqtgraph 0.13.1 pypi_0 pypi
python 3.10.4 h14b404e_0_cpython conda-forge
python-dateutil 2.8.2 pypi_0 pypi
python_abi 3.10 2_cp310 conda-forge
pytorch 1.14.0.dev20221116 py3.10_0 pytorch-nightly
pytorch-ranger 0.1.1 pypi_0 pypi
pywavelets 1.4.1 pypi_0 pypi
pyzmq 24.0.1 pypi_0 pypi
qtpy 2.3.0 pypi_0 pypi
readline 8.1.2 h46ed386_0 conda-forge
requests 2.28.1 pypi_0 pypi
rsa 4.9 pypi_0 pypi
scikit-image 0.19.3 pypi_0 pypi
scikit-learn 1.1.3 pypi_0 pypi
scipy 1.9.3 pypi_0 pypi
setuptools 65.5.1 pyhd8ed1ab_0 conda-forge
six 1.16.0 pypi_0 pypi
snappy 1.1.9 h17c5cce_2 conda-forge
sqlite 3.39.4 h2229b38_0 conda-forge
stack-data 0.6.2 pypi_0 pypi
superqt 0.4.0 pypi_0 pypi
sympy 1.11.1 pypi_0 pypi
threadpoolctl 3.1.0 pypi_0 pypi
tifffile 2022.10.10 pypi_0 pypi
tk 8.6.12 he1e0b03_0 conda-forge
torch 1.13.0 pypi_0 pypi
torch-optimizer 0.3.0 pypi_0 pypi
tornado 6.2 pypi_0 pypi
tqdm 4.64.1 pypi_0 pypi
traitlets 5.6.0 pypi_0 pypi
typing_extensions 4.4.0 pyha770c72_0 conda-forge
tzdata 2022f h191b570_0 conda-forge
urllib3 1.26.12 pypi_0 pypi
wcwidth 0.2.5 pypi_0 pypi
wheel 0.38.4 pyhd8ed1ab_0 conda-forge
xz 5.2.6 h57fd34a_0 conda-forge
zfp 1.0.0 hb6e4faa_3 conda-forge
zlib-ng 2.0.6 he4db4b2_0 conda-forge
zstd 1.5.2 h8128057_4 conda-forge
and the pip list:
> pip list
Package Version
------------------------ -------------------
appnope 0.1.3
asttokens 2.2.0
backcall 0.2.0
cachetools 5.2.0
cellpose-omni 0.7.2
certifi 2022.9.24
charset-normalizer 2.1.1
colour 0.1.5
comm 0.1.1
contourpy 1.0.6
cycler 0.11.0
darkdetect 0.7.1
debugpy 1.6.4
decorator 5.1.1
edt 2.3.0
entrypoints 0.4
executing 1.2.0
fastremap 1.13.3
fonttools 4.38.0
google-api-core 2.10.2
google-auth 2.14.1
google-cloud-core 2.3.2
google-cloud-storage 2.6.0
google-crc32c 1.5.0
google-resumable-media 2.4.0
googleapis-common-protos 1.57.0
idna 3.4
imagecodecs 2022.9.26
imageio 2.22.4
ipykernel 6.18.2
ipython 8.7.0
jedi 0.18.2
joblib 1.2.0
jupyter_client 7.4.7
jupyter_core 5.1.0
kiwisolver 1.4.4
llvmlite 0.39.1
mahotas 1.4.13
matplotlib 3.6.2
matplotlib-inline 0.1.6
mgen 1.2.0
mpmath 1.2.1
natsort 8.2.0
ncolor 1.1.8
nest-asyncio 1.5.6
networkx 2.8.8
numba 0.56.4
numpy 1.23.4
omnipose 0.3.4.dev9+g88ccc8d
omnipose-theme 1.1.2
opencv-python-headless 4.6.0.66
packaging 21.3
parso 0.8.3
pbr 5.11.0
pexpect 4.8.0
pickleshare 0.7.5
Pillow 9.3.0
pip 22.3.1
platformdirs 2.5.4
prompt-toolkit 3.0.33
protobuf 4.21.9
psutil 5.9.4
ptyprocess 0.7.0
pure-eval 0.2.2
pyasn1 0.4.8
pyasn1-modules 0.2.8
Pygments 2.13.0
pyparsing 3.0.9
PyQt6 6.4.0
PyQt6-Qt6 6.4.0
PyQt6-sip 13.4.0
pyqtgraph 0.13.1
python-dateutil 2.8.2
pytorch-ranger 0.1.1
PyWavelets 1.4.1
pyzmq 24.0.1
QtPy 2.3.0
requests 2.28.1
rsa 4.9
scikit-image 0.19.3
scikit-learn 1.1.3
scipy 1.9.3
setuptools 65.5.1
six 1.16.0
stack-data 0.6.2
superqt 0.4.0
sympy 1.11.1
threadpoolctl 3.1.0
tifffile 2022.10.10
torch 1.14.0.dev20221116
torch-optimizer 0.3.0
tornado 6.2
tqdm 4.64.1
traitlets 5.6.0
typing_extensions 4.4.0
urllib3 1.26.12
wcwidth 0.2.5
wheel 0.38.4
Thank you!
I'm also having the exact same issue. Clean environment to start. Anyone figured out a solution or work around?
Hi @kevinjohncutler, I seem to get this same issue when trying to run training in 3D on a custom dataset that I prepared. My images are saved as ZYX and I started off with a clean conda environment. This is the error message I am seeing:
I am running on the CLI: python -m omnipose --train --use_gpu --dir /combined_dataset_omnipose --mask_filter _mask --n_epochs 10 --pretrained_model None --verbose --dim 3 --save_each
Has anyone figured out any solutions for this?
@gouthamr321 @Reardonlab @mccruz07 @audreyeternal Sorry for the delay, I wasn't able to reproduce this issue but it dawned on me that none of the commands posted are using --all_channels. This is the source of the error. See the training docs page showing the use of this command on 3D data. I apologize that this was not emphasized. but this argument is required for mono-channel images in 3D. Arguably, the necessity of this for mono-channel images should be considered a bug, and I will work on a fix. For now, please let me know if this solves your issue. I have not tried training on 3D multichannel images either, so I am curious if anyone has attempted this.
Hey @kevinjohncutler
Sorry to bother you again, but the initial issue was resolved using the --all_channels flag, but now I'm seeing another error:
> python -m omnipose --train --use_gpu --dir ./omni/new --mask_filter _masks --n_epochs 400 --pretrained_model None --learning_rate 0.1 --save_every 50 --save_each --verbose --look_one_level_down --all_channels --dim 3 --RAdam --batch_size 4
2023-02-16 10:07:34,799 [INFO] WRITING LOG OUTPUT TO /Users/mcruz/.cellpose/run.log
log file /Users/mcruz/.cellpose/run.log
2023-02-16 10:07:34,801 [INFO] ** TORCH GPU version installed and working. **
2023-02-16 10:07:34,801 [INFO] >>>> using GPU
Omnipose enabled. See Omnipose repo for licencing details.
2023-02-16 10:07:34,801 [INFO] Training omni model. Setting nclasses=4, RAdam=True
2023-02-16 10:07:34,814 [INFO] not all flows are present, will run flow generation for all images
2023-02-16 10:07:34,819 [INFO] training from scratch
2023-02-16 10:07:34,819 [INFO] during training rescaling images to fixed diameter of 30.0 pixels
2023-02-16 10:07:34,940 [INFO] Training with rescale = 1.00
2023-02-16 10:07:35,430 [INFO] No precomuting flows with Omnipose. Computed during training.
2023-02-16 10:07:35,455 [WARNING] channels is set to None, input must therefore have nchan channels (default is 2)
2023-02-16 10:07:35,462 [INFO] >>> Using RAdam optimizer
Traceback (most recent call last):
File "/Users/mcruz/opt/anaconda3/envs/omnipose1/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/Users/mcruz/opt/anaconda3/envs/omnipose1/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/Users/mcruz/opt/anaconda3/envs/omnipose1/lib/python3.9/site-packages/omnipose/__main__.py", line 3, in <module>
main(omni_CLI=True)
File "/Users/mcruz/opt/anaconda3/envs/omnipose1/lib/python3.9/site-packages/cellpose_omni/__main__.py", line 480, in main
cpmodel_path = model.train(images, labels, links, train_files=image_names,
File "/Users/mcruz/opt/anaconda3/envs/omnipose1/lib/python3.9/site-packages/cellpose_omni/models.py", line 1071, in train
model_path = self._train_net(train_data, train_labels, train_links,
File "/Users/mcruz/opt/anaconda3/envs/omnipose1/lib/python3.9/site-packages/cellpose_omni/core.py", line 935, in _train_net
if links is not None:
NameError: name 'links' is not defined