talmolab
INVALID_ARGUMENT: Conv2DSlowBackpropInput: Size of out_backprop doesn't match computed Error During Training
Bug description
During the training of a model: single instance, unet (max stride:16, filters:16[rate=2], sigma: 1.5) we get this error at the end the training loop when the model has reached the plateau:
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found. (0) INVALID_ARGUMENT: Conv2DSlowBackpropInput: Size of out_backprop doesn't match computed: actual = 39, computed = 40 spatial_dim: 3 input: 80 filter: 3 output: 39 stride: 2 dilation: 1 [[node single_instance_inference_model/single_instance_inference_layer_1/model/stack0_dec0_s16_to_s8_trans_conv/conv2d_transpose (defined at C:\Users\SNeurobiology\miniconda3\envs\sleap\lib\site-packages\keras\backend.py:5532) ]] [[single_instance_inference_model/single_instance_inference_layer_1/PartitionedCall_1/range/_20]] (1) INVALID_ARGUMENT: Conv2DSlowBackpropInput: Size of out_backprop doesn't match computed: actual = 39, computed = 40 spatial_dim: 3 input: 80 filter: 3 output: 39 stride: 2 dilation: 1 [[node single_instance_inference_model/single_instance_inference_layer_1/model/stack0_dec0_s16_to_s8_trans_conv/conv2d_transpose (defined at C:\Users\SNeurobiology\miniconda3\envs\sleap\lib\site-packages\keras\backend.py:5532) ]] 0 successful operations.
Expected behaviour
We have trained the model before and we did not encountered such error. We have also run the model inference over all the video used for extracting the training labels and no error has been observed.
Actual behaviour
Your personal set up
- OS: Windows 10
- Version(s): Python 3.7.12, SLEAP 1.3.3
- SLEAP installation method:
- [ ] pip package
Environment packages
absl-py 1.0.0 pypi_0 pypi
astunparse 1.6.3 pypi_0 pypi
attrs 21.4.0 pyhd8ed1ab_0 conda-forge
backcall 0.2.0 pyh9f0ad1d_0 conda-forge
backports 1.0 pyhd8ed1ab_4 conda-forge
backports-zoneinfo 0.2.1 pypi_0 pypi
backports.functools_lru_cache 2.0.0 pyhd8ed1ab_0 conda-forge
blosc 1.21.3 h6c2663c_0
bottleneck 1.3.5 py37h080aedc_0
brotli 1.1.0 hcfcfb64_1 conda-forge
brotli-bin 1.1.0 hcfcfb64_1 conda-forge
bzip2 1.0.8 h2bbff1b_6
ca-certificates 2024.9.24 haa95532_0
cached-property 1.5.2 pypi_0 pypi
cachetools 4.2.4 pypi_0 pypi
cattrs 1.1.1 pyhd8ed1ab_0 conda-forge
certifi 2024.8.30 pyhd8ed1ab_0 conda-forge
charset-normalizer 2.0.9 pypi_0 pypi
cloudpickle 2.2.1 pyhd8ed1ab_0 conda-forge
colorama 0.4.6 pyhd8ed1ab_0 conda-forge
cuda-nvcc 11.3.58 hb8d16a4_0 nvidia
cudatoolkit 11.3.1 hf2f0253_13 conda-forge
cudnn 8.2.1.32 h754d62a_0 conda-forge
cycler 0.11.0 pyhd8ed1ab_0 conda-forge
cytoolz 0.12.0 py37hcc03f2d_0 conda-forge
dask-core 2022.2.0 pyhd8ed1ab_0 conda-forge
debugpy 1.6.3 py37hf2a7229_0 conda-forge
decorator 5.1.1 pyhd8ed1ab_0 conda-forge
efficientnet 1.0.0 pypi_0 pypi
entrypoints 0.4 pyhd8ed1ab_0 conda-forge
flatbuffers 2.0 pypi_0 pypi
fonttools 4.38.0 py37h51bd9d9_0 conda-forge
freeglut 3.2.2 he0c23c2_3 conda-forge
freetype 2.12.1 hdaf720e_2 conda-forge
fsspec 2023.1.0 pyhd8ed1ab_0 conda-forge
gast 0.4.0 pypi_0 pypi
geos 3.11.0 h39d44d4_0 conda-forge
google-auth 2.3.3 pypi_0 pypi
google-auth-oauthlib 0.4.6 pypi_0 pypi
google-pasta 0.2.0 pypi_0 pypi
grpcio 1.43.0 pypi_0 pypi
h5py 3.1.0 pypi_0 pypi
hdf5 1.10.6 h1756f20_1
hdmf 3.6.1 pypi_0 pypi
icc_rt 2022.1.0 h6049295_2
icu 69.1 h0e60522_0 conda-forge
idna 3.3 pypi_0 pypi
image-classifiers 1.0.0 pypi_0 pypi
imagecodecs-lite 2019.12.3 py37h0b711f8_5 conda-forge
imageio 2.34.2 pyh12aca89_0 conda-forge
imgaug 0.4.0 pyhd8ed1ab_1 conda-forge
imgstore 0.2.9 pypi_0 pypi
importlib-metadata 4.2.0 pypi_0 pypi
importlib-resources 5.12.0 pypi_0 pypi
intel-openmp 2024.2.0 h57928b3_979 conda-forge
ipykernel 6.16.2 pyh025b116_0 conda-forge
ipython 7.33.0 py37h03978a9_0 conda-forge
jasper 2.0.33 hc2e4405_1 conda-forge
jedi 0.19.1 pyhd8ed1ab_0 conda-forge
joblib 1.3.2 pyhd8ed1ab_0 conda-forge
jpeg 9e hcfcfb64_3 conda-forge
jsmin 3.0.1 pyhd8ed1ab_0 conda-forge
jsonpickle 1.2 py_0 conda-forge
jsonschema 4.17.3 pypi_0 pypi
jupyter_client 7.4.9 pyhd8ed1ab_0 conda-forge
jupyter_core 4.11.1 py37h03978a9_0 conda-forge
keras 2.7.0 pypi_0 pypi
keras-applications 1.0.8 pypi_0 pypi
keras-preprocessing 1.1.2 pypi_0 pypi
kiwisolver 1.4.4 py37h8c56517_0 conda-forge
lcms2 2.14 h90d422f_0 conda-forge
lerc 4.0.0 h63175ca_0 conda-forge
libblas 3.9.0 22_win64_mkl conda-forge
libbrotlicommon 1.1.0 hcfcfb64_1 conda-forge
libbrotlidec 1.1.0 hcfcfb64_1 conda-forge
libbrotlienc 1.1.0 hcfcfb64_1 conda-forge
libcblas 3.9.0 22_win64_mkl conda-forge
libclang 12.0.0 pypi_0 pypi
libdeflate 1.14 hcfcfb64_0 conda-forge
libhwloc 2.11.0 default_h8125262_1000 conda-forge
libiconv 1.17 hcfcfb64_2 conda-forge
liblapack 3.9.0 22_win64_mkl conda-forge
liblapacke 3.9.0 22_win64_mkl conda-forge
libopencv 4.5.5 py37h542666b_10 conda-forge
libpng 1.6.43 h19919ed_0 conda-forge
libprotobuf 3.20.3 h12be248_0 conda-forge
libsodium 1.0.18 h8d14728_1 conda-forge
libsqlite 3.46.0 h2466b09_0 conda-forge
libtiff 4.4.0 hc4f729c_5 conda-forge
libwebp-base 1.4.0 hcfcfb64_0 conda-forge
libxcb 1.13 hcd874cb_1004 conda-forge
libxml2 2.12.7 h283a6d9_1 conda-forge
libxslt 1.1.39 h3df6e99_0 conda-forge
libzlib 1.2.13 h2466b09_6 conda-forge
locket 1.0.0 pyhd8ed1ab_0 conda-forge
lz4-c 1.9.4 h2bbff1b_1
lzo 2.10 he774522_2
m2w64-gcc-libgfortran 5.3.0 6 conda-forge
m2w64-gcc-libs 5.3.0 7 conda-forge
m2w64-gcc-libs-core 5.3.0 7 conda-forge
m2w64-gmp 6.1.0 2 conda-forge
m2w64-libwinpthread-git 5.0.0.4634.697f757 2 conda-forge
markdown 3.3.6 pypi_0 pypi
markdown-it-py 2.2.0 pyhd8ed1ab_0 conda-forge
matplotlib-base 3.5.3 py37hbaab90a_2 conda-forge
matplotlib-inline 0.1.7 pyhd8ed1ab_0 conda-forge
mdurl 0.1.2 pyhd8ed1ab_0 conda-forge
mkl 2024.1.0 h66d3029_692 conda-forge
mock 4.0.3 pyhd3eb1b0_0
msys2-conda-epoch 20160418 1 conda-forge
munkres 1.1.4 pyh9f0ad1d_0 conda-forge
ndx-pose 0.1.1 pypi_0 pypi
nest-asyncio 1.6.0 pyhd8ed1ab_0 conda-forge
networkx 2.6.3 pyhd8ed1ab_1 conda-forge
nixio 1.5.3 pypi_0 pypi
numexpr 2.7.3 py37h9386db6_2 conda-forge
numpy 1.19.5 pypi_0 pypi
oauthlib 3.1.1 pypi_0 pypi
opencv 4.5.5 py37h03978a9_10 conda-forge
opencv-python-headless 4.2.0.34 pypi_0 pypi
openjpeg 2.5.0 hc9384bd_1 conda-forge
openssl 1.1.1w h2bbff1b_0
opt-einsum 3.3.0 pypi_0 pypi
packaging 21.3 pypi_0 pypi
pandas 1.3.5 py37h6214cd6_0
parso 0.8.4 pyhd8ed1ab_0 conda-forge
partd 1.4.1 pyhd8ed1ab_0 conda-forge
patsy 0.5.6 pyhd8ed1ab_0 conda-forge
pickleshare 0.7.5 py_1003 conda-forge
pillow 9.2.0 py37h42a8222_2 conda-forge
pip 24.0 pyhd8ed1ab_0 conda-forge
pkgutil-resolve-name 1.3.10 pypi_0 pypi
plotly 5.18.0 pypi_0 pypi
prompt-toolkit 3.0.48 pyha770c72_0 conda-forge
protobuf 3.19.1 pypi_0 pypi
psutil 5.9.3 py37h51bd9d9_0 conda-forge
pthread-stubs 0.4 hcd874cb_1001 conda-forge
pthreads-win32 2.9.1 hfa6e2cd_3 conda-forge
py-opencv 4.5.5 py37h90c5f73_10 conda-forge
pyasn1 0.4.8 pypi_0 pypi
pyasn1-modules 0.2.8 pypi_0 pypi
pygments 2.17.2 pyhd8ed1ab_0 conda-forge
pykalman 0.9.7 pyhd8ed1ab_0 conda-forge
pynwb 2.3.3 pypi_0 pypi
pyparsing 3.0.6 pypi_0 pypi
pyrsistent 0.19.3 pypi_0 pypi
pyside2 5.13.2 py37h760f651_8 conda-forge
pytables 3.7.0 py37h388bc9b_1
python 3.7.12 h7840368_100_cpython conda-forge
python-dateutil 2.9.0 pyhd8ed1ab_0 conda-forge
python-rapidjson 1.9 py37h7f67f24_0 conda-forge
python_abi 3.7 4_cp37m conda-forge
pytz 2024.1 pyhd8ed1ab_0 conda-forge
pywavelets 1.3.0 py37h3a130e4_1 conda-forge
pywin32 303 py37hcc03f2d_0 conda-forge
pyyaml 6.0 py37hcc03f2d_4 conda-forge
pyzmq 24.0.1 py37h7347f05_0 conda-forge
qimage2ndarray 1.10.0 pypi_0 pypi
qt 5.12.9 h556501e_6 conda-forge
qtpy 2.4.1 pyhd8ed1ab_0 conda-forge
requests 2.26.0 pypi_0 pypi
requests-oauthlib 1.3.0 pypi_0 pypi
rich 13.7.1 pyhd8ed1ab_0 conda-forge
ruamel-yaml 0.17.32 pypi_0 pypi
ruamel-yaml-clib 0.2.7 pypi_0 pypi
scikit-image 0.19.3 py37h3182a2c_1 conda-forge
scikit-learn 1.0 py37ha78be43_1 conda-forge
scikit-video 1.1.11 pyh24bf2e0_0 conda-forge
scipy 1.7.3 py37hb6553fb_0 conda-forge
seaborn 0.12.2 hd8ed1ab_0 conda-forge
seaborn-base 0.12.2 pyhd8ed1ab_0 conda-forge
segmentation-models 1.0.1 pypi_0 pypi
setuptools 59.8.0 py37h03978a9_1 conda-forge
setuptools-scm 6.3.2 pypi_0 pypi
shapely 1.8.5 py37h475e9a0_0 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
sleap 1.3.3 pypi_0 pypi
sqlite 3.46.0 h2466b09_0 conda-forge
statsmodels 0.13.2 py37h3a130e4_0 conda-forge
tbb 2021.12.0 hc790b64_2 conda-forge
tenacity 8.2.3 pypi_0 pypi
tensorboard 2.7.0 pypi_0 pypi
tensorboard-data-server 0.6.1 pypi_0 pypi
tensorboard-plugin-wit 1.8.0 pypi_0 pypi
tensorflow 2.7.0 pypi_0 pypi
tensorflow-estimator 2.7.0 pypi_0 pypi
tensorflow-hub 0.12.0 pyhca92ed8_0 conda-forge
tensorflow-io-gcs-filesystem 0.23.1 pypi_0 pypi
termcolor 1.1.0 pypi_0 pypi
threadpoolctl 3.1.0 pyh8a188c0_0 conda-forge
tifffile 2020.6.3 py_0 conda-forge
tk 8.6.13 h5226925_1 conda-forge
tomli 2.0.0 pypi_0 pypi
toolz 0.12.1 pyhd8ed1ab_0 conda-forge
tornado 6.2 py37hcc03f2d_0 conda-forge
tqdm 4.66.6 pypi_0 pypi
traitlets 5.9.0 pyhd8ed1ab_0 conda-forge
typing-extensions 4.0.1 pypi_0 pypi
typing_extensions 4.7.1 pyha770c72_0 conda-forge
tzdata 2023.3 pypi_0 pypi
tzlocal 5.0.1 pypi_0 pypi
ucrt 10.0.22621.0 h57928b3_0 conda-forge
unicodedata2 14.0.0 py37hcc03f2d_1 conda-forge
urllib3 1.26.7 pypi_0 pypi
vc 14.3 h8a93ad2_20 conda-forge
vc14_runtime 14.40.33810 ha82c5b3_20 conda-forge
vs2015_runtime 14.40.33810 h3bf8584_20 conda-forge
wcwidth 0.2.10 pyhd8ed1ab_0 conda-forge
werkzeug 2.0.2 pypi_0 pypi
wheel 0.42.0 pyhd8ed1ab_0 conda-forge
wrapt 1.13.3 pypi_0 pypi
xorg-libxau 1.0.11 hcd874cb_0 conda-forge
xorg-libxdmcp 1.1.3 hcd874cb_0 conda-forge
xz 5.2.6 h8d14728_0 conda-forge
yaml 0.2.5 h8ffe710_2 conda-forge
zeromq 4.3.4 h0e60522_1 conda-forge
zipp 3.15.0 pypi_0 pypi
zlib 1.2.13 h2466b09_6 conda-forge
zstd 1.5.6 h0ea2cb4_0 conda-forge
Logs
# paste relevant logs here, if any
Screenshots
How to reproduce
- Go to '...'
- Click on '....'
- Scroll down to '....'
- See error
Hi,
Could you send us your training package so we can troubleshoot? You can upload it here.
Thanks!
Elizabeth
Dear Elizabeth,
Thank you for reaching out. I have uploaded our training package, let me know if other files are needed.
Thank you again, Thomas
Il giorno mar 10 dic 2024 alle ore 22:39 Elizabeth @.***> ha scritto:
Hi,
Could you send us your training package so we can troubleshoot? You can upload it here https://docs.google.com/forms/d/e/1FAIpQLSfWBwcuZH8lxGyFabMAODLjSUE4YrSCYbrrnYvXCpgBqZpcJA/viewform .
Thanks!
Elizabeth
— Reply to this email directly, view it on GitHub https://github.com/talmolab/sleap/issues/2039#issuecomment-2532945981, or unsubscribe https://github.com/notifications/unsubscribe-auth/A3C4KU7C6C44NZTC4K6MUFT2E5NQJAVCNFSM6AAAAABTIN7RQ6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMZSHE2DKOJYGE . You are receiving this because you authored the thread.Message ID: @.***>
Hi @Thomasbush9,
Apologies for the delay! We're looking into this issue, Could you please let me know how you installed SLEAP? (We also have an updated stable version: https://github.com/talmolab/sleap/tree/v1.3.4)
It would be great if you could also share your training_config.json and the training ckpts folder with us to debug better. And, did the training data change when you reran the model? It would be helpful if you could share what changed between the last time you trained (ran successfully without the error) and now.
Thanks,
Divya
Update:
I tried installing the same SLEAP version as yours and tried training with your data. I couldn't replicate the error. Could you try updating SLEAP to 1.3.4?
Hi Divya,
Sorry for the delay and thank you for the help!
We have installed SLEAP through pip install. I have attached to this email the training_config.json file.
Thanks again for your help, Thomas
Il giorno mar 17 dic 2024 alle ore 00:20 DivyaSesh @.***> ha scritto:
Hi @Thomasbush9 https://github.com/Thomasbush9,
Apologies for the delay! We're looking into this issue, Could you please let me know how you installed SLEAP? It would be great if you could also share your training_config.json with us to debug better?
Thanks,
Divya
— Reply to this email directly, view it on GitHub https://github.com/talmolab/sleap/issues/2039#issuecomment-2547101358, or unsubscribe https://github.com/notifications/unsubscribe-auth/A3C4KU4ZKUJTFOBDXHQRTLT2F5N4TAVCNFSM6AAAAABTIN7RQ6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNBXGEYDCMZVHA . You are receiving this because you were mentioned.Message ID: @.***>
Hi @Thomasbush9,
Sorry I don't see the attachment in my email. Could you please upload it here? Could you also try to uodate SLEAP to v1.3.4 and check if it's working?
Thanks,
Divya
Hi!
Sorry for the late reply, but I’ve come back to the office this week.
I have uploaded the training config.
Thank you again for the support. Kind regards, Thomas W. Bush
On Fri, 20 Dec 2024 at 18:26, DivyaSesh @.***> wrote:
Hi @Thomasbush9 https://github.com/Thomasbush9,
Sorry I don't see the attachment in my email. Could you please upload it here https://docs.google.com/forms/d/e/1FAIpQLSfWBwcuZH8lxGyFabMAODLjSUE4YrSCYbrrnYvXCpgBqZpcJA/viewform ?
Thanks,
Divya
— Reply to this email directly, view it on GitHub https://github.com/talmolab/sleap/issues/2039#issuecomment-2557426755, or unsubscribe https://github.com/notifications/unsubscribe-auth/A3C4KU6RGANQSQNUE2ADQEL2GRHLNAVCNFSM6AAAAABTIN7RQ6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNJXGQZDMNZVGU . You are receiving this because you were mentioned.Message ID: @.***>