sleap
sleap copied to clipboard
talmolab
issues running inference with ID tracking
I am trying to run sleap tracking. I have a model that runs fine on this particular mp4 file when I do not specify any tracking parameters. Below is the command I am running. I cannot get it to produce any predictions with this command on this mp4. I cannot tell if this is a memory issue - I am currently running it on 4 GPUs and with 160 gb of RAM on a Linux system, so I hope that is not the issue.
sleap-track 22_object_brighter.mp4 -m models/250522_171948.centroid -m models/250522_171948.centered_instance -o test_withtracks.predictions.slp --verbosity json --tracking.tracker flow --tracking.pre_cull_to_target 2 --tracking.pre_cull_iou_threshold 0.8 --tracking.similarity instance --tracking.match greedy --tracking.track_window 5 --tracking.save_shifted_instances 0 --tracking.post_connect_single_breaks 2 --tracking.max_tracking True --tracking.max_tracks 2
More system info
OS: Linux LSB Version: :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch Distributor ID: RedHatEnterprise Description: Red Hat Enterprise Linux release 8.10 (Ootpa) Release: 8.10 Codename: Ootpa
Version(s): SLEAP v1.3.3 , python 3.7.12
SLEAP installation method (listed here): [ x] Conda from package
Environmental packages
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
absl-py 1.0.0 pypi_0 pypi
alsa-lib 1.2.3.2 h166bdaf_0 conda-forge
astunparse 1.6.3 pypi_0 pypi
attrs 21.4.0 pyhd8ed1ab_0 conda-forge
backports-zoneinfo 0.2.1 pypi_0 pypi
blosc 1.21.5 h0f2a231_0 conda-forge
brotli 1.0.9 h166bdaf_9 conda-forge
brotli-bin 1.0.9 h166bdaf_9 conda-forge
brunsli 0.1 h9c3ff4c_0 conda-forge
bzip2 1.0.8 h4bc722e_7 conda-forge
c-ares 1.34.5 hb9d3cd8_0 conda-forge
c-blosc2 2.12.0 hb4ffafa_0 conda-forge
ca-certificates 2025.4.26 hbd8a1cb_0 conda-forge
cached-property 1.5.2 hd8ed1ab_1 conda-forge
cached_property 1.5.2 pyha770c72_1 conda-forge
cachetools 4.2.4 pypi_0 pypi
cairo 1.16.0 h6cf1ce9_1008 conda-forge
cattrs 1.1.1 pyhd8ed1ab_0 conda-forge
certifi 2024.8.30 pyhd8ed1ab_0 conda-forge
cfitsio 4.0.0 h9a35b8e_0 conda-forge
charls 2.3.4 h9c3ff4c_0 conda-forge
charset-normalizer 2.0.9 pypi_0 pypi
cloudpickle 2.2.1 pyhd8ed1ab_0 conda-forge
cuda-nvcc 11.3.58 h2467b9f_0 nvidia
cudatoolkit 11.3.1 hb98b00a_13 conda-forge
cudnn 8.2.1.32 h86fa8c9_0 conda-forge
cycler 0.11.0 pyhd8ed1ab_0 conda-forge
cytoolz 0.12.0 py37h540881e_0 conda-forge
dask-core 2022.2.0 pyhd8ed1ab_0 conda-forge
dbus 1.13.6 h5008d03_3 conda-forge
efficientnet 1.0.0 pypi_0 pypi
expat 2.7.0 h5888daf_0 conda-forge
ffmpeg 4.3.2 h37c90e5_3 conda-forge
flatbuffers 2.0 pypi_0 pypi
fontconfig 2.14.2 h14ed4e7_0 conda-forge
fonttools 4.38.0 py37h540881e_0 conda-forge
freetype 2.12.1 h267a509_2 conda-forge
fsspec 2023.1.0 pyhd8ed1ab_0 conda-forge
gast 0.4.0 pypi_0 pypi
geos 3.11.0 h27087fc_0 conda-forge
gettext 0.24.1 h5888daf_0 conda-forge
gettext-tools 0.24.1 h5888daf_0 conda-forge
giflib 5.2.2 hd590300_0 conda-forge
gmp 6.3.0 hac33072_2 conda-forge
gnutls 3.6.13 h85f3911_1 conda-forge
google-auth 2.3.3 pypi_0 pypi
google-auth-oauthlib 0.4.6 pypi_0 pypi
google-pasta 0.2.0 pypi_0 pypi
graphite2 1.3.13 h59595ed_1003 conda-forge
grpcio 1.43.0 pypi_0 pypi
gst-plugins-base 1.18.5 hf529b03_3 conda-forge
gstreamer 1.18.5 h9f60fe5_3 conda-forge
h5py 3.1.0 nompi_py37h1e651dc_100 conda-forge
harfbuzz 2.9.1 h83ec7ef_1 conda-forge
hdf5 1.10.6 nompi_h6a2412b_1114 conda-forge
icu 68.2 h9c3ff4c_0 conda-forge
idna 3.3 pypi_0 pypi
image-classifiers 1.0.0 pypi_0 pypi
imagecodecs 2021.11.20 py37h119f88a_2 conda-forge
imageio 2.36.0 pyh12aca89_1 conda-forge
imgaug 0.4.0 pyhd8ed1ab_1 conda-forge
imgstore 0.2.9 pypi_0 pypi
importlib-metadata 4.10.0 pypi_0 pypi
importlib-resources 5.12.0 pypi_0 pypi
jasper 1.900.1 h07fcdf6_1006 conda-forge
joblib 1.3.2 pyhd8ed1ab_0 conda-forge
jpeg 9e h0b41bf4_3 conda-forge
jsmin 3.0.1 pyhd8ed1ab_0 conda-forge
jsonpickle 1.2 py_0 conda-forge
jsonschema 4.17.3 pypi_0 pypi
jxrlib 1.1 hd590300_3 conda-forge
keras 2.7.0 pypi_0 pypi
keras-applications 1.0.8 pypi_0 pypi
keras-preprocessing 1.1.2 pypi_0 pypi
keyutils 1.6.1 h166bdaf_0 conda-forge
kiwisolver 1.4.4 py37h7cecad7_0 conda-forge
krb5 1.19.3 h3790be6_0 conda-forge
lame 3.100 h166bdaf_1003 conda-forge
lcms2 2.14 h6ed2654_0 conda-forge
ld_impl_linux-64 2.43 h712a8e2_4 conda-forge
lerc 3.0 h9c3ff4c_0 conda-forge
libaec 1.1.3 h59595ed_0 conda-forge
libasprintf 0.24.1 h8e693c7_0 conda-forge
libasprintf-devel 0.24.1 h8e693c7_0 conda-forge
libblas 3.9.0 20_linux64_openblas conda-forge
libbrotlicommon 1.0.9 h166bdaf_9 conda-forge
libbrotlidec 1.0.9 h166bdaf_9 conda-forge
libbrotlienc 1.0.9 h166bdaf_9 conda-forge
libcblas 3.9.0 20_linux64_openblas conda-forge
libclang 12.0.0 pypi_0 pypi
libcurl 7.86.0 h7bff187_1 conda-forge
libdeflate 1.10 h7f98852_0 conda-forge
libedit 3.1.20250104 pl5321h7949ede_0 conda-forge
libev 4.33 hd590300_2 conda-forge
libevent 2.1.10 h9b69904_4 conda-forge
libexpat 2.7.0 h5888daf_0 conda-forge
libffi 3.4.6 h2dba641_1 conda-forge
libgcc 15.1.0 h767d61c_2 conda-forge
libgcc-ng 15.1.0 h69a702a_2 conda-forge
libgettextpo 0.24.1 h5888daf_0 conda-forge
libgettextpo-devel 0.24.1 h5888daf_0 conda-forge
libgfortran 15.1.0 h69a702a_2 conda-forge
libgfortran-ng 15.1.0 h69a702a_2 conda-forge
libgfortran5 15.1.0 hcea5267_2 conda-forge
libglib 2.80.2 hf974151_0 conda-forge
libgomp 15.1.0 h767d61c_2 conda-forge
libiconv 1.18 h4ce23a2_1 conda-forge
liblapack 3.9.0 20_linux64_openblas conda-forge
liblapacke 3.9.0 20_linux64_openblas conda-forge
libllvm11 11.1.0 he0ac6c6_5 conda-forge
liblzma 5.8.1 hb9d3cd8_1 conda-forge
liblzma-devel 5.8.1 hb9d3cd8_1 conda-forge
libnghttp2 1.51.0 hdcd2b5c_0 conda-forge
libnsl 2.0.1 hd590300_0 conda-forge
libogg 1.3.5 hd0c01bc_1 conda-forge
libopenblas 0.3.25 pthreads_h413a1c8_0 conda-forge
libopencv 4.5.3 py37h25009ff_1 conda-forge
libopus 1.5.2 hd0c01bc_0 conda-forge
libpng 1.6.43 h2797004_0 conda-forge
libpq 13.8 hd77ab85_0 conda-forge
libprotobuf 3.16.0 h780b84a_0 conda-forge
libsodium 1.0.18 h36c2ea0_1 conda-forge
libsqlite 3.46.0 hde9e2c9_0 conda-forge
libssh2 1.10.0 haa6b8db_3 conda-forge
libstdcxx 15.1.0 h8f9b012_2 conda-forge
libstdcxx-ng 15.1.0 h4852527_2 conda-forge
libtiff 4.4.0 h0fcbabc_0 conda-forge
libuuid 2.38.1 h0b41bf4_0 conda-forge
libvorbis 1.3.7 h9c3ff4c_0 conda-forge
libwebp-base 1.5.0 h851e524_0 conda-forge
libxcb 1.13 h7f98852_1004 conda-forge
libxkbcommon 1.0.3 he3ba5ed_0 conda-forge
libxml2 2.9.12 h72842e0_0 conda-forge
libxslt 1.1.33 h15afd5d_2 conda-forge
libzlib 1.2.13 h4ab18f5_6 conda-forge
libzopfli 1.0.3 h9c3ff4c_0 conda-forge
locket 1.0.0 pyhd8ed1ab_0 conda-forge
lz4-c 1.9.3 h9c3ff4c_1 conda-forge
markdown 3.3.6 pypi_0 pypi
markdown-it-py 2.2.0 pyhd8ed1ab_0 conda-forge
matplotlib-base 3.5.3 py37hf395dca_2 conda-forge
mdurl 0.1.2 pyhd8ed1ab_0 conda-forge
munkres 1.1.4 pyh9f0ad1d_0 conda-forge
mysql-common 8.0.32 h14678bc_0 conda-forge
mysql-libs 8.0.32 h54cf53e_0 conda-forge
ncurses 6.5 h2d0b736_3 conda-forge
ndx-pose 0.1.1 pypi_0 pypi
nettle 3.6 he412f7d_0 conda-forge
networkx 2.6.3 pyhd8ed1ab_1 conda-forge
nixio 1.5.3 pypi_0 pypi
nspr 4.36 h5888daf_0 conda-forge
nss 3.100 hca3bf56_0 conda-forge
numpy 1.19.5 pypi_0 pypi
oauthlib 3.1.1 pypi_0 pypi
opencv 4.5.3 py37h89c1867_1 conda-forge
opencv-python-headless 4.2.0.34 pypi_0 pypi
openh264 2.1.1 h780b84a_0 conda-forge
openjpeg 2.5.0 h7d73246_1 conda-forge
openssl 1.1.1w hd590300_0 conda-forge
opt-einsum 3.3.0 pypi_0 pypi
packaging 21.3 pypi_0 pypi
pandas 1.3.5 py37he8f5f7f_0 conda-forge
partd 1.4.1 pyhd8ed1ab_0 conda-forge
patsy 0.5.6 pyhd8ed1ab_0 conda-forge
pcre2 10.43 hcad00b1_0 conda-forge
pillow 9.2.0 py37h850a105_2 conda-forge
pip 24.0 pyhd8ed1ab_0 conda-forge
pixman 0.46.0 h29eaf8c_0 conda-forge
pkgutil-resolve-name 1.3.10 pypi_0 pypi
protobuf 3.19.1 pypi_0 pypi
psutil 5.9.3 py37h540881e_0 conda-forge
pthread-stubs 0.4 hb9d3cd8_1002 conda-forge
py-opencv 4.5.3 py37h6531663_1 conda-forge
pyasn1 0.4.8 pypi_0 pypi
pyasn1-modules 0.2.8 pypi_0 pypi
pygments 2.17.2 pyhd8ed1ab_0 conda-forge
pykalman 0.9.7 pyhd8ed1ab_0 conda-forge
pynwb 2.3.3 pypi_0 pypi
pyparsing 3.0.6 pypi_0 pypi
pyrsistent 0.19.3 pypi_0 pypi
pyside2 5.13.2 py37hfa98aef_7 conda-forge
python 3.7.12 hb7a2778_100_cpython conda-forge
python-dateutil 2.9.0 pyhd8ed1ab_0 conda-forge
python-rapidjson 1.9 py37hd23a5d3_0 conda-forge
python_abi 3.7 4_cp37m conda-forge
pytz 2024.2 pyhd8ed1ab_0 conda-forge
pywavelets 1.3.0 py37hda87dfa_1 conda-forge
pyyaml 6.0 py37h540881e_4 conda-forge
pyzmq 24.0.1 py37h0c0c2a8_0 conda-forge
qimage2ndarray 1.10.0 pypi_0 pypi
qt 5.12.9 hda022c4_4 conda-forge
qtpy 2.4.2 pyhdecd6ff_0 conda-forge
readline 8.2 h8c095d6_2 conda-forge
requests 2.26.0 pypi_0 pypi
requests-oauthlib 1.3.0 pypi_0 pypi
rich 13.8.1 pyhd8ed1ab_0 conda-forge
ruamel-yaml 0.17.32 pypi_0 pypi
ruamel-yaml-clib 0.2.7 pypi_0 pypi
scikit-image 0.19.3 py37hfb7772e_1 conda-forge
scikit-learn 1.0 py37hf0f1638_1 conda-forge
scikit-video 1.1.11 pyh24bf2e0_0 conda-forge
scipy 1.7.3 py37hf2a6cf1_0 conda-forge
seaborn 0.12.2 hd8ed1ab_0 conda-forge
seaborn-base 0.12.2 pyhd8ed1ab_0 conda-forge
segmentation-models 1.0.1 pypi_0 pypi
setuptools 59.8.0 py37h89c1867_1 conda-forge
setuptools-scm 6.3.2 pypi_0 pypi
shapely 1.8.5 py37ha4e3bd1_0 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
sleap 1.3.3 pypi_0 pypi
snappy 1.1.10 hdb0a2a9_1 conda-forge
sqlite 3.46.0 h6d4b2fc_0 conda-forge
statsmodels 0.13.2 py37hda87dfa_0 conda-forge
tensorboard 2.7.0 pypi_0 pypi
tensorboard-data-server 0.6.1 pypi_0 pypi
tensorboard-plugin-wit 1.8.0 pypi_0 pypi
tensorflow 2.7.0 pypi_0 pypi
tensorflow-estimator 2.7.0 pypi_0 pypi
tensorflow-hub 0.13.0 pyh56297ac_0 conda-forge
tensorflow-io-gcs-filesystem 0.23.1 pypi_0 pypi
termcolor 1.1.0 pypi_0 pypi
threadpoolctl 3.1.0 pyh8a188c0_0 conda-forge
tifffile 2021.11.2 pyhd8ed1ab_0 conda-forge
tk 8.6.13 noxft_h4845f30_101 conda-forge
tomli 2.0.0 pypi_0 pypi
toolz 0.12.1 pyhd8ed1ab_0 conda-forge
typing-extensions 4.0.1 pypi_0 pypi
typing_extensions 4.7.1 pyha770c72_0 conda-forge
tzlocal 5.0.1 pypi_0 pypi
unicodedata2 14.0.0 py37h540881e_1 conda-forge
urllib3 1.26.7 pypi_0 pypi
werkzeug 2.0.2 pypi_0 pypi
wheel 0.42.0 pyhd8ed1ab_0 conda-forge
wrapt 1.13.3 pypi_0 pypi
x264 1!161.3030 h7f98852_1 conda-forge
xorg-kbproto 1.0.7 hb9d3cd8_1003 conda-forge
xorg-libice 1.1.2 hb9d3cd8_0 conda-forge
xorg-libsm 1.2.6 he73a12e_0 conda-forge
xorg-libx11 1.8.4 h0b41bf4_0 conda-forge
xorg-libxau 1.0.12 hb9d3cd8_0 conda-forge
xorg-libxdmcp 1.1.5 hb9d3cd8_0 conda-forge
xorg-libxext 1.3.4 h0b41bf4_2 conda-forge
xorg-libxrender 0.9.10 h7f98852_1003 conda-forge
xorg-renderproto 0.11.1 hb9d3cd8_1003 conda-forge
xorg-xextproto 7.3.0 hb9d3cd8_1004 conda-forge
xorg-xproto 7.0.31 hb9d3cd8_1008 conda-forge
xz 5.8.1 hbcc6ac9_1 conda-forge
xz-gpl-tools 5.8.1 hbcc6ac9_1 conda-forge
xz-tools 5.8.1 hb9d3cd8_1 conda-forge
yaml 0.2.5 h7f98852_2 conda-forge
zeromq 4.3.5 h59595ed_1 conda-forge
zfp 0.5.5 h9c3ff4c_8 conda-forge
zipp 3.6.0 pypi_0 pypi
zlib 1.2.13 h4ab18f5_6 conda-forge
zlib-ng 2.0.7 h0b41bf4_0 conda-forge
zstd 1.5.6 ha6fb4c9_0 conda-forge
SLEAP OUTPUT
2025-06-06 12:48:14.166953: I tensorflow/core/common_runtime/bfc_allocator.cc:1086] Stats:
Limit: 68719476736
InUse: 68631663616
MaxInUse: 68655283968
NumAllocs: 188124
MaxAllocSize: 45466624
Reserved: 0
PeakReserved: 0
LargestFreeBlock: 0
2025-06-06 12:48:14.194856: W tensorflow/core/common_runtime/bfc_allocator.cc:474] ****************************************************************************************************
Traceback (most recent call last):
File "/blue/npadillacoreano/mcum/conda/envs/sleap/bin/sleap-track", line 33, in <module>
sys.exit(load_entry_point('sleap==1.3.3', 'console_scripts', 'sleap-track')())
File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/sleap/nn/inference.py", line 5424, in main
labels_pr = predictor.predict(provider)
File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/sleap/nn/inference.py", line 526, in predict
self._make_labeled_frames_from_generator(generator, data)
File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/sleap/nn/inference.py", line 2633, in _make_labeled_frames_from_generator
for ex in generator:
File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/sleap/nn/inference.py", line 457, in _predict_generator
ex = process_batch(ex)
File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/sleap/nn/inference.py", line 399, in process_batch
preds = self.inference_model.predict_on_batch(ex, numpy=True)
File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/sleap/nn/inference.py", line 1069, in predict_on_batch
outs = super().predict_on_batch(data, **kwargs)
File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/keras/engine/training.py", line 1986, in predict_on_batch
outputs = self.predict_function(iterator)
File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute
inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
(0) RESOURCE_EXHAUSTED: OOM when allocating tensor with shape[12,496,496,3] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator gpu_host_bfc
[[node cond/CropAndResize
(defined at /blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/sleap/nn/peak_finding.py:180)
]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.
[[top_down_inference_model/find_instance_peaks/PartitionedCall/mod/_208]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.
(1) RESOURCE_EXHAUSTED: OOM when allocating tensor with shape[12,496,496,3] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator gpu_host_bfc
[[node cond/CropAndResize
(defined at /blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/sleap/nn/peak_finding.py:180)
]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.
0 successful operations.
0 derived errors ignored. [Op:__inference_predict_function_4555]
Errors may have originated from an input operation.
Input Source operations connected to node cond/CropAndResize:
In[0] cond/CropAndResize/inputs_1:
In[1] cond/truediv (defined at /blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/sleap/nn/data/instance_cropping.py:89)
In[2] cond/CropAndResize/PartitionedCall_1:
In[3] cond/Cast (defined at /blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/sleap/nn/peak_finding.py:170)
Operation defined at: (most recent call last)
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/bin/sleap-track", line 33, in <module>
>>> sys.exit(load_entry_point('sleap==1.3.3', 'console_scripts', 'sleap-track')())
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/sleap/nn/inference.py", line 5424, in main
>>> labels_pr = predictor.predict(provider)
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/sleap/nn/inference.py", line 526, in predict
>>> self._make_labeled_frames_from_generator(generator, data)
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/sleap/nn/inference.py", line 2633, in _make_labeled_frames_from_generator
>>> for ex in generator:
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/sleap/nn/inference.py", line 457, in _predict_generator
>>> ex = process_batch(ex)
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/sleap/nn/inference.py", line 399, in process_batch
>>> preds = self.inference_model.predict_on_batch(ex, numpy=True)
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/sleap/nn/inference.py", line 1069, in predict_on_batch
>>> outs = super().predict_on_batch(data, **kwargs)
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/keras/engine/training.py", line 1986, in predict_on_batch
>>> outputs = self.predict_function(iterator)
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/keras/engine/training.py", line 1621, in predict_function
>>> return step_function(self, iterator)
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/keras/engine/training.py", line 1611, in step_function
>>> outputs = model.distribute_strategy.run(run_step, args=(data,))
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/keras/engine/training.py", line 1604, in run_step
>>> outputs = model.predict_step(data)
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/keras/engine/training.py", line 1572, in predict_step
>>> return self(x, training=False)
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>> return fn(*args, **kwargs)
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>> outputs = call_fn(inputs, *args, **kwargs)
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>> return fn(*args, **kwargs)
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/sleap/nn/inference.py", line 2254, in call
>>> crop_output = self.centroid_crop(example)
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>> return fn(*args, **kwargs)
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>> outputs = call_fn(inputs, *args, **kwargs)
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>> return fn(*args, **kwargs)
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/sleap/nn/inference.py", line 1813, in call
>>> if n_peaks > 0:
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/sleap/nn/inference.py", line 1886, in call
>>> crops = sleap.nn.peak_finding.crop_bboxes(
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/sleap/nn/peak_finding.py", line 180, in crop_bboxes
>>> crops = tf.image.crop_and_resize(
>>>
Input Source operations connected to node cond/CropAndResize:
In[0] cond/CropAndResize/inputs_1:
In[1] cond/truediv (defined at /blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/sleap/nn/data/instance_cropping.py:89)
In[2] cond/CropAndResize/PartitionedCall_1:
In[3] cond/Cast (defined at /blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/sleap/nn/peak_finding.py:170)
Operation defined at: (most recent call last)
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/bin/sleap-track", line 33, in <module>
>>> sys.exit(load_entry_point('sleap==1.3.3', 'console_scripts', 'sleap-track')())
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/sleap/nn/inference.py", line 5424, in main
>>> labels_pr = predictor.predict(provider)
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/sleap/nn/inference.py", line 526, in predict
>>> self._make_labeled_frames_from_generator(generator, data)
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/sleap/nn/inference.py", line 2633, in _make_labeled_frames_from_generator
>>> for ex in generator:
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/sleap/nn/inference.py", line 457, in _predict_generator
>>> ex = process_batch(ex)
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/sleap/nn/inference.py", line 399, in process_batch
>>> preds = self.inference_model.predict_on_batch(ex, numpy=True)
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/sleap/nn/inference.py", line 1069, in predict_on_batch
>>> outs = super().predict_on_batch(data, **kwargs)
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/keras/engine/training.py", line 1986, in predict_on_batch
>>> outputs = self.predict_function(iterator)
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/keras/engine/training.py", line 1621, in predict_function
>>> return step_function(self, iterator)
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/keras/engine/training.py", line 1611, in step_function
>>> outputs = model.distribute_strategy.run(run_step, args=(data,))
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/keras/engine/training.py", line 1604, in run_step
>>> outputs = model.predict_step(data)
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/keras/engine/training.py", line 1572, in predict_step
>>> return self(x, training=False)
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>> return fn(*args, **kwargs)
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>> outputs = call_fn(inputs, *args, **kwargs)
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>> return fn(*args, **kwargs)
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/sleap/nn/inference.py", line 2254, in call
>>> crop_output = self.centroid_crop(example)
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>> return fn(*args, **kwargs)
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>> outputs = call_fn(inputs, *args, **kwargs)
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>> return fn(*args, **kwargs)
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/sleap/nn/inference.py", line 1813, in call
>>> if n_peaks > 0:
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/sleap/nn/inference.py", line 1886, in call
>>> crops = sleap.nn.peak_finding.crop_bboxes(
>>>
>>> File "/blue/npadillacoreano/mcum/conda/envs/sleap/lib/python3.7/site-packages/sleap/nn/peak_finding.py", line 180, in crop_bboxes
>>> crops = tf.image.crop_and_resize(
>>>
Function call stack:
predict_function -> cond_true_3378 -> predict_function -> cond_true_3378
this looks like a problem with the method flow potentailly, as the below command worked, but when i changed the method to flow i got the same none error but no prediction file was created.
sleap-track 22_object_brighter.mp4 --tracking.tracker simplemaxtracker \
--tracking.similarity iou \
--tracking.match greedy \
--batch_size 1 \
--max_instances 2 \
--tracking.clean_instance_count 2 \
--tracking.target_instance_count 2 \
-m models/250522_171948.centroid \
-m models/250522_171948.centered_instance \
-o test_tracks_prediction2.slp \
--tracking.max_tracking 1 \
--tracking.max_tracks 2
Hi @mcum96,
Is your GPU being used at all? Please type nvidia-smi in the terminal and let us know what the output is.
Thanks!
Elizabeth
