talmolab
Render video only works for evenly-sized videos
Bug description
When trying to render video with clip instances, I get an empty mp4 file.
Input file is .avi and it looks something like this:
I have a single video (20 seconds of mice behaviour, where I perform training and inference on) and its the same video I want to export.
Expected behaviour
I expect that I can open the .mp4 afterwards and look at the tracked instances.
Actual behaviour
Empty video gets produced (0 KB)
Your personal set up
- OS: Windows 11 Home, 64-bit operating system, x64-based processor Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz 2.59 GHz
- Version(s): [SLEAP v1.2.8, Python 3.7.12]
- SLEAP installation method:
- [This method was used ] Conda from package
Environment packages
# Name Version Build Channel
absl-py 0.15.0 pypi_0 pypi
aom 3.4.0 h0e60522_1 conda-forge
astunparse 1.6.3 pypi_0 pypi
attrs 21.2.0 pypi_0 pypi
backports-zoneinfo 0.2.1 pypi_0 pypi
bzip2 1.0.8 h8ffe710_4 conda-forge
ca-certificates 2022.6.15.2 h5b45459_0 conda-forge
cached-property 1.5.2 hd8ed1ab_1 conda-forge
cached_property 1.5.2 pyha770c72_1 conda-forge
cachetools 4.2.4 pypi_0 pypi
cattrs 1.1.1 pypi_0 pypi
certifi 2021.10.8 pypi_0 pypi
charset-normalizer 2.0.12 pypi_0 pypi
clang 5.0 pypi_0 pypi
colorama 0.4.5 pypi_0 pypi
commonmark 0.9.1 pypi_0 pypi
cuda-nvcc 11.3.58 hb8d16a4_0 nvidia
cudatoolkit 11.3.1 h280eb24_10 conda-forge
cudnn 8.2.1.32 h754d62a_0 conda-forge
cycler 0.11.0 pypi_0 pypi
efficientnet 1.0.0 pypi_0 pypi
expat 2.4.8 h39d44d4_0 conda-forge
ffmpeg 5.1.1 gpl_h7b28927_101 conda-forge
flatbuffers 1.12 pypi_0 pypi
font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge
font-ttf-inconsolata 3.000 h77eed37_0 conda-forge
font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge
font-ttf-ubuntu 0.83 hab24e00_0 conda-forge
fontconfig 2.14.0 hce3cb01_0 conda-forge
fonts-conda-ecosystem 1 0 conda-forge
fonts-conda-forge 1 0 conda-forge
fonttools 4.37.1 pypi_0 pypi
freetype 2.12.1 h546665d_0 conda-forge
gast 0.4.0 pypi_0 pypi
geos 3.9.1 h39d44d4_2 conda-forge
google-auth 1.35.0 pypi_0 pypi
google-auth-oauthlib 0.4.6 pypi_0 pypi
google-pasta 0.2.0 pypi_0 pypi
grpcio 1.44.0 pypi_0 pypi
h5py 3.1.0 nompi_py37h19fda09_100 conda-forge
hdf5 1.10.6 nompi_he0bbb20_101 conda-forge
hdmf 3.4.2 pypi_0 pypi
idna 3.3 pypi_0 pypi
image-classifiers 1.0.0 pypi_0 pypi
imageio 2.15.0 pypi_0 pypi
imgaug 0.4.0 pypi_0 pypi
imgstore 0.2.9 pypi_0 pypi
importlib-metadata 4.11.1 pypi_0 pypi
importlib-resources 5.9.0 pypi_0 pypi
intel-openmp 2022.1.0 h57928b3_3787 conda-forge
joblib 1.1.0 pypi_0 pypi
jpeg 9e h8ffe710_2 conda-forge
jsmin 3.0.1 pypi_0 pypi
jsonpickle 1.2 pypi_0 pypi
jsonschema 4.16.0 pypi_0 pypi
keras 2.6.0 pypi_0 pypi
keras-applications 1.0.8 pypi_0 pypi
keras-preprocessing 1.1.2 pypi_0 pypi
kiwisolver 1.4.4 pypi_0 pypi
lcms2 2.12 h2a16943_0 conda-forge
lerc 3.0 h0e60522_0 conda-forge
libblas 3.9.0 16_win64_mkl conda-forge
libcblas 3.9.0 16_win64_mkl conda-forge
libdeflate 1.10 h8ffe710_0 conda-forge
libiconv 1.16 he774522_0 conda-forge
liblapack 3.9.0 16_win64_mkl conda-forge
libpng 1.6.37 h1d00b33_4 conda-forge
libsqlite 3.39.3 hcfcfb64_0 conda-forge
libtiff 4.3.0 hc4061b1_4 conda-forge
libxml2 2.9.14 hf5bbc77_4 conda-forge
libzlib 1.2.12 hcfcfb64_3 conda-forge
m2w64-gcc-libgfortran 5.3.0 6 conda-forge
m2w64-gcc-libs 5.3.0 7 conda-forge
m2w64-gcc-libs-core 5.3.0 7 conda-forge
m2w64-gmp 6.1.0 2 conda-forge
m2w64-libwinpthread-git 5.0.0.4634.697f757 2 conda-forge
markdown 3.3.6 pypi_0 pypi
matplotlib 3.5.3 pypi_0 pypi
mkl 2022.1.0 h6a75c08_874 conda-forge
msys2-conda-epoch 20160418 1 conda-forge
ndx-pose 0.1.1 pypi_0 pypi
networkx 2.6.3 pypi_0 pypi
numpy 1.19.5 py37h4c2b6ed_3 conda-forge
oauthlib 3.2.0 pypi_0 pypi
olefile 0.46 pyh9f0ad1d_1 conda-forge
opencv-python 4.5.5.62 pypi_0 pypi
opencv-python-headless 4.5.5.62 pypi_0 pypi
openh264 2.3.0 h0e60522_0 conda-forge
openjpeg 2.4.0 hb211442_1 conda-forge
openssl 3.0.5 hcfcfb64_2 conda-forge
opt-einsum 3.3.0 pypi_0 pypi
packaging 21.3 pyhd8ed1ab_0 conda-forge
pandas 1.3.5 py37h9386db6_0 conda-forge
pillow 8.4.0 py37hd7d9ad0_0 conda-forge
pip 22.2.2 pyhd8ed1ab_0 conda-forge
pkgutil-resolve-name 1.3.10 pypi_0 pypi
protobuf 3.19.4 pypi_0 pypi
psutil 5.9.2 pypi_0 pypi
pyasn1 0.4.8 pypi_0 pypi
pyasn1-modules 0.2.8 pypi_0 pypi
pygments 2.13.0 pypi_0 pypi
pykalman 0.9.5 pypi_0 pypi
pynwb 2.1.0 pypi_0 pypi
pyparsing 3.0.6 pypi_0 pypi
pyreadline 2.1 py37h03978a9_1006 conda-forge
pyrsistent 0.18.1 pypi_0 pypi
pyside2 5.14.1 pypi_0 pypi
python 3.7.12 h900ac77_100_cpython conda-forge
python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge
python-rapidjson 1.8 pypi_0 pypi
python_abi 3.7 2_cp37m conda-forge
pytz 2022.2.1 pyhd8ed1ab_0 conda-forge
pytz-deprecation-shim 0.1.0.post0 pypi_0 pypi
pywavelets 1.3.0 pypi_0 pypi
pyzmq 23.2.1 pypi_0 pypi
qimage2ndarray 1.9.0 pypi_0 pypi
qtpy 2.2.0 pyhd8ed1ab_0 conda-forge
requests 2.27.1 pypi_0 pypi
requests-oauthlib 1.3.1 pypi_0 pypi
rich 10.16.1 pypi_0 pypi
ruamel-yaml 0.17.21 pypi_0 pypi
ruamel-yaml-clib 0.2.6 pypi_0 pypi
scikit-image 0.19.3 pypi_0 pypi
scikit-learn 1.0.2 pypi_0 pypi
scikit-video 1.1.11 pypi_0 pypi
scipy 1.7.3 py37hb6553fb_0 conda-forge
seaborn 0.12.0 pypi_0 pypi
segmentation-models 1.0.1 pypi_0 pypi
setuptools 59.8.0 py37h03978a9_1 conda-forge
setuptools-scm 6.4.2 pypi_0 pypi
shapely 1.7.1 py37hc520ffa_5 conda-forge
shiboken2 5.14.1 pypi_0 pypi
six 1.15.0 pyh9f0ad1d_0 conda-forge
sleap 1.2.8 pypi_0 pypi
sqlite 3.39.3 hcfcfb64_0 conda-forge
svt-av1 1.2.1 h0e60522_0 conda-forge
tbb 2021.5.0 h91493d7_2 conda-forge
tensorboard 2.6.0 pypi_0 pypi
tensorboard-data-server 0.6.1 pypi_0 pypi
tensorboard-plugin-wit 1.8.1 pypi_0 pypi
tensorflow 2.6.3 pypi_0 pypi
tensorflow-estimator 2.6.0 pypi_0 pypi
termcolor 1.1.0 pypi_0 pypi
threadpoolctl 3.1.0 pypi_0 pypi
tifffile 2021.11.2 pypi_0 pypi
tk 8.6.12 h8ffe710_0 conda-forge
tomli 2.0.1 pypi_0 pypi
typing-extensions 3.10.0.2 pypi_0 pypi
tzdata 2022.2 pypi_0 pypi
tzlocal 4.2 pypi_0 pypi
ucrt 10.0.20348.0 h57928b3_0 conda-forge
urllib3 1.26.8 pypi_0 pypi
vc 14.2 hb210afc_7 conda-forge
vs2015_runtime 14.29.30139 h890b9b1_7 conda-forge
werkzeug 2.0.3 pypi_0 pypi
wheel 0.37.1 pyhd8ed1ab_0 conda-forge
wrapt 1.12.1 pypi_0 pypi
x264 1!164.3095 h8ffe710_2 conda-forge
x265 3.5 h2d74725_3 conda-forge
xz 5.2.6 h8d14728_0 conda-forge
zipp 3.7.0 pypi_0 pypi
zlib 1.2.12 hcfcfb64_3 conda-forge
zstd 1.5.2 h7755175_4 conda-forge
Logs
Writing video with 508 frame images...
INFO:sleap.io.visuals:Chunks: 8, chunk size: 64
Done in 1.3444831000000477 s, fps = 377.8403759779368.
Exception in thread Thread-12:
Traceback (most recent call last):
File "C:\Users\ttunj\Anaconda3\envs\sleap\lib\site-packages\skvideo\io\abstract.py", line 514, in writeFrame
self._proc.stdin.write(vid.tostring())
BrokenPipeError: [Errno 32] Broken pipe
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\ttunj\Anaconda3\envs\sleap\lib\threading.py", line 926, in _bootstrap_inner
self.run()
File "C:\Users\ttunj\Anaconda3\envs\sleap\lib\threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\ttunj\Anaconda3\envs\sleap\lib\site-packages\sleap\io\visuals.py", line 148, in writer
raise e
File "C:\Users\ttunj\Anaconda3\envs\sleap\lib\site-packages\sleap\io\visuals.py", line 135, in writer
writer_object.add_frame(img, bgr=True)
File "C:\Users\ttunj\Anaconda3\envs\sleap\lib\site-packages\sleap\io\videowriter.py", line 108, in add_frame
self._writer.writeFrame(img)
File "C:\Users\ttunj\Anaconda3\envs\sleap\lib\site-packages\skvideo\io\abstract.py", line 519, in writeFrame
raise IOError(msg)
OSError: [Errno 32] Broken pipe
FFMPEG COMMAND:
C:\Users\ttunj\Anaconda3\envs\sleap\Library\bin/ffmpeg.exe -y -r 25 -f rawvideo -pix_fmt rgb24 -s 1272x909 -i - -c:v libx264 -preset superfast -framerate 25 -crf 21 -pix_fmt yuv420p C:\Users\ttunj\OneDrive\Desktop\labels_whole_mice.v002.slp.mp4
FFMPEG STDERR OUTPUT:
How to reproduce
- Go to 'Viewn'
- Click on 'rendering video with clip instances.'
- See error
Hi @ttunja,
Thanks for creating the issue - just double checking that the log above is from your most recent try (with the latest version of SLEAP)?
Thanks, Liezl
Strange -- seems to be failing to write the images!
At the end of your logs it says:
FFMPEG COMMAND:
C:\Users\ttunj\Anaconda3\envs\sleap\Library\bin/ffmpeg.exe -y -r 25 -f rawvideo -pix_fmt rgb24 -s 1272x909 -i - -c:v libx264 -preset superfast -framerate 25 -crf 21 -pix_fmt yuv420p C:\Users\ttunj\OneDrive\Desktop\labels_whole_mice.v002.slp.mp4
FFMPEG STDERR OUTPUT:
Do you see anything below the last line that says "FFMPEG STDERR OUTPUT"?
Hi @roomrys, yes this is the most recent try (version 1.2.8)
Hi @talmo, no there is nothing, it is just "hanging" there, and one would need to exit the sleap-label GUI in order to write again.
Gotcha -- I think we'll need to troubleshoot on our end then. I think one of our recent changes to the video writer might be masking the true error in this case.
Do you mind sharing your video(s) + .slp project file with us? You can send it to [email protected] in a Google Drive folder, or however you prefer.
We'll get to the bottom of this!
Thank you, I have sent out the email titled: "Githbub bug slp files and video."
Hi @ttunja,
Thanks for sharing the data! I found the issues.
Issue 1: The rendering.
The problem here is that your video resolution is not divisible 2, which is a requirement for x264 compression. SLEAP renders the visualization video at the same resolution as the source video, so it was failing when trying to write out the clip at an odd-valued video size.
This is evident when trying to re-encode it in ffmpeg, which throws this error:
[libx264 @ 0x125808e00] height not divisible by 2 (1272x909)
Unfortunately, due to #842, SLEAP wasn't displaying that error which is why we didn't catch it earlier in the logs. We'll make a note of that and maybe just account for it during rendering.
In the meantime, I got around this by re-encoding your video with a 1 pixel cropping like this:
ffmpeg -i 4Mice_20140219T151840_1.avi -c:v libx264 -preset superfast -vf "scale=trunc(iw/2)*2:trunc(ih/2)*2" -crf 23 -pix_fmt yuv420p 4Mice_20140219T151840_1.sf.avi
(You can also pad it instead by changing the trunc to ceil I think.)
After reencoding the video with the new size, you can go to SLEAP -> File -> Replace videos... and select the new video to replace the old one. All the poses will stay in the same place since the change in video size is from the bottom-right side.
Tada: https://user-images.githubusercontent.com/3187454/190829583-c5fa5cfb-7f8e-434c-880d-494288edcb24.mp4
Issue 2: Labeling procedure.
It looks like you're putting the landmarks in the right locations, but you're leaving them all marked "not visible" (except for the head node in most cases):
The visibility is indicated by the grey text with a dark background. Right click those nodes to mark them as visible.
Once you've done that, you should be able to retrain and get models that detect all of the landmarks on each animal.
BTW: This might actually address most of the problems that you were trying to solve by tuning the hyperparameters in #948. I'll follow up on there as well.
Give these a go and let us know if you're running into any issues!
Cheers,
Talmo
Hi @talmo @roomrys,
thank you so much for this it works. One final small step, once I export the video, its not exported with the trail length. I find this super important for a viewer to follow through. How to include trail length in the exported video?
Once the coloured trail is there, is there any way to access the colour from the h5 file?
Hi @ttunja,
We currently do not overlay the trails in the video, but it is a great idea! Please post this to the ideas page if you'd like, and we will see that it gets added.
The trail color is assigned in the GUI based on the user's preferred color scheme, but is not stored in the exported analysis file.
Thanks, Liezl
This feature is now available in the (pre) release 1.3.0a0, to install, first uninstall and then:
conda (Windows/Linux/GPU):
conda create -y -n sleap -c sleap -c sleap/label/dev -c nvidia -c conda-forge sleap=1.3.0a0
pip (any OS except Apple Silicon):
pip install sleap==1.3.0a0
Warning: This is a pre-release! Expect bugs and strange behavior when testing.