qaz812345
Inference issue
Hi @Va6lue,
I have made some updates to dataset.py. Please let me know if the issue still exists.
I can see the progress bar begins at 0%.
However, I find out that my 32GB RAM explodes just in a few seconds, so I terminate the process right after that.
Typically, predict.py won't consume 32GB of memory unless it is dealing with large video inputs. Based on your description, I require more information to assist you effectively. Specifically, I need additional details about any error messages, your input video, and your environment setup. Providing this information will help me better understand your issue and offer accurate assistance.
My input video file is 1920x1080, 25fps and only 7 seconds. This kind of memory explosion occurs on my two systems. Unfortunately, there is no any error message coming out. The process gradually consumes all the system memory in a few seconds, and causes system stutters.
System 1 (i5-11400, 16GB RAM, RTX 3060 12GB): Windows 11 (latest version) Python 3.11.7 64bit PyTorch 2.1.2+cu121
details
absl-py==2.1.0 addict==2.4.0 albumentations==1.4.0 anyio==4.2.0 argon2-cffi==23.1.0 argon2-cffi-bindings==21.2.0 arrow==1.3.0 asttokens==2.4.1 async-lru==2.0.4 attrs==23.2.0 Babel==2.14.0 beautifulsoup4==4.12.3 bleach==6.1.0 cachetools==5.3.2 certifi==2022.12.7 cffi==1.16.0 charset-normalizer==2.1.1 colorama==0.4.6 comm==0.2.1 contourpy==1.2.0 cycler==0.12.1 debugpy==1.8.0 decorator==4.4.2 defusedxml==0.7.1 et-xmlfile==1.1.0 executing==2.0.1 fastjsonschema==2.19.1 filelock==3.9.0 fonttools==4.49.0 fqdn==1.5.1 fsspec==2023.4.0 gitdb==4.0.11 GitPython==3.1.42 google-auth==2.27.0 google-auth-oauthlib==1.2.0 grpcio==1.60.0 idna==3.4 imageio==2.34.0 imageio-ffmpeg==0.4.9 ipykernel==6.29.0 ipython==8.20.0 ipywidgets==8.1.1 isoduration==20.11.0 jedi==0.19.1 Jinja2==3.1.2 joblib==1.3.2 json5==0.9.14 jsonpointer==2.4 jsonschema==4.21.1 jsonschema-specifications==2023.12.1 jupyter==1.0.0 jupyter-console==6.6.3 jupyter-events==0.9.0 jupyter-lsp==2.2.2 jupyter_client==8.6.0 jupyter_core==5.7.1 jupyter_server==2.12.5 jupyter_server_terminals==0.5.2 jupyterlab==4.0.11 jupyterlab-widgets==3.0.9 jupyterlab_pygments==0.3.0 jupyterlab_server==2.25.2 kiwisolver==1.4.5 lazy_loader==0.3 Markdown==3.5.2 MarkupSafe==2.1.3 matplotlib==3.8.3 matplotlib-inline==0.1.6 mistune==3.0.2 moviepy==1.0.3 mpmath==1.3.0 nbclient==0.9.0 nbconvert==7.14.2 nbformat==5.9.2 nest-asyncio==1.6.0 networkx==3.0 notebook==7.0.7 notebook_shim==0.2.3 numpy==1.26.3 oauthlib==3.2.2 opencv-python==4.9.0.80 opencv-python-headless==4.9.0.80 openpyxl==3.1.2 overrides==7.6.0 packaging==23.2 pandas==2.2.1 pandocfilters==1.5.1 parse==1.20.1 parso==0.8.3 Pillow==9.3.0 platformdirs==4.1.0 proglog==0.1.10 prometheus-client==0.19.0 prompt-toolkit==3.0.43 protobuf==4.23.4 psutil==5.9.8 pure-eval==0.2.2 py-cpuinfo==9.0.0 pyarrow==15.0.0 pyasn1==0.5.1 pyasn1-modules==0.3.0 pycocotools==2.0.7 pycparser==2.21 Pygments==2.17.2 pyparsing==3.1.1 python-dateutil==2.8.2 python-json-logger==2.0.7 pytz==2023.3.post1 pywin32==306 pywinpty==2.0.12 PyYAML==6.0.1 pyzmq==25.1.2 qqdm==0.0.7 qtconsole==5.5.1 QtPy==2.4.1 qudida==0.0.4 referencing==0.32.1 requests==2.31.0 requests-oauthlib==1.3.1 rfc3339-validator==0.1.4 rfc3986-validator==0.1.1 rpds-py==0.17.1 rsa==4.9 scikit-image==0.22.0 scikit-learn==1.4.1.post1 scipy==1.12.0 seaborn==0.13.2 Send2Trash==1.8.2 six==1.16.0 smmap==5.0.1 sniffio==1.3.0 soupsieve==2.5 stack-data==0.6.3 sympy==1.12 tensorboard==2.15.1 tensorboard-data-server==0.7.2 terminado==0.18.0 thop==0.1.1.post2209072238 threadpoolctl==3.3.0 tifffile==2024.2.12 tinycss2==1.2.1 torch==2.1.2+cu121 torchaudio==2.1.2+cu121 torchvision==0.16.2+cu121 tornado==6.4 tqdm==4.66.1 traitlets==5.14.1 types-python-dateutil==2.8.19.20240106 typing_extensions==4.4.0 tzdata==2023.4 ultralytics==8.1.24 uri-template==1.3.0 urllib3==1.26.13 wcwidth==0.2.13 webcolors==1.13 webencodings==0.5.1 websocket-client==1.7.0 Werkzeug==3.0.1 widgetsnbextension==4.0.9
System 2 (R7-7700, 32GB RAM, RTX 4090 24GB): Windows 11 (latest version) Python 3.11.6 64bit PyTorch 2.2.1+cu121
details
absl-py==2.0.0 addict==2.4.0 albumentations==1.4.0 anyio==4.2.0 argon2-cffi==23.1.0 argon2-cffi-bindings==21.2.0 arrow==1.3.0 asttokens==2.4.1 async-lru==2.0.4 attrs==23.2.0 Babel==2.14.0 beautifulsoup4==4.12.2 bleach==6.1.0 blinker==1.7.0 cachetools==5.3.2 certifi==2022.12.7 cffi==1.16.0 charset-normalizer==2.1.1 click==8.1.7 colorama==0.4.6 comm==0.2.1 contourpy==1.2.0 cycler==0.12.1 dash==2.15.0 dash-core-components==2.0.0 dash-html-components==2.0.0 dash-table==5.0.0 debugpy==1.8.0 decorator==4.4.2 defusedxml==0.7.1 einops==0.7.0 et-xmlfile==1.1.0 executing==2.0.1 fastjsonschema==2.19.1 filelock==3.9.0 Flask==3.0.2 fonttools==4.47.2 fqdn==1.5.1 fsspec==2023.12.2 gitdb==4.0.11 GitPython==3.1.42 google-auth==2.26.2 google-auth-oauthlib==1.2.0 grpcio==1.60.0 huggingface-hub==0.20.2 idna==3.4 imageio==2.34.0 imageio-ffmpeg==0.4.9 importlib-metadata==7.0.1 ipykernel==6.28.0 ipython==8.20.0 ipywidgets==8.1.1 isoduration==20.11.0 itsdangerous==2.1.2 jedi==0.19.1 Jinja2==3.1.2 joblib==1.3.2 json5==0.9.14 jsonpointer==2.4 jsonschema==4.20.0 jsonschema-specifications==2023.12.1 jupyter==1.0.0 jupyter-console==6.6.3 jupyter-events==0.9.0 jupyter-lsp==2.2.1 jupyter_client==8.6.0 jupyter_core==5.7.1 jupyter_server==2.12.4 jupyter_server_terminals==0.5.1 jupyterlab==4.0.10 jupyterlab-widgets==3.0.9 jupyterlab_pygments==0.3.0 jupyterlab_server==2.25.2 kiwisolver==1.4.5 lazy_loader==0.3 Markdown==3.5.2 MarkupSafe==2.1.2 matplotlib==3.8.2 matplotlib-inline==0.1.6 mistune==3.0.2 moviepy==1.0.3 mpmath==1.3.0 nbclient==0.9.0 nbconvert==7.14.1 nbformat==5.9.2 nest-asyncio==1.5.8 networkx==3.0 notebook==7.0.6 notebook_shim==0.2.3 numpy==1.26.4 oauthlib==3.2.2 opencv-python==4.9.0.80 opencv-python-headless==4.9.0.80 openpyxl==3.1.2 overrides==7.4.0 packaging==23.2 pandas==2.2.1 pandocfilters==1.5.0 parse==1.20.1 parso==0.8.3 pillow==10.2.0 platformdirs==4.1.0 plotly==5.19.0 proglog==0.1.10 prometheus-client==0.19.0 prompt-toolkit==3.0.43 protobuf==4.23.4 psutil==5.9.7 pure-eval==0.2.2 py-cpuinfo==9.0.0 pyarrow==15.0.0 pyasn1==0.5.1 pyasn1-modules==0.3.0 pycocotools==2.0.7 pycparser==2.21 Pygments==2.17.2 pyparsing==3.1.1 python-dateutil==2.8.2 python-json-logger==2.0.7 pytz==2023.3.post1 pywin32==306 pywinpty==2.0.12 PyYAML==6.0.1 pyzmq==25.1.2 qqdm==0.0.7 qtconsole==5.5.1 QtPy==2.4.1 qudida==0.0.4 referencing==0.32.1 regex==2023.12.25 requests==2.31.0 requests-oauthlib==1.3.1 retrying==1.3.4 rfc3339-validator==0.1.4 rfc3986-validator==0.1.1 rpds-py==0.17.1 rsa==4.9 safetensors==0.4.1 scikit-image==0.22.0 scikit-learn==1.4.1.post1 scipy==1.12.0 seaborn==0.13.2 Send2Trash==1.8.2 simplekml==1.3.6 six==1.16.0 smmap==5.0.1 sniffio==1.3.0 soupsieve==2.5 stack-data==0.6.3 sympy==1.12 tenacity==8.2.3 tensorboard==2.16.2 tensorboard-data-server==0.7.2 terminado==0.18.0 thop==0.1.1.post2209072238 threadpoolctl==3.3.0 tifffile==2024.2.12 tinycss2==1.2.1 tokenizers==0.15.0 torch==2.2.1+cu121 torchaudio==2.2.1+cu121 torchvision==0.17.1+cu121 tornado==6.4 tqdm==4.66.2 traitlets==5.14.1 transformers==4.38.2 types-python-dateutil==2.8.19.20240106 typing_extensions==4.9.0 tzdata==2023.3 ultralytics==8.1.24 uri-template==1.3.0 urllib3==1.26.13 wcwidth==0.2.13 webcolors==1.13 webencodings==0.5.1 websocket-client==1.7.0 Werkzeug==3.0.1 widgetsnbextension==4.0.9 zipp==3.17.0
In predict or test has a code "frame_list, fps, (w, h) = generate_frames(args.video_file)", and this will get all frames from video, which is saved in a cache list, may get the RAM out of memory.
In predict or test has a code
frame_list, fps, (w, h) = generate_frames(args.video_file), and this will get all frames from video, which is saved in a cache list, may get the RAM out of memory.
We have the same issue when using Tracknetv3 with a full 50 minutes 720p badminton match video. The line frame_list, fps, (w, h) = generate_frames(args.video_file) eats up to 400G of RAM (on the server) before crashing (ie. not enough RAM remaining to charge all the frames). We're trying to make a sliding windows, to charge chunks of videos instead of the whole video and feed generate_frames(args.video_file) with these chunks.
Hi @Geoffroy-Berthelot,
Sorry for the late response. I've added an IterableDataset to address the OOM issue during large video inference. You can find more details here.
If you have any further questions, please let me know.
Now using --large_video can infer successfully, but the inference speed is slow.
However, there is another small problem in the inference result. I find out that the "Visibility" is always 1, even though there is no shuttlecock in the specific frame or even the frames in a period of time.
Hi @Va6lue,
About the inference speed, I am still searching a better implementation. If you have any advice, it would be greatly appreciated.
For "Visibility," in our implementation, it will be 1 when either the "X" or "Y" coordinate of the shuttlecock is greater than 0. Do you mean that you have results where "Visibility" is 1 while both "X" and "Y" are 0? Or are you suggesting that the model mistakenly detects another object as the shuttlecock? I may need some examples to better understand the problem.
Yeah, there are X and Y coordinate values both greater than 0 when the "Visibility" is 1.
The scenario is that when the inference frames don't have any shuttlecocks (we can easily determine where the shuttlecocks should be or even shouldn't be there), the model still predicted them for example in my case (X, Y) = (609, 178). Moreover, I found out that these faulse predictions (X, Y) are usually stuck at (609, 178) on my 720p video, but the other version of "TrackNetV3" with a different model architecture also based on TrackNetV2 infering the same 720p video can give a correct "Visibility"=0 in some cases that there isn't any shuttlecock actually there.
Although this isn't a big problem.