FAST_Anime_VSR
FAST_Anime_VSR copied to clipboard
Kiteretsu77
tensorRT installation + Running instruction
I'm running: Windows 10 Pro 10.0.19044 Build 19044 Python 3.7.12 CUDA 11.7 cuDNN 8.6 (I also tried 8.9.0 -as suggested at NVIDIA doc) RTX 3090 FE
Trying to install "tensorrt" following author's suggested #3 step (from TensorRT 8.6 GA - TensorRT 8.6 GA for Windows 10 and CUDA 11.0, 11.1, 11.2, 11.3, 11.4, 11.5, 11.6, 11.7 and 11.8 ZIP Package](https://developer.nvidia.com/downloads/compute/machine-learning/tensorrt/secure/8.6.1/zip/TensorRT-8.6.1.6.Windows10.x86_64.cuda-11.8.zip):
python.exe -m pip install tensorrt-8.6.1-cp37-none-win_amd64.whl
Installation passed successfully and "tensorrt" appears among "conda list" installed packages Unfortunately attempt "import tensorrt" throws an error:
>>> import tensorrt
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "c:\programdata\miniconda3\lib\site-packages\tensorrt\__init__.py", line 129, in <module>
ctypes.CDLL(find_lib(lib))
File "c:\programdata\miniconda3\lib\ctypes\__init__.py", line 364, in __init__
self._handle = _dlopen(self._name, mode)
OSError: [WinError 126] The specified module could not be found
>>>
This error happens regardless "graphsurgeon,uff,onnx_graphsurgeon" are installed. Is there any special instruction author can provide me with? My great appreciation for suggestions.
Thanks for the update! Personally, I recommend checking the step4 of https://docs.nvidia.com/deeplearning/tensorrt/install-guide/index.html#installing-zip It is necessary to make sure that the environment variable is set correctly. I am not sure if there is any other step needed to go on with conda. Please be aware of the conda container difference. (I didn't install tensorRT inside conda, so I am not very sure about conda part)
Thanks for the prompt response. The problem was in CUDA and CuDNN versions. For my RTX 3090 - I installed CUDA 12 and cuDNN 8.9 and TensorRT 8.6 GA for CUDA 12 - after that everything went flawlessly. However attempt to run main.py with input video in ProRes 422 HQ format it throws an error "This format is not supported". What video and single image formats are supported?
The format I have tested is "mp4" and "mkv". For other input video formats, I didn't test them, so on line 11 of mass_production.py, I didn't add other accepted formats. You can add your input video formats to the accepted_format lists. Since the backend of video decode depends on moviepy which calls FFMPEG API. As long as the ffmpeg supports it. I think that it should be fine.
The format I have tested is "mp4" and "mkv". For other input video formats, I didn't test them, so on line 11 of mass_production.py, I didn't add other accepted formats. You can add your input video formats to the accepted_format lists. Since the backend of video decode depends on moviepy which calls FFMPEG API. As long as the ffmpeg supports it. I think that it should be fine.
I've add "mov" format on line 11, however mass_production.py ended up without an error and no processing. ffmpeg supports this format. Now I'm testing MP4, so far the processing keeps going.
Gotcha! Do you have any sample video input sources? Can you share a short duration of that video with me? I think that directly debugging on my side will be faster to debug this. Thanks!
Gotcha! Do you have any sample video input sources? Can you share a short duration of that video with me? I think that directly debugging on my side will be faster to debug this. Thanks!
I'll send you sample tomorrow (here11.00PM). BTW so far only main.py processes the mp4 video, mass_production.py and single_production ended up without an error and no processing MP4. Also processed video produced by main.py is not synchronized with the original video (frames in processed video are not match with the frames in original. Another question - is there an option to control noise reduction level (similar option "alfa" in RealCugan)? Thanks for your kind support.
For "Also processed video produced by main.py is not synchronized with the original video (frames in processed video are not match with the frames in original." This is because I map all videos to 24 FPS. This part is talked about in the document slide (https://docs.google.com/presentation/d/1Gxux9MdWxwpnT4nDZln8Ip_MeqalrkBesX34FVupm2A/edit#slide=id.p) I map to 24 FPS because Anime is created for 24 FPS/ 23.98 FPS. Thus, some 30 or more FPS anime video is falsely interpolated with unnecessary frames from my perspective. I directly map all video decoding with 24 FPS because this will contain the most essential frame information. If you want to change your target fps, you can modify line 116 of inference.py. You can set self.target_fps to be the fps you want. BTW, I will also recently modify this part to be something that can be controlled by config.py directly. I think that this is a good part to modify to me because not everyone needs 24FPS video.
For the "alpha" in the original RealCuGAN, I think this parameter is changing the line strength of the image (e.g. sharpness of objects and characters' edges) instead of the noise reduction level. To control noise reduction levels, I recommend downloading another pre-trained (pth) weight from the weight provided (https://drive.google.com/drive/folders/1jAJyBf2qKe2povySwsGXsVMnzVyQzqDD). (Since currently I only support 2x, so please only download up2x for now.) And rename that weight file as "cunet_weight.pth" and put it under the "weights/" folder. (Don't forget to delete the original weight.)
Here the link to sample video in the ProRes format I’ve mentioned https://drive.google.com/file/d/1Pt0Vj_tY0CS9DD4zdzTT_wdtxhEyygi2/view?usp=share_link
(original FPS 23.976)
From: Kiteretsu77 @.> Sent: Thursday, May 11, 2023 8:32 AM To: Kiteretsu77/FAST_Anime_VSR @.> Cc: makemota @.>; Author @.> Subject: Re: [Kiteretsu77/FAST_Anime_VSR] tensorRT installation instruction (Issue #1)
For "Also processed video produced by main.py is not synchronized with the original video (frames in processed video are not match with the frames in original." This is because I map all videos to 24 FPS. This part is talked about in the document slide (https://docs.google.com/presentation/d/1Gxux9MdWxwpnT4nDZln8Ip_MeqalrkBesX34FVupm2A/edit#slide=id.p) I map to 24 FPS because Anime is created for 24 FPS/ 23.98 FPS. Thus, some 30 or more FPS anime video is falsely interpolated with unnecessary frames from my perspective. I directly map all video decoding with 24 FPS because this will contain the most essential frame information. If you want to change your target fps, you can modify line 116 of inference.py. You can set self.target_fps to be the fps you want. BTW, I will also recently modify this part to be something that can be controlled by config.py directly. I think that this is a good part to modify to me because not everyone needs 24FPS video.
For the "alpha" in the original RealCuGAN, I think that that parameter is changing the line strength of the image. This part is not explicitly said in their English document, but it's said in their Chinese document. To control noise reduction levels, I recommend downloading another pre-trained (pth) weight from the weight provided (https://github.com/bilibili/ailab/releases/tag/Real-CUGAN). And rename that weight as "cunet_weight.pth" and put it under the "weights/" folder. (Don't forget to delete the original weight.)
— Reply to this email directly, view it on GitHub https://github.com/Kiteretsu77/FAST_Anime_VSR/issues/1#issuecomment-1544206434 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AXTGB37URJRSWOQDJLQD763XFUA5NANCNFSM6AAAAAAX4BGEQY . You are receiving this because you authored the thread. https://github.com/notifications/beacon/AXTGB33TCTW6H6IH3IH3V2TXFUA5NA5CNFSM6AAAAAAX4BGEQ2WGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTS4BK4GE.gif Message ID: @.*** @.***> >
Tried the last release. Still "mass_production.py" and "single_process.py do nothing with input in any format ("mp4" and "mkv"). "mov" format still is not assepted. After add "mov" on line 11 of mass_production.py got an error running main.py:
Microsoft Windows [Version 10.0.19044.2075]
(c) Microsoft Corporation. All rights reserved.
D:\Restoration\FAST_Anime_VSR-main>python main.py
In c:\programdata\miniconda3\lib\site-packages\matplotlib\mpl-data\stylelib\_classic_test.mplstyle:
The text.latex.unicode rcparam was deprecated in Matplotlib 3.0 and will be removed in 3.2.
In c:\programdata\miniconda3\lib\site-packages\matplotlib\mpl-data\stylelib\_classic_test.mplstyle:
The savefig.frameon rcparam was deprecated in Matplotlib 3.1 and will be removed in 3.3.
In c:\programdata\miniconda3\lib\site-packages\matplotlib\mpl-data\stylelib\_classic_test.mplstyle:
The pgf.debug rcparam was deprecated in Matplotlib 3.0 and will be removed in 3.2.
In c:\programdata\miniconda3\lib\site-packages\matplotlib\mpl-data\stylelib\_classic_test.mplstyle:
The verbose.level rcparam was deprecated in Matplotlib 3.1 and will be removed in 3.3.
In c:\programdata\miniconda3\lib\site-packages\matplotlib\mpl-data\stylelib\_classic_test.mplstyle:
The verbose.fileo rcparam was deprecated in Matplotlib 3.1 and will be removed in 3.3.
We are going to process all videos in input
All files begin
input\test_original.mov output\test_original_processed.mp4
1
ffmpeg version N-108614-g37ee36f689-20221011 Copyright (c) 2000-2022 the FFmpeg developers
built with gcc 12.1.0 (crosstool-NG 1.25.0.55_3defb7b)
configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static --pkg-config=pkg-config --cross-prefix=x86_64-w64-mingw32- --arch=x86_64 --target-os=mingw32 --enable-gpl --enable-version3 --disable-debug --disable-w32threads --enable-pthreads --enable-iconv --enable-libxml2 --enable-zlib --enable-libfreetype --enable-libfribidi --enable-gmp --enable-lzma --enable-fontconfig --enable-libvorbis --enable-opencl --disable-libpulse --enable-libvmaf --disable-libxcb --disable-xlib --enable-amf --enable-libaom --enable-libaribb24 --enable-avisynth --enable-libdav1d --enable-libdavs2 --disable-libfdk-aac --enable-ffnvcodec --enable-cuda-llvm --enable-frei0r --enable-libgme --enable-libkvazaar --enable-libass --enable-libbluray --enable-libjxl --enable-libmp3lame --enable-libopus --enable-librist --enable-libssh --enable-libtheora --enable-libvpx --enable-libwebp --enable-lv2 --enable-libmfx --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-librav1e --enable-librubberband --enable-schannel --enable-sdl2 --enable-libsoxr --enable-libsrt --enable-libsvtav1 --enable-libtwolame --enable-libuavs3d --disable-libdrm --disable-vaapi --enable-libvidstab --enable-vulkan --enable-libshaderc --enable-libplacebo --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libzimg --enable-libzvbi --extra-cflags=-DLIBTWOLAME_STATIC --extra-cxxflags= --extra-ldflags=-pthread --extra-ldexeflags= --extra-libs=-lgomp --extra-version=20221011
libavutil 57. 39.100 / 57. 39.100
libavcodec 59. 50.100 / 59. 50.100
libavformat 59. 34.101 / 59. 34.101
libavdevice 59. 8.101 / 59. 8.101
libavfilter 8. 49.101 / 8. 49.101
libswscale 6. 8.112 / 6. 8.112
libswresample 4. 9.100 / 4. 9.100
libpostproc 56. 7.100 / 56. 7.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'input\test_original.mov':
Metadata:
major_brand : qt
minor_version : 512
compatible_brands: qt
encoder : Lavf57.79.100
Duration: 00:00:05.92, start: 0.000000, bitrate: 87588 kb/s
Stream #0:0[0x1](eng): Video: prores (HQ) (apch / 0x68637061), yuv422p10le(tv, bt470bg/unknown/unknown), 1440x1080, 87586 kb/s, 23.98 fps, 23.98 tbr, 24k tbn (default)
Metadata:
handler_name : VideoHandler
vendor_id : FFMP
Stream map '0:s:0' matches no streams.
To ignore this, add a trailing '?' to the map.
ffmpeg version N-108614-g37ee36f689-20221011 Copyright (c) 2000-2022 the FFmpeg developers
built with gcc 12.1.0 (crosstool-NG 1.25.0.55_3defb7b)
configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static --pkg-config=pkg-config --cross-prefix=x86_64-w64-mingw32- --arch=x86_64 --target-os=mingw32 --enable-gpl --enable-version3 --disable-debug --disable-w32threads --enable-pthreads --enable-iconv --enable-libxml2 --enable-zlib --enable-libfreetype --enable-libfribidi --enable-gmp --enable-lzma --enable-fontconfig --enable-libvorbis --enable-opencl --disable-libpulse --enable-libvmaf --disable-libxcb --disable-xlib --enable-amf --enable-libaom --enable-libaribb24 --enable-avisynth --enable-libdav1d --enable-libdavs2 --disable-libfdk-aac --enable-ffnvcodec --enable-cuda-llvm --enable-frei0r --enable-libgme --enable-libkvazaar --enable-libass --enable-libbluray --enable-libjxl --enable-libmp3lame --enable-libopus --enable-librist --enable-libssh --enable-libtheora --enable-libvpx --enable-libwebp --enable-lv2 --enable-libmfx --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-librav1e --enable-librubberband --enable-schannel --enable-sdl2 --enable-libsoxr --enable-libsrt --enable-libsvtav1 --enable-libtwolame --enable-libuavs3d --disable-libdrm --disable-vaapi --enable-libvidstab --enable-vulkan --enable-libshaderc --enable-libplacebo --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libzimg --enable-libzvbi --extra-cflags=-DLIBTWOLAME_STATIC --extra-cxxflags= --extra-ldflags=-pthread --extra-ldexeflags= --extra-libs=-lgomp --extra-version=20221011
libavutil 57. 39.100 / 57. 39.100
libavcodec 59. 50.100 / 59. 50.100
libavformat 59. 34.101 / 59. 34.101
libavdevice 59. 8.101 / 59. 8.101
libavfilter 8. 49.101 / 8. 49.101
libswscale 6. 8.112 / 6. 8.112
libswresample 4. 9.100 / 4. 9.100
libpostproc 56. 7.100 / 56. 7.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'input\test_original.mov':
Metadata:
major_brand : qt
minor_version : 512
compatible_brands: qt
encoder : Lavf57.79.100
Duration: 00:00:05.92, start: 0.000000, bitrate: 87588 kb/s
Stream #0:0[0x1](eng): Video: prores (HQ) (apch / 0x68637061), yuv422p10le(tv, bt470bg/unknown/unknown), 1440x1080, 87586 kb/s, 23.98 fps, 23.98 tbr, 24k tbn (default)
Metadata:
handler_name : VideoHandler
vendor_id : FFMP
Stream map '0:a' matches no streams.
To ignore this, add a trailing '?' to the map.
All Processes Start
In c:\programdata\miniconda3\lib\site-packages\matplotlib\mpl-data\stylelib\_classic_test.mplstyle:
The text.latex.unicode rcparam was deprecated in Matplotlib 3.0 and will be removed in 3.2.
In c:\programdata\miniconda3\lib\site-packages\matplotlib\mpl-data\stylelib\_classic_test.mplstyle:
The savefig.frameon rcparam was deprecated in Matplotlib 3.1 and will be removed in 3.3.
In c:\programdata\miniconda3\lib\site-packages\matplotlib\mpl-data\stylelib\_classic_test.mplstyle:
The pgf.debug rcparam was deprecated in Matplotlib 3.0 and will be removed in 3.2.
In c:\programdata\miniconda3\lib\site-packages\matplotlib\mpl-data\stylelib\_classic_test.mplstyle:
The verbose.level rcparam was deprecated in Matplotlib 3.1 and will be removed in 3.3.
In c:\programdata\miniconda3\lib\site-packages\matplotlib\mpl-data\stylelib\_classic_test.mplstyle:
The verbose.fileo rcparam was deprecated in Matplotlib 3.1 and will be removed in 3.3.
Set new attr for inp_path to be tmp/part0.mp4
Set new attr for opt_path to be tmp/part0_res.mp4
supported resolution is defaultdict(<class 'list'>, {1440: [1080, 368], 720: [188, 540], 960: [188, 540]})
Process Process-1:
Traceback (most recent call last):
File "C:\Users\Operator\AppData\Roaming\Python\Python37\site-packages\moviepy\video\io\ffmpeg_reader.py", line 285, in ffmpeg_parse_infos
line = [l for l in lines if keyword in l][index]
IndexError: list index out of range
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\programdata\miniconda3\lib\multiprocessing\process.py", line 297, in _bootstrap
self.run()
File "c:\programdata\miniconda3\lib\multiprocessing\process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "D:\Restoration\FAST_Anime_VSR-main\single_process.py", line 179, in single_process
config_preprocess(params, configuration)
File "D:\Restoration\FAST_Anime_VSR-main\single_process.py", line 86, in config_preprocess
weight_justify(config)
File "D:\Restoration\FAST_Anime_VSR-main\single_process.py", line 43, in weight_justify
video = VideoFileClip(config.inp_path)
File "C:\Users\Operator\AppData\Roaming\Python\Python37\site-packages\moviepy\video\io\VideoFileClip.py", line 91, in __init__
fps_source=fps_source)
File "C:\Users\Operator\AppData\Roaming\Python\Python37\site-packages\moviepy\video\io\ffmpeg_reader.py", line 36, in __init__
fps_source)
File "C:\Users\Operator\AppData\Roaming\Python\Python37\site-packages\moviepy\video\io\ffmpeg_reader.py", line 291, in ffmpeg_parse_infos
filename, infos))
OSError: MoviePy error: failed to read the duration of file tmp/part0.mp4.
Here are the file infos returned by ffmpeg:
ffmpeg version 4.2.2 Copyright (c) 2000-2019 the FFmpeg developers
built with gcc 9.2.1 (GCC) 20200122
configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libdav1d --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libmfx --enable-amf --enable-ffnvcodec --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth --enable-libopenmpt
libavutil 56. 31.100 / 56. 31.100
libavcodec 58. 54.100 / 58. 54.100
libavformat 58. 29.100 / 58. 29.100
libavdevice 58. 8.100 / 58. 8.100
libavfilter 7. 57.100 / 7. 57.100
libswscale 5. 5.100 / 5. 5.100
libswresample 3. 5.100 / 3. 5.100
libpostproc 55. 5.100 / 55. 5.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0000020abdc99700] Format mov,mp4,m4a,3gp,3g2,mj2 detected only with low score of 1, misdetection possible!
[mov,mp4,m4a,3gp,3g2,mj2 @ 0000020abdc99700] moov atom not found
tmp/part0.mp4: Invalid data found when processing input
All Processes End
Total time spent for this video is 0 min 4 s
After finish one thing, sleep for a moment!
I think that my newest commit (just pushed) should work on your file (have tested). The bug is due to the file format I set before. In the past, I segment videos into several parts with some accelerated ffmpeg command. For this command, they could only support video in the same format (container). Thus, I used to save all of the files as "mp4"; now, I add a mechanism to automatically detect your input video source and it's not needed to fear the input format issue now.
Your last commit works perfectly accepting MOV. Thanks a lot for your kind support. However processed video stays in TMP folder not in specified OUTPUT in config. "mass_production.py" and "single_process.py still do nothing
Done! I think that this bug is related to an edge of the case of silent video input. In the past, I tested all videos with audio. When there is a video without audio, the combine_video function cannot work well because I didn't add a checker there. I think that the newest commit should be fine now. Thanks!
Is there is a way to output processed video in the same format as input or (in the worst case scenario) as image sequence. The problem is that the MP4 compressed stream couldn't be edited frame by frame but keyframes only (even with CRF=0). Actually MP4 is consumer format that doesn't accepted for professional usage. Please consider. Thanks!
This is a good idea. Generally, using "ffmpeg -i .\input.mp4 -acodec copy -vcodec copy .\output.mov" will convert the format in less than a second. The audio and video codec can be directly copied to a new container. I think that I will set up a parameter at config.py to let users choose what kind of output format they want. Thanks for the advice!
This is a good idea. Generally, using "ffmpeg -i .\input.mp4 -acodec copy -vcodec copy .\output.mov" will convert the format in less than a second. The audio and video codec can be directly copied to a new container. I think that I will set up a parameter at config.py to let users choose what kind of output format they want. Thanks for the advice!
Any update on ProRes output?
