cdgriffith
NVEnc crashes after conversion from MP4 source to MKV AV1 NVenC
FastFlix Version: 5.12.2 Operating System:
Describe the bug A clear and concise description of what the bug is. When coding an MP4 video file from h264 to AV1 using NVEncC AV1 profile, NVEnc crashes after coding is complete. It probably does not matter the source codec, just that the resulting file is AV1 NVencC and that it crashes at completion. Let it be noted that I have an RTX5090, the log states 4090. Doubt that has anything to do with it as MKV to AV1 does not crash.
To Reproduce Steps to reproduce the behavior Open an mp4 video file. Ensure setting is set to AV1 NvencC. Convert Away!
Screenshots If applicable, add screenshots to help explain your problem.
Logs (If applicable) In FastFlix you can go to Help > Open Log Directory and sort by latest to find the newest log file to add here.
flix_gui_2025-07-08T11.06.07.log
flix_conversion_Video1_2025-07-08T11.35.28.log
Please make sure to remove any personal information or video names in the screenshots and logs!
Quick look at your logs I see almost exact same as mine:
2025-07-08 11:35:29,169 - NVEncC (x64) 8.07 (r3354) by rigaya, Jun 23 2025 13:27:31 (VC 1944/Win)
2025-07-08 11:35:29,180 - OS Version Windows 11 x64 (26100) [UTF-8]
2025-07-08 11:35:29,192 - CPU 12th Gen Intel Core i9-12900K [5.02GHz] (8P+8E,16C/24T)
2025-07-08 11:35:29,204 - GPU #0: NVIDIA GeForce RTX 5090 (21760 cores, 2512 MHz)[PCIe5x16][576.80]
2025-07-08 11:35:29,216 - NVENC / CUDA NVENC API 13.0, CUDA 12.9, schedule mode: auto
2025-07-08 11:35:29,227 - Input Buffers CUDA, 44 frames
2025-07-08 11:35:29,238 - Input Info avsw: h264(yv12)->nv12 [AVX2], 1920x1080, 24000/1001 fps
2025-07-08 11:35:29,249 - Vpp Filters cspconv(nv12 -> yv12(16bit))
2025-07-08 11:35:29,259 - cspconv(yv12(16bit) -> p010)
2025-07-08 11:35:29,270 - Output Info AV1 main 10bit @ Level auto
2025-07-08 11:35:29,281 - 1920x1080p 1:1 23.976fps (24000/1001fps)
2025-07-08 11:35:29,292 - avwriter: av1, eac3 => matroska
Mine:
NVEncC (x64) 8.07 (r3354) by rigaya, Jun 23 2025 13:27:31 (VC 1944/Win)
OS Version Windows 11 x64 (26100) [UTF-8]
CPU AMD Ryzen 9 9950X3D 16-Core Processor [5.63GHz] (16C/32T)
GPU #0: NVIDIA GeForce RTX 5090 (21760 cores, 2407 MHz)[PCIe5x16][576.80]
NVENC / CUDA NVENC API 13.0, CUDA 12.9, schedule mode: auto
Input Buffers CUDA, 20 frames
Input Info avsw: hevc(yv12)->nv12 [AVX2], 1080x1920, 60000/1001 fps
Vpp Filters cspconv(nv12 -> yv12(16bit))
cspconv(yv12(16bit) -> p010)
Output Info AV1 main 10bit @ Level auto
1080x1920p 1:1 59.940fps (60000/1001fps)
avwriter: av1, aac => matroska
Using same NVEncC version and nvidia driver version. Only difference is I see you have your 5090 overclocked which maybe it's unstable for the encode is the only guess?
I would try updating to newest NVEncC as 8.08 recently came out, and updating driver and seeing if that fixes it. Otherwise may try lowering your overclock.
Greetings. I am not OCing my 5090, it is an MSI Vanguard, which has a higher 'base' clock. I can do entire folders of mkv files just fine. And as you see from the logs, GPU seldom goes over 10% load. Low enough that the fan doesn't ever kick on.