PeerTube
PeerTube copied to clipboard
Chocobozzz
Video stuck in "being transcoded" state
Describe the current behavior
If your video fails to transcode and you manually restart the transcoding job "run HLS transcoding" then if it fails again it will be stuck in the failed state. Re-running "run HLS transcode" errors out with "this video is already being transcoded" despite not being transcoded.
It's possible this will clear up on it's own, but I've tried to restart peertube with no success.
Steps to reproduce
Describe the expected behavior
No response
Additional information
- PeerTube instance:
- Version: 5.1.0-rc.1...2c525a546
- NodeJS version: 18
- Ffmpeg version: git
Hi,
I'll see if I can't recreate the issue. Relevant logs:
error[3/13/2023, 5:18:37 PM] Error in ffmpeg.
{
"stdout": "",
"stderr": " Metadata:\n encoder : Lavf60.4.100\n Stream #0:0(eng): Video: av1, p010le(tv, bt709, progressive), 640x360 [SAR 1:1 DAR 16:9], q=2-31, 500 kb/s, 60 fps, 15360 tbn (default)\n Metadata:\n handler_name : VideoHandler\n vendor_id : [0][0][0][0]\n encoder : Lavc60.6.101 av1_qsv\n Stream #0:1(eng): Audio: opus, 48000 Hz, stereo, flt, 128 kb/s (default)\n Metadata:\n handler_name : SoundHandler\n vendor_id : [0][0][0][0]\n encoder : Lavc60.6.101 libopus\nframe= 0 fps=0.0 q=0.0 size=N/A time=00:00:00.20 bitrate=N/A speed=1.92x \nframe= 51 fps=0.0 q=-0.0 size=N/A time=00:00:01.24 bitrate=N/A speed=2.05x \nframe= 159 fps=142 q=-0.0 size=N/A time=00:00:03.02 bitrate=N/A speed= 2.7x \nframe= 229 fps=139 q=-0.0 size=N/A time=00:00:04.18 bitrate=N/A speed=2.54x \nframe= 311 fps=144 q=-0.0 size=N/A time=00:00:05.56 bitrate=N/A speed=2.57x \nframe= 399 fps=149 q=-0.0 size=N/A time=00:00:07.02 bitrate=N/A speed=2.63x \nframe= 487 fps=153 q=-0.0 size=N/A time=00:00:08.48 bitrate=N/A speed=2.67x \nframe= 562 fps=153 q=-0.0 size=N/A time=00:00:09.74 bitrate=N/A speed=2.65x \nframe= 644 fps=154 q=-0.0 size=N/A time=00:00:11.10 bitrate=N/A speed=2.65x \nframe= 728 fps=155 q=-0.0 size=N/A time=00:00:12.50 bitrate=N/A speed=2.66x \nframe= 816 fps=157 q=-0.0 size=N/A time=00:00:13.96 bitrate=N/A speed=2.68x \nframe= 899 fps=157 q=-0.0 size=N/A time=00:00:15.36 bitrate=N/A speed=2.68x \nframe= 973 fps=156 q=-0.0 size=N/A time=00:00:16.58 bitrate=N/A speed=2.66x \nframe= 1042 fps=155 q=-0.0 size=N/A time=00:00:17.76 bitrate=N/A speed=2.64x \nframe= 1129 fps=156 q=-0.0 size=N/A time=00:00:19.18 bitrate=N/A speed=2.65x \nframe= 1218 fps=157 q=-0.0 size=N/A time=00:00:20.68 bitrate=N/A speed=2.67x \nframe= 1310 fps=159 q=-0.0 size=N/A time=00:00:22.20 bitrate=N/A speed=2.69x \nframe= 1404 fps=160 q=-0.0 size=N/A time=00:00:23.78 bitrate=N/A speed=2.72x \nframe= 1498 fps=162 q=-0.0 size=N/A time=00:00:25.36 bitrate=N/A speed=2.74x \nframe= 1592 fps=163 q=-0.0 size=N/A time=00:00:26.90 bitrate=N/A speed=2.76x \nframe= 1684 fps=164 q=-0.0 size=N/A time=00:00:28.44 bitrate=N/A speed=2.77x \nframe= 1766 fps=164 q=-0.0 size=N/A time=00:00:29.80 bitrate=N/A speed=2.77x \nframe= 1945 fps=173 q=-0.0 size=N/A time=00:00:32.80 bitrate=N/A speed=2.91x \nframe= 2016 fps=171 q=-0.0 size=N/A time=00:00:33.96 bitrate=N/A speed=2.89x \nframe= 2112 fps=172 q=-0.0 size=N/A time=00:00:35.56 bitrate=N/A speed= 2.9x \nframe= 2212 fps=173 q=-0.0 size=N/A time=00:00:37.24 bitrate=N/A speed=2.91x \nframe= 2274 fps=171 q=-0.0 size=N/A time=00:00:38.26 bitrate=N/A speed=2.88x \nframe= 2356 fps=171 q=-0.0 size=N/A time=00:00:39.64 bitrate=N/A speed=2.87x \nframe= 2451 fps=171 q=-0.0 size=N/A time=00:00:41.24 bitrate=N/A speed=2.88x \nframe= 2579 fps=174 q=-0.0 size=N/A time=00:00:43.36 bitrate=N/A speed=2.93x \nframe= 2715 fps=177 q=-0.0 size=N/A time=00:00:45.64 bitrate=N/A speed=2.98x \nframe= 2876 fps=182 q=-0.0 size=N/A time=00:00:48.32 bitrate=N/A speed=3.06x \nframe= 3042 fps=186 q=-0.0 size=N/A time=00:00:51.08 bitrate=N/A speed=3.13x \nframe= 3122 fps=186 q=-0.0 size=N/A time=00:00:52.40 bitrate=N/A speed=3.12x \nframe= 3202 fps=185 q=-0.0 size=N/A time=00:00:53.76 bitrate=N/A speed= 3.1x \nframe= 3278 fps=184 q=-0.0 size=N/A time=00:00:55.00 bitrate=N/A speed=3.08x \nframe= 3373 fps=184 q=-0.0 size=N/A time=00:00:56.58 bitrate=N/A speed=3.08x \nframe= 3463 fps=183 q=-0.0 size=N/A time=00:00:58.08 bitrate=N/A speed=3.07x \nframe= 3580 fps=185 q=-0.0 size=N/A time=00:01:00.06 bitrate=N/A speed= 3.1x \nframe= 3733 fps=188 q=-0.0 size=N/A time=00:01:02.58 bitrate=N/A speed=3.15x \nframe= 3811 fps=187 q=-0.0 size=N/A time=00:01:03.90 bitrate=N/A speed=3.13x \nframe= 3888 fps=186 q=-0.0 size=N/A time=00:01:05.16 bitrate=N/A speed=3.12x \nframe= 3977 fps=186 q=-0.0 size=N/A time=00:01:06.66 bitrate=N/A speed=3.11x \nframe= 4063 fps=185 q=-0.0 size=N/A time=00:01:08.10 bitrate=N/A speed=3.11x \nframe= 4150 fps=185 q=-0.0 size=N/A time=00:01:09.56 bitrate=N/A speed= 3.1x \nframe= 4241 fps=185 q=-0.0 size=N/A time=00:01:11.06 bitrate=N/A speed= 3.1x \nframe= 4328 fps=185 q=-0.0 size=N/A time=00:01:12.52 bitrate=N/A speed= 3.1x \nframe= 4416 fps=185 q=-0.0 size=N/A time=00:01:13.98 bitrate=N/A speed=3.09x \nframe= 4489 fps=184 q=-0.0 size=N/A time=00:01:15.18 bitrate=N/A speed=3.08x \nframe= 4599 fps=184 q=-0.0 size=N/A time=00:01:17.02 bitrate=N/A speed=3.09x \nframe= 4703 fps=185 q=-0.0 size=N/A time=00:01:18.76 bitrate=N/A speed=3.09x \nframe= 4800 fps=185 q=-0.0 size=N/A time=00:01:20.36 bitrate=N/A speed= 3.1x \nframe= 4904 fps=185 q=-0.0 size=N/A time=00:01:22.12 bitrate=N/A speed= 3.1x \nframe= 4984 fps=185 q=-0.0 size=N/A time=00:01:23.44 bitrate=N/A speed=3.09x \nframe= 5076 fps=185 q=-0.0 size=N/A time=00:01:24.98 bitrate=N/A speed=3.09x \nframe= 5153 fps=184 q=-0.0 size=N/A time=00:01:26.26 bitrate=N/A speed=3.08x \nframe= 5254 fps=184 q=-0.0 size=N/A time=00:01:27.96 bitrate=N/A speed=3.09x \nframe= 5364 fps=185 q=-0.0 size=N/A time=00:01:29.78 bitrate=N/A speed= 3.1x \nframe= 5469 fps=185 q=-0.0 size=N/A time=00:01:31.54 bitrate=N/A speed= 3.1x \nframe= 5573 fps=186 q=-0.0 size=N/A time=00:01:33.26 bitrate=N/A speed=3.11x \nframe= 5658 fps=185 q=-0.0 size=N/A time=00:01:34.66 bitrate=N/A speed= 3.1x \nframe= 5730 fps=185 q=-0.0 size=N/A time=00:01:35.86 bitrate=N/A speed=3.09x \nframe= 5826 fps=185 q=-0.0 size=N/A time=00:01:37.48 bitrate=N/A speed=3.09x \nframe= 5920 fps=185 q=-0.0 size=N/A time=00:01:39.04 bitrate=N/A speed=3.09x \nframe= 5992 fps=184 q=-0.0 size=N/A time=00:01:40.26 bitrate=N/A speed=3.08x \nframe= 6115 fps=185 q=-0.0 size=N/A time=00:01:42.28 bitrate=N/A speed= 3.1x \nframe= 6195 fps=185 q=-0.0 size=N/A time=00:01:43.64 bitrate=N/A speed=3.09x \nframe= 6288 fps=185 q=-0.0 size=N/A time=00:01:45.16 bitrate=N/A speed=3.09x \nframe= 6405 fps=185 q=-0.0 size=N/A time=00:01:47.12 bitrate=N/A speed= 3.1x \nframe= 6524 fps=186 q=-0.0 size=N/A time=00:01:49.10 bitrate=N/A speed=3.11x \nframe= 6616 fps=186 q=-0.0 size=N/A time=00:01:50.66 bitrate=N/A speed=3.11x \nframe= 6710 fps=186 q=-0.0 size=N/A time=00:01:52.20 bitrate=N/A speed=3.11x \nframe= 6796 fps=186 q=-0.0 size=N/A time=00:01:53.64 bitrate=N/A speed= 3.1x \nframe= 6892 fps=186 q=-0.0 size=N/A time=00:01:55.26 bitrate=N/A speed=3.11x \nframe= 7018 fps=187 q=-0.0 size=N/A time=00:01:57.36 bitrate=N/A speed=3.12x \nframe= 7107 fps=186 q=-0.0 size=N/A time=00:01:58.82 bitrate=N/A speed=3.12x \nframe= 7178 fps=186 q=-0.0 size=N/A time=00:02:00.00 bitrate=N/A speed=3.11x \nframe= 7297 fps=186 q=-0.0 size=N/A time=00:02:02.00 bitrate=N/A speed=3.12x \nframe= 7350 fps=185 q=-0.0 size=N/A time=00:02:02.88 bitrate=N/A speed= 3.1x \nframe= 7474 fps=186 q=-0.0 size=N/A time=00:02:04.96 bitrate=N/A speed=3.11x \nframe= 7610 fps=187 q=-0.0 size=N/A time=00:02:07.20 bitrate=N/A speed=3.13x \nframe= 7774 fps=189 q=-0.0 size=N/A time=00:02:09.96 bitrate=N/A speed=3.16x \n[av1_qsv @ 0x560b846b7440] extract_extradata filter failed to receive output packet\n[vost#0:0/av1_qsv @ 0x560b846b7140] Error submitting video frame to the encoder\n[libopus @ 0x560b846f5140] 1 frames left in the queue on closing\nConversion failed!\n",
"shellCommand": "ffmpeg -n 15 /usr/bin/ffmpeg -i /var/www/peertube/storage/videos/private/b45a8c79-9e70-470c-8ed8-084ab4229b38-1440.mp4 -y -acodec libopus -vcodec av1_qsv -threads 24 -f mp4 -movflags faststart -max_muxing_queue_size 1024 -map_metadata -1 -pix_fmt yuv420p10le -b:a 128k -strict -2 -vbr on -vf scale=w=-2:h=360 -b:v 500k -minrate 100k -maxrate 500k -b_strategy 1 -look_ahead 1 -look_ahead_depth 100 -g 300 -preset 1 -profile 1 -tile_cols 2 -tile_rows 2 -hls_time 4 -hls_list_size 0 -hls_playlist_type vod -hls_segment_filename /var/www/peertube/storage/tmp/hls/3705d0b9-62e5-486b-b9c1-63720d3702b5-360-fragmented.mp4 -hls_segment_type fmp4 -f hls -hls_flags single_file /var/www/peertube/storage/tmp/hls/3705d0b9-62e5-486b-b9c1-63720d3702b5-360.m3u8",
"tags": [
"ffmpeg"
]
}
error[3/13/2023, 5:18:37 PM] Cannot execute job a2a329e4-aa10-4fcd-a656-a6fbfd6513ec in queue video-transcoding.
{
"payload": {
"type": "new-resolution-to-hls",
"videoUUID": "23f0fd78-f20b-4d60-980c-6217619c0500",
"resolution": 360,
"hasAudio": true,
"copyCodecs": false,
"isNewVideo": false,
"autoDeleteWebTorrentIfNeeded": false,
"isMaxQuality": false
},
"err": {
"stack": "Error: ffmpeg exited with code 1: Conversion failed!\n\n at ChildProcess.<anonymous> (/var/www/peertube/versions/peertube-develop/node_modules/fluent-ffmpeg/lib/processor.js:182:22)\n at ChildProcess.emit (node:events:513:28)\n at ChildProcess._handle.onexit (node:internal/child_process:291:12)",
"message": "ffmpeg exited with code 1: Conversion failed!\n"
}
}
I tried with a new video. It failed the first time but was able to work when re-running the hls job this time.
I checked for errors when it was stuck in the "this video is already being transcoded" a few hours ago and didn't see any though other then the above ffmpeg errors.
Just as a FYI this is happening because the AV1_QSV encoder on the arc cards with the intel driver bugs out on low resolution (360p) videos. 480p seems to be better off. Peertube still should be able to re-transcode videos after the first failed attempt though.
Blocked by https://github.com/taskforcesh/bullmq/issues/800 (the last job, that is a parent job is not executed when children jobs fail)
Workaround in https://github.com/Chocobozzz/PeerTube/commit/89aa3331106874266f6feeee7bff852da2c1727e
You will be able to force transcoding if you consider the video is in a broken state (in peertube v6)