Kubuntu: Can't save media anymore

[umbertones]

I run VidCutter 6.0.0 on a fully patched Kubuntu 18.04 LTS with Kernel 5.0.0.37

VidCutter stopped to save the media. It runs fine (load media, create clips) until I click "Save media".

The progress bar stays at 0%, the Icon in the Taskbar is filled about 66% (looks like two third of the work is done) but this goes on forever. Tried saving on local and network drives, no difference.

I clicked on verboose logging in the settings, but I can't find any log. Any hints?

When I click on "Close" in the upper right corner it says that "Media is currently processed"...unfortunately it is not. The file to save does not appear on disk not even with 0 bytes length.

Any idea? I had it working but maybe there was an update that killed any function?

ffmpeg is installed and works when called solo.

Many thanks in advance.

[umbertones]

Finally found the vidcutter.log:

2019-12-02 19:39 - root - CRITICAL - File "/usr/lib/python3/dist-packages/vidcutter/videocutter.py", line 1363, in saveMedia self.smartcutter(file, source_file, source_ext) File "/usr/lib/python3/dist-packages/vidcutter/videocutter.py", line 1413, in smartcutter allstreams=True) File "/usr/lib/python3/dist-packages/vidcutter/libs/videoservice.py", line 331, in smartcut bisections = self.getGOPbisections(source, start, end) File "/usr/lib/python3/dist-packages/vidcutter/libs/videoservice.py", line 589, in getGOPbisections start_pos = bisect_left(keyframes, start)

2019-12-02 19:39 - root - CRITICAL - <class 'TypeError'>: '<' not supported between instances of 'str' and 'float'

Wrong Python version? Works with Python2 only?

[ratcashdev]

very similar for me. Python 3.8 on Manjaro stack trace:

2019-12-25 15:12 - vidcutter.libs.mpvwidget - INFO - [libmpv_render/cuda-nvdec] cu->cuGLGetDevices(&device_count, &display_dev, 1, CU_GL_DEVICE_LIST_ALL) failed -> CUDA_ERROR_UNKNOWN: unknown error
2019-12-25 15:13 - vidcutter.videocutter - INFO - source file is MPEG based so join via MPEG-TS
2019-12-25 15:13 - vidcutter.videocutter - INFO - MPEG-TS based join failed, will retry using standard concat
2019-12-25 15:13 - root - CRITICAL -   File "/usr/lib/python3.8/site-packages/vidcutter/videocutter.py", line 1390, in saveMedia
    self.joinMedia(filelist)
  File "/usr/lib/python3.8/site-packages/vidcutter/videocutter.py", line 1439, in joinMedia
    rc = self.videoService.join(filelist, self.finalFilename, True, chapters)
  File "/usr/lib/python3.8/site-packages/vidcutter/libs/videoservice.py", line 465, in join
    ffmetadata = self.getChapterFile(inputs, chapters)
  File "/usr/lib/python3.8/site-packages/vidcutter/libs/videoservice.py", line 480, in getChapterFile
    duration = self.duration(scene)
  File "/usr/lib/python3.8/site-packages/vidcutter/libs/videoservice.py", line 256, in duration
    matches = re.search(r'Duration:\s(?P<hrs>\d+?):(?P<mins>\d+?):(?P<secs>\d+\.\d+?),',

[pikim]

@ratcashdev Your error is not the same as @umbertones.

@umbertones I had the same error message in the log. The problem in my case was that ffprobe -v error -show_packets -select_streams v -show_entries packet=pts_time,flags -of csv /path/to/video.avi outputs only lines in the form of packet,N/A,K_ or packet,N/A,__. You can also try this from the console. Vidcutter uses this command to get the time stamps of the key frames. When pts_time is N/A all time stamps remain at 0 which can't be handled. In my case the problem appears only when being called on h264 files.

I modified the function getKeyframes in videoservice.py so that it also retrieves the dts_time from ffprobe. Now, if pts_time is N/A it takes dts_time instead. I don't know whether this is a good and viable solution, but it solved my issue and I didn't notice any problems (yet).

I also saw that the last time stamp of the keyframes is a str, but all other time stamps are float. I don't know whether this was by intention, but I also fixed this. Now all time stamps are float or str, but not mixed any more.

My modified videoservice.py: videoservice.txt

[ratcashdev]

This dts_time and pts_time stuff is interesting. I did a couple of fixes yesterday, building on the work fork of https://github.com/pmhahn/vidcutter (fixed project load). In my own fork I have also fixed the project save (media was not included in the project file - at least not on Manjaro, Gnome 3.34.x) and also this ffprobe parsing (https://github.com/ratcashdev/vidcutter/commit/17366cd2ef5eed23ff1d76727d7676258758ccdf). It's interesting, that for my file (MPEG2-TS) ffprobe outputs something that vidcutter seemingly was not designed to parse (flags=K_ vs. .*,K). Check out my master branch.

[pikim]

@ratcashdev Your version doesn't work on Arch Linux. I had to change the args in getKeyFrames into 'packet=pts_time,dts_time,flags', '-of', 'csv', and change the ffprobe result parsing into:

        for line in result.split('\n'):
            if line.split(',')[1] != 'N/A':
                timecode = line.split(',')[1]
            else:
                timecode = line.split(',')[2]
            if re.search(',K', line):
                if formatted_time:
                    keyframe_times.append(timecode[:-3])
                else:
                    keyframe_times.append(float(timecode))

The args -of csv where thrown away in https://github.com/ratcashdev/vidcutter/commit/e7260e1f05b1c69e812194106aeeb549c088480e#diff-d0da34f4c32327cd90eb985f6e593f8c. I don't know if that was accidentally or intentionally. In my opinion these args make sense as they shorten the ffprobe output without losing any information.

With these changes it still crashes, but later in the process with the following message:

2019-12-30 12:36 - root - CRITICAL -   File "/usr/lib/python3.8/site-packages/vidcutter/videocutter.py", line 1360, in saveMedia
    self.smartcutter(file, source_file, source_ext)
  File "/usr/lib/python3.8/site-packages/vidcutter/videocutter.py", line 1405, in smartcutter
    self.videoService.smartcut(index=index,
  File "/usr/lib/python3.8/site-packages/vidcutter/libs/videoservice.py", line 382, in smartcut
    startproc.setArguments(

2019-12-30 12:36 - root - CRITICAL - <class 'TypeError'>: index 9 has type 'list' but 'str' is expected

[ratcashdev]

@pikim Thanks for pointing that out. I found the culprit commit (part of the fork I used as a baseline): https://github.com/ratcashdev/vidcutter/commit/e7260e1f05b1c69e812194106aeeb549c088480e Will have to fix that and then revert some other subsequently useless fixes.

[ratcashdev]

including the csv output format argument changed a lot, certainly. However, still not ideal, as the output is on every second line only, so that will have to be handled as well.

packet,95942.310667,__side_data,MPEGTS Stream ID
{empty line}
packet,95942.470667,K_side_data,MPEGTS Stream ID
{empty line}
packet,95942.390667,__side_data,MPEGTS Stream ID

PS: Would that condition for 'N/A' get actually triggered, like ever?

[pikim]

including the csv output format argument changed a lot, certainly. However, still not ideal, as the output is only on every second line only, so that will have to be handled as well.

This should already work and is the expected behaviour. Not every frame is a key frame and therefore the corresponding flag is not set for every frame.

PS: Would that condition for 'N/A' get actually get triggered, like ever?

This already works as expected, at least in my version. The original version has the problem that the timestamp remains 0 when pts_time is N/A.

I only get N/A when I probe a h264 movie. For a mpeg4 movie I get correct dts_times, see:

$ ffprobe -v error -show_packets -select_streams v -show_entries packet=pts_time,dts_time,flags -of csv mpeg4.avi | head -n3
packet,0.000000,0.000000,K_
packet,0.080000,0.080000,__
packet,0.120000,0.120000,__

$ ffprobe -v error -show_packets -select_streams v -show_entries packet=pts_time,dts_time,flags -of csv h264.avi | head -n3
packet,N/A,0.000000,K_
packet,N/A,0.040000,__
packet,N/A,0.080000,__

Note that pts_time and dts_time are equal when both exist and pts_time is N/A for h264 content.

[umbertones]

@Pikim:

My modified videoservice.py: videoservice.txt

Thanks! I'll check that on my system when I'll get back to it next year. :-)

[ratcashdev]

This should already work and is the expected behaviour. Not every frame is a key frame and therefore the corresponding flag is not set for every frame.

I have updated the snippet in my previous post. See the {empty line}s in there. They were causing the

2019-12-30 14:34 - root - CRITICAL -   File "/vidcutter/vidcutter/videocutter.py", line 1360, in saveMedia
    self.smartcutter(file, source_file, source_ext)
  File "/vidcutter/vidcutter/videocutter.py", line 1405, in smartcutter
    self.videoService.smartcut(index=index,
  File "/vidcutter/vidcutter/libs/videoservice.py", line 371, in smartcut
    bisections = self.getGOPbisections(source, start, end)
  File "/vidcutter/vidcutter/libs/videoservice.py", line 678, in getGOPbisections
    keyframes = self.getKeyframes(source)
  File "/vidcutter/vidcutter/libs/videoservice.py", line 652, in getKeyframes
    if parts[1] != 'N/A':

2019-12-30 14:34 - root - CRITICAL - <class 'IndexError'>: list index out of range

exception. To fix, I had to add something like this:

 for line in result.split('\n'):
            parts=line.split(',')
            if len(parts) > 1:
...

[pikim]

I don't know why your ffprobe output looks different than mine. Do you call ffprobe manually from the console? Could you copy&paste my call from above and only exchange the movie file?

[ratcashdev]

@pikim

Could you copy&paste my call from above and only exchange the movie file?

That's exactly what I did. For an MPEG4 stream it produced expected output on every single line, just like yours. For my MPEG2TS stream it only produced output on every second line though. So something with the video itself.

$ ffprobe -v info -show_entries packet -of csv ~/Videos/works/vid.ts |head -n3
ffprobe version n4.2.1 Copyright (c) 2007-2019 the FFmpeg developers
  built with gcc 9.2.0 (GCC)
  configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libdrm --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libjack --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-nvdec --enable-nvenc --enable-omx --enable-shared --enable-version3
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
Input #0, mpegts, from '~/Videos/works/vid.ts':
  Duration: 02:35:00.62, start: 1.400000, bitrate: 4056 kb/s
  Program 1 
    Metadata:
      service_name    : Service01
      service_provider: FFmpeg
    Stream #0:0[0x100]: Video: mpeg2video (Main) ([2][0][0][0] / 0x0002), yuv420p(tv, top first), 720x576 [SAR 64:45 DAR 16:9], 3590 kb/s, 25 fps, 25 tbr, 90k tbn, 50 tbc
    Stream #0:1[0x101](slo): Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, fltp, 192 kb/s
packet,audio,1,126000,1.400000,126000,1.400000,2160,0.024000,N/A,N/A,576,564,K_side_data,MPEGTS Stream ID

packet,audio,1,128160,1.424000,128160,1.424000,2160,0.024000,N/A,N/A,576,N/A,K_

[pikim]

Strange. There seems to be a problem with the video stream, as the two lines that appear come from the audio stream. Did you also try another video (from another source)?

[ratcashdev]

@pikim Initially the file was a DVB-T recorded stream with 2 audio streams, 1 video stream and 1 SUBTITLE stream. This I later losslessly 'copied' into a new file with only 2 stream (1 video, 1 audio), as the SUBTITLE stream was making problems for ffmpeg during the smart encode process (error: no codec).

So I checked the original file to see if the csv output also generates empty lines. Check out this output - some lines are empty, other are not:

$ ffprobe -v warning -show_entries packet=pts_time,dts_time,flags,codec_type,stream_index -of csv ~/Videos/works/VIDEO-H5-303.ts |tail -n20
[mpegts @ 0x560de9457e40] PES packet size mismatch
    Last message repeated 2 times
Unsupported codec with id 94215 for input stream 2
[mpegts @ 0x560de9457e40] PES packet size mismatch
    Last message repeated 176 times
packet,video,0,95942.790667,95942.790667,__side_data,

packet,video,0,95942.950667,95942.830667,__side_data,

packet,subtitle,2,95942.510667,95942.510667,K_side_data,

packet,video,0,95942.870667,95942.870667,__side_data,

packet,audio,1,95942.464222,95942.464222,K_side_data,

packet,audio,1,95942.488222,95942.488222,K_
packet,subtitle,2,95942.550667,95942.550667,K_side_data,

packet,audio,3,95942.464222,95942.464222,K_side_data,

packet,audio,3,95942.488222,95942.488222,K_
packet,video,0,95942.910667,95942.910667,__
packet,audio,1,95942.512222,95942.512222,K_
packet,audio,3,95942.512222,95942.512222,K_

Worse, yet, it seems that different streams have keyframes at different positions.

Wouldn't this make the smart-cut feature practically impossible? At a minimum, we'd need to use a more exact output of ffprobe to identify VIDEO stream keyframes (not just look for the K flag of any arbitrary sub-stream). Furthermore, we'd need to re-encode various substreams at different cut points/timecodes and then to MUX them back keeping sync between the streams...

The empty lines seem to come from empty SIDE_DATA segments, based on this output here (last 6 entries copied, not using the -of csv argument) :

$ ffprobe -v warning -show_entries packet=pts_time,dts_time,flags,codec_type,stream_index ~/Videos/works/VIDEO-H5-303.ts |tail -n200
...
[PACKET]
codec_type=audio
stream_index=3
pts_time=95942.464222
dts_time=95942.464222
flags=K_
[SIDE_DATA]
[/SIDE_DATA]
[/PACKET]
[PACKET]
codec_type=audio
stream_index=3
pts_time=95942.488222
dts_time=95942.488222
flags=K_
[/PACKET]
[PACKET]
codec_type=video
stream_index=0
pts_time=95942.910667
dts_time=95942.910667
flags=__
[/PACKET]
[PACKET]
codec_type=audio
stream_index=1
pts_time=95942.512222
dts_time=95942.512222
flags=K_
[/PACKET]
[PACKET]
codec_type=audio
stream_index=3
pts_time=95942.512222
dts_time=95942.512222
flags=K_
[/PACKET]

So whenever the csv contains the side_data flag, an empty line follows in the output.

[pikim]

You could try to change to loop into

        for line in result.split('\n'):
            if line != '':
                if line.split(',')[1] != 'N/A':
                    timecode = line.split(',')[1]
                else:
                    timecode = line.split(',')[2]
                if re.search(',K', line):
                    if formatted_time:
                        keyframe_times.append(timecode[:-3])
                    else:
                        keyframe_times.append(float(timecode))

This should ignore empty lines. Or the following alternative if you like it more:

        for line in result.split('\n'):
            if line == '':
                continue
            if line.split(',')[1] != 'N/A':
                timecode = line.split(',')[1]
            else:
                timecode = line.split(',')[2]
            if re.search(',K', line):
                if formatted_time:
                    keyframe_times.append(timecode[:-3])
                else:
                    keyframe_times.append(float(timecode))

EDIT: just saw what you committed: Does it work that way? I don't know whether it's any problem or not when different streams have key frames at different positions.

I see the same when I ffprobe a ts stream recorded via DVB-S. Up to now I always cut those with DVBcut.

[ratcashdev]

@pikim Yes, my fix for the empty lines work just OK. Yours will catch the empty lines sooner though, but then again, does the split multiple times. Sadly, DVBCut is not an option for me any more on my distribution. Hence my interest in this SW. Still trying to figure out if the audio will stay in sync in spite of the (sometimes +- 500ms) keyframe differences. Just have to get past the join, which does not work, (yet|too). Oh well.

EDIT: the join does not work, if the MPEG2-TS source file has a .ts extension, since the output file in the merge process also produces a .ts and thus tries to overwrite an existing file, in-place. Fixed in: https://github.com/ratcashdev/vidcutter/commit/4668600c3740fc5c65f21a1ad72d22dd36e33765

[ratcashdev]

@pikim Got past the JOIN bugs. The result:

1. Extremely low quality and very lossy video. VIDEO is the same res, but much less bitrate, interlaced is converted to progressive, rest is the same:

Stream #0:0[0x836]: Video: mpeg2video (Main) ([2][0][0][0] / 0x0002), yuv420p(tv, top first), 720x576 [SAR 64:45 DAR 16:9], 3590 kb/s, 25 fps, 25 tbr, 90k tbn, 50 tbc

gets converted/cut to

Stream #0:0[0x100]: Video: mpeg2video (Main) ([2][0][0][0] / 0x0002), yuv420p(tv, progressive), 720x576 [SAR 64:45 DAR 16:9], 25 fps, 25 tbr, 90k tbn, 50 tbc

This apparently applies not just around the cut points, but overall to the whole file. Even the MIDDLE segment is re-encoded. 2. Audio at cut points is lost (exactly those couple hundred ms because of the keyframe differences). But overall, the audio stays in sync.

[pikim]

I noticed that it doesn't cut h264 frame accurate at all. I didn't test other formats, yet.

[ratcashdev]

@pikim do you plan to try to fix it? Seems like the number of bugs and media incompatibilities are far too numerous given my own limited time and experience with ffmpeg/media/python. But I still need a smart-cut solution for mpeg2-ts :)

[ratcashdev]

Just wondering. Wouldn't it be easier to simply first FORCE keyframes at the desidered cut points and then simply cut/merge without ANY re-encode? Idea comes from https://video.stackexchange.com/questions/27513/encoding-insert-keyframes-at-certain-positions

[pikim]

Sadly, DVBCut is not an option for me any more on my distribution.

Could you tell me why?

do you plan to try to fix it?

To be honest I thought about extending DVBcut. But then it still could only be used for DVB files. VidCutter is more versatile, so maybe I'll try to fix it there.

[ratcashdev]

Could you tell me why?

I am on Manjaro. Building https://aur.archlinux.org/packages/dvbcut fails. Same with the QT4 port of it.

VidCutter is more versatile, so maybe I'll try to fix it there.

I had some success in fixing the terrible quality (again look in my commit-log), but yet to fix the frame-accuracy.

[pikim]

I am on Manjaro. Building https://aur.archlinux.org/packages/dvbcut fails. Same with the QT4 port of it.

Just for completeness: I'm using Arch and was able to compile and use https://github.com/bernhardu/dvbcut-deb

[pmhahn]

So I checked the original file to see if the csv output also generates empty lines. Check out this output - some lines are empty, other are not:

$ ffprobe -v warning -show_entries packet=pts_time,dts_time,flags,codec_type,stream_index -of csv ~/Videos/works/VIDEO-H5-303.ts |tail -n20
packet,video,0,95942.790667,95942.790667,__side_data,

...

The empty lines seem to come from empty SIDE_DATA segments, based on this output here (last 6 entries copied, not using the -of csv argument) :

This is FFMPEG issue 7153

[pmhahn]

@ratcashdev Your version doesn't work on Arch Linux. I had to change the args in getKeyFrames into 'packet=pts_time,dts_time,flags', '-of', 'csv', and change the ffprobe result parsing into: ... The args -of csv where thrown away in ratcashdev@e7260e1#diff-d0da34f4c32327cd90eb985f6e593f8c. I don't know if that was accidentally or intentionally.

By accident - fixed in my internal version already.

[ratcashdev]

@pmhahn you may be interested in reviewing my own fixes (master branch) for the same (and many other) bugs. To sum up:

• no more exceptions during cut or smartcut
• consistent use of timecodes (as float) throughout smartcut
• lossless copy for cut or the middle segment of smartcut (before: lossy re-encode)
• audio in sync at cue-points (before: out of sync)
• start-segment keyframe misaligment fixed
• more precise cut thanks to output seeking