mlt
mlt copied to clipboard
mltframework
frei0r plugins prevent mlt from multiprocessing.
I'm currently working on problem with openSUSE mlt where exporting video from kdenlive and shotcut doesn't use the same processing power as ffmpeg for the same file. I can't duplicate this but the two reporters say that this problem only occurs when frei0r plugins are present. Now the one reporter has removed all frei0r plugins listed as not thread safe in src/modules/frei0r/not_thread_safe.txt and melt works as expected using maximum cpu resources. What I can't understand is the reporter downloaded the stand alone shotcut and kdenlive packages and they work as expected. shotcut-linux-x86_64-171203.tar.bz2 has a full compliment of frei0r plugins and yet it works. I tried to get the reporter to build shotcut sources using build-shotcut.sh but this fails to build movit properly and fails mlt. What is wrong with my packaging?
This bug report is difficult to follow. For starters, Shotcut (and perhaps Kdenlive appimage but not sure) are using MLT and frei0r git master when released each month. There has been a lot of changes in the past year in both projects to facilitate more concurrency, and you need both to achieve it.
For the FFmpeg integration, there was an ABI change in FFmpeg 3.2 with respect to multi-threading that required a change in MLT: https://github.com/mltframework/mlt/commit/0f8e807aa911c53e221e940556370ee983f49c64#diff-4f241487ea1a24089964f1fad57aaa89
Basically, the default in libavcodec changed from auto to 1 (but not the ffmpeg cmd line util), and MLT was not supplying a value if its corresponding property was not set by the user. However, none of that explains the frei0r interaction. If I understand correctly, the problem is reproduced by playing a video clip with melt with frei0r installed and observing CPU usage and thread activity. And then comparing without frei0r installed.. All that not_thread_safe.txt in MLT does is introduce mutex locks to inhibit concurrency. The MLT-frei0r integration does not create threads or use sched_getaffinity(). Also, your note "I still can't understand why the frei0r-plugins are used in a simple export unless they are part of the production." is correct. They are not used.
I wonder if the problem is tied to frei0r plugins built against OpenCV. I know OpenCV does some things with threading, OpenMP, and who knows what all else. When MLT initializes, it checks to see what all frei0r plugins exists, and simply loading an OpenCV-based plugin may initialize something in OpenCV without that plugin being used in a composition. Shotcut's builds do not build against OpenCV because that is dependency rich, and for a while some OpenCV packages were still built against Qt4 when Shotcut had converted to Qt5.
If you look here, you can see which frei0r plugins use OpenCV: https://github.com/dyne/frei0r/blob/master/src/Makefile.am#L154 Simply removing those installed plugin files is a simple test that does not require rebuilding.
Another similar thing could be happening with gavl, which Shotcut also does not include. Scroll up in that same Makefile.am to see the 3 gavl plugins that can be removed for a quick test.
files_not_threadsafe.txt I've worked around this bug by splitting the frei0r plugins and packaging the ones marked as not thread safe (see above files list) so one or more of the plugins in the list cause the problem. I see that the two OpenCV plugins are in the list. I can add these back and the reporter can test. I see the gavl plugins are also in the list.
I will take a look soon at what change needs to be made in OpenCV frei0r plugins. In the meantime, you should think about a packaging workaround considering possibly not including them. There are 2 face recognition-related plugins that most users of video editors are not using.
I've packaged them as frei0r-plugins-opencv subpackage with a warning in the description. Shotcut only requires the main frei0r-plugins package. Kdenlive only recomends frei0r-plugins and the only other app I can see that uses frei0r is webcamoid, I'll warn the maintainer.
@ddennedy The openSUSE bug appeared again and was reopened. After detective work it was found that the issue was caused by libmltopencv.so, which isn't present in the app image. You should remove this from mlt completely.
@plater I disagree that it should be removed. Think about people already using this in scenarios where they care more about obtaining a result than how quickly it runs. Meanwhile, there is still a root cause that we may find and resolve. You can disable the module in your build using configure's --disable-opencv. It can be argued, however, that we should change configure to require it be explicitly enabled. I will reopen this issue to track that as well as the root cause threading issue.
@j-b-m Do you agree to make the opencv module required to be explicitly enabled until the root issue is identified and resolved?
Yes sure no problem to disable the build by default. I plan to work again on the opencv module in the next months to add some more filters, but not sure I can fix this issue, and for sure not in the short term.
The mlt build against an opencv lower than 3.4.1, in the case of openSUSE the previous version was 3.3.1 failed to build the opencv module due to a missing tracker, that's why this bug has only just appeared.
@ddennedy https://bugzilla.opensuse.org/show_bug.cgi?id=1184430 has been opened stating that mlt built without opencv disables the motion tracker effect in kdenlive. Is it safe to enable opencv in the build again?
I do not know; I doubt it. I have not reproduced it, and it has been a lone time since I tried. I generally avoid OpenCV now due to its version compatibility problems. Also, no root cause was identified. It is my understanding the Kdenlive AppImage includes OpenCV, but I do not do much with kdenlive.