mlt
mlt copied to clipboard
mltframework
Color shift after first frame of output when using two composite transitions
Hi there! I'm diving into MLT to composite two videos next to each other at specific sizes and orientations. They may animate between different orientations, but for this issue, I kept it to just one to demonstrate the problem. The problem is that I often see a noticeable color shift right after the video starts. If there's no transition, I don't see a color shift in the output. (The color is slightly different from the original, but maybe close enough to get by. But what I mean is, it doesn't noticeably change from the first frame to the second.)
I can get rid of the shift if I switch the profile colorspace from 709 to 601, though I'm not sure if that means all but the first frame is output in 601 or if the smaller colorspace is just masking the color shift. The output does seem slightly darker than the original, at least in this example.
I was initially having this issue using the mlt library via swig, but I was also able to reproduce it with melt. The input XML in my example was created using the XML consumer from a ruby script, hence quite a bit of unnecessary attributes. But I included it all for completeness.
Ideally my output would be in the 709 colorspace and be as true to the original colors as possible while also having no discernible color shift in the output. Am I doing that correctly or is there a better way? What am I missing?
The input file: https://embed-ssl.wistia.com/deliveries/d2e5245e04b6bcdce7981a5bb550b602/pets000.mov?disposition=attachment
The output file (you can see the color shift here): https://embed-ssl.wistia.com/deliveries/0fa094f2a99c63b1253f01d7675082ef/output.mov?disposition=attachment
The command I'm using:
melt \
xml:input.xml \
-consumer \
avformat:output.mov \
vcodec=libx264 \
f=mov \
crf=23 \
x264opts=keyint=89:min-keyint=89:scenecut=-1 \
movflags=faststart \
strict=strict \
threads=0 \
width=1280 \
height=720
input.xml:
<?xml version="1.0"?>
<mlt LC_NUMERIC="en_US.UTF-8" version="6.23.0">
<profile description="DV/DVD PAL" width="1280" height="720" progressive="1" sample_aspect_num="1" sample_aspect_den="1" display_aspect_num="1" display_aspect_den="1" frame_rate_num="30" frame_rate_den="1" colorspace="709"/>
<producer id="producer0" in="0" out="114">
<property name="length">15000</property>
<property name="eof">continue</property>
<property name="resource">0x000000ff</property>
<property name="aspect_ratio">1</property>
<property name="mlt_service">colour</property>
<property name="colour">0x000000FF</property>
</producer>
<producer id="producer1" in="0" out="114">
<property name="length">358</property>
<property name="eof">continue</property>
<property name="resource">pets000.mov</property>
<property name="meta.media.nb_streams">2</property>
<property name="meta.media.0.stream.type">video</property>
<property name="meta.media.0.stream.frame_rate">23.976</property>
<property name="meta.media.0.stream.sample_aspect_ratio">0</property>
<property name="meta.media.0.codec.width">1280</property>
<property name="meta.media.0.codec.height">720</property>
<property name="meta.media.0.codec.rotate">0</property>
<property name="meta.media.0.codec.frame_rate">24000</property>
<property name="meta.media.0.codec.pix_fmt">yuv420p</property>
<property name="meta.media.0.codec.sample_aspect_ratio">0</property>
<property name="meta.media.0.codec.colorspace">709</property>
<property name="meta.media.0.codec.color_trc">1</property>
<property name="meta.media.0.codec.name">h264</property>
<property name="meta.media.0.codec.long_name">H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10</property>
<property name="meta.media.0.codec.bit_rate">5948147</property>
<property name="meta.attr.0.stream.language.markup">eng</property>
<property name="meta.attr.0.stream.handler_name.markup">Apple Video Media Handler</property>
<property name="meta.media.1.stream.type">audio</property>
<property name="meta.media.1.codec.sample_fmt">s16</property>
<property name="meta.media.1.codec.sample_rate">44100</property>
<property name="meta.media.1.codec.channels">2</property>
<property name="meta.media.1.codec.name">pcm_s16le</property>
<property name="meta.media.1.codec.long_name">PCM signed 16-bit little-endian</property>
<property name="meta.media.1.codec.bit_rate">1411200</property>
<property name="meta.attr.1.stream.language.markup">eng</property>
<property name="meta.attr.1.stream.handler_name.markup">Apple Sound Media Handler</property>
<property name="meta.attr.major_brand.markup">qt </property>
<property name="meta.attr.minor_version.markup">512</property>
<property name="meta.attr.compatible_brands.markup">qt </property>
<property name="meta.attr.encoder.markup">Lavf58.20.100</property>
<property name="seekable">1</property>
<property name="meta.media.sample_aspect_num">1</property>
<property name="meta.media.sample_aspect_den">1</property>
<property name="aspect_ratio">1</property>
<property name="audio_index">1</property>
<property name="video_index">0</property>
<property name="mute_on_pause">1</property>
<property name="mlt_service">avformat</property>
</producer>
<producer id="producer2" in="0" out="114">
<property name="length">358</property>
<property name="eof">continue</property>
<property name="resource">pets000.mov</property>
<property name="meta.media.nb_streams">2</property>
<property name="meta.media.0.stream.type">video</property>
<property name="meta.media.0.stream.frame_rate">23.976</property>
<property name="meta.media.0.stream.sample_aspect_ratio">0</property>
<property name="meta.media.0.codec.width">1280</property>
<property name="meta.media.0.codec.height">720</property>
<property name="meta.media.0.codec.rotate">0</property>
<property name="meta.media.0.codec.frame_rate">24000</property>
<property name="meta.media.0.codec.pix_fmt">yuv420p</property>
<property name="meta.media.0.codec.sample_aspect_ratio">0</property>
<property name="meta.media.0.codec.colorspace">709</property>
<property name="meta.media.0.codec.color_trc">1</property>
<property name="meta.media.0.codec.name">h264</property>
<property name="meta.media.0.codec.long_name">H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10</property>
<property name="meta.media.0.codec.bit_rate">5948147</property>
<property name="meta.attr.0.stream.language.markup">eng</property>
<property name="meta.attr.0.stream.handler_name.markup">Apple Video Media Handler</property>
<property name="meta.media.1.stream.type">audio</property>
<property name="meta.media.1.codec.sample_fmt">s16</property>
<property name="meta.media.1.codec.sample_rate">44100</property>
<property name="meta.media.1.codec.channels">2</property>
<property name="meta.media.1.codec.name">pcm_s16le</property>
<property name="meta.media.1.codec.long_name">PCM signed 16-bit little-endian</property>
<property name="meta.media.1.codec.bit_rate">1411200</property>
<property name="meta.attr.1.stream.language.markup">eng</property>
<property name="meta.attr.1.stream.handler_name.markup">Apple Sound Media Handler</property>
<property name="meta.attr.major_brand.markup">qt </property>
<property name="meta.attr.minor_version.markup">512</property>
<property name="meta.attr.compatible_brands.markup">qt </property>
<property name="meta.attr.encoder.markup">Lavf58.20.100</property>
<property name="seekable">1</property>
<property name="meta.media.sample_aspect_num">1</property>
<property name="meta.media.sample_aspect_den">1</property>
<property name="aspect_ratio">1</property>
<property name="audio_index">1</property>
<property name="video_index">0</property>
<property name="mute_on_pause">1</property>
<property name="mlt_service">avformat</property>
</producer>
<tractor id="tractor0" global_feed="1" in="" out="">
<track producer="producer0"/>
<track producer="producer1"/>
<track producer="producer2"/>
<transition id="transition0" out="114">
<property name="a_track">0</property>
<property name="b_track">1</property>
<property name="start">-213.33333333333337/120.0:853.3333333333335x480.00000000000006</property>
<property name="factory">loader</property>
<property name="aligned">1</property>
<property name="progressive">1</property>
<property name="mlt_service">composite</property>
<property name="end">-213.33333333333337/120.0:853.3333333333335x480.00000000000006</property>
<property name="distort">1</property>
</transition>
<transition id="transition1" out="114">
<property name="a_track">0</property>
<property name="b_track">2</property>
<property name="start">426.66666666666663/120.0:853.3333333333335x480.00000000000006</property>
<property name="factory">loader</property>
<property name="aligned">1</property>
<property name="progressive">1</property>
<property name="mlt_service">composite</property>
<property name="end">426.66666666666663/120.0:853.3333333333335x480.00000000000006</property>
<property name="distort">1</property>
</transition>
</tractor>
</mlt>
I compiled mlt from commit 0e756b692b4b1248d27254d124cd41f580878372--I upgraded from 16.6.0 where I had the same issue--with this configuration:
./configure --disable-decklink --disable-dv --disable-feeds --disable-frei0r --disable-gtk2 --disable-jackrack --disable-kdevlive --disable-kino --disable-linsys --disable-lumas --disable-motion_est --disable-ndi --disable-normalize --disable-oldfilm --disable-opencv --disable-opengl --disable-qt --disable-resample --disable-rtaudio --disable-sox --disable-swfdec --disable-vid.stab --disable-videostab --disable-vmfx --disable-vorbis --disable-xine --swig-languages=ruby
And I'm on Mac OS Catalina 10.15.7.
If it's helpful at all, here's a shorter version of input.xml where I stripped out the pieces that didn't seem to be relevant. It should still reproduce the issue.
<mlt LC_NUMERIC="en_US.UTF-8" version="6.23.0">
<profile description="DV/DVD PAL" width="1280" height="720" progressive="1" sample_aspect_num="1" sample_aspect_den="1" display_aspect_num="1" display_aspect_den="1" frame_rate_num="30" frame_rate_den="1" colorspace="709"/>
<producer id="producer0" in="0" out="114">
<property name="mlt_service">colour</property>
<property name="colour">0x000000FF</property>
</producer>
<producer id="producer1" in="0" out="114">
<property name="resource">pets000.mov</property>
</producer>
<producer id="producer2" in="0" out="114">
<property name="resource">pets000.mov</property>
</producer>
<tractor id="tractor0" global_feed="1" in="" out="">
<track producer="producer0"/>
<track producer="producer1"/>
<track producer="producer2"/>
<transition id="transition0" out="114">
<property name="a_track">0</property>
<property name="b_track">1</property>
<property name="start">-213.33333333333337/120.0:853.3333333333335x480.00000000000006</property>
<property name="progressive">1</property>
<property name="mlt_service">composite</property>
<property name="end">-213.33333333333337/120.0:853.3333333333335x480.00000000000006</property>
<property name="distort">1</property>
</transition>
<transition id="transition1" out="114">
<property name="a_track">0</property>
<property name="b_track">2</property>
<property name="start">426.66666666666663/120.0:853.3333333333335x480.00000000000006</property>
<property name="progressive">1</property>
<property name="mlt_service">composite</property>
<property name="end">426.66666666666663/120.0:853.3333333333335x480.00000000000006</property>
<property name="distort">1</property>
</transition>
</tractor>
</mlt>
I wonder if this problem still exists in MLT master?
I do not reproduce the problem, but I also do not have your source file pets000.mov
One possible explanation is if your source file might be incorrectly identifying the colorspace. It says it is 709, but maybe it is actually 601. If this is the case, then the first time it require conversion it will experience a subtle color shift. It is not uncommon for a transition to trigger a colorspace conversion since it operates in 4:2:2 while the source file is 4:2:0.
Hm, I just pulled and recompiled (SHA 817e4d8f9396913c0ed8410409eb84eebb8e79aa), but the output seems the same for me. The source file pets000.mov can be downloaded here if you decide to look any further: https://embed-ssl.wistia.com/deliveries/d2e5245e04b6bcdce7981a5bb550b602/pets000.mov?disposition=attachment.
To your point, relevant pieces of ffprobe on pets000.mov:
pix_fmt=yuv420p
color_range=unknown
color_space=unknown
color_transfer=unknown
color_primaries=unknown
chroma_location=left
And from mediainfo:
Color space : YUV
Chroma subsampling : 4:2:0
Still, if MLT is converting the output to a different colorspace, and the transition applies to the entire duration of the file, why wouldn't it be converted on the first frame, too? Then again, if it's not showing up for you when testing with the same input file, maybe there's something else entirely going on with my setup.
A workaround for your problem is to replace the composite transitions with affine instead. It is as easy as changing:
<property name="mlt_service">composite</property>
to
<property name="mlt_service">affine</property>
Perhaps there is a bug in the composite transition. I did not investigate inside the composite transition because it has so many other deficiencies relative to the affine transition that I typically recommend people use the affine transition in all cases. The only drawback is that the affine transition is slightly slower.