flowframes
flowframes copied to clipboard
n00mkrad
Flowframes somehow desaturates videos with full color range
If you have a full color range video (I was personally interpolating some TikToks and found some of these - it's not rare as you'd think) and you encode it with any set of options available in Flowframes, the colors will get BTFO. I initially thought that the solution is to use:
-pix_fmt yuvj420p
but this doesn't seem to fix it
Re-encoding a full color range yuv420p h264 .mp4 with the following ffmpeg switches outside of flowframes yields correct color retention, despite clipping the color range from "full" to "limited" (???)
ffmpeg -i "%~1" -c:v libx264 -pix_fmt yuv420p -threads 8 -ac 2 -movflags faststart -preset veryfast -c:a copy -crf 18 -b:v 0 "%~n1_264%~x1"
Likewise, specifying yuvj420p yields the correct colors here
On the other hand, I cannot seem to get flowframes to retain the correct colors for full color range videos with any combination of GUI settings or CLI switches.
See this link for some relevant info: https://news.ycombinator.com/item?id=20036710
Here's a file that does it
https://mega.nz/folder/5EMWDDrS#MlG_UA9CHvG633ldHvmnTQ
It does not seem to be all full color range files, but rather a subset of them, I don't know what distinguishes them.
The videos are pretty different actually, inside (putting even aside their source.. I'm not sure what kind of platform would be delivering you H265 clips)
Anyhow, the problem can be already seen at extraction time.
ffmpeg.exe -i "this one does it.mp4" -q:v 1 "%9d.jpg"
If you extract to png though, everything works as expected. @ValZapod do we have a ticket for this?
(putting even aside their source.. I'm not sure what kind of platform would be delivering you H265 clips)
videos are tiktok source files
TikTok delivers watermarked h264 576p/540p files for its website, unwatermarked h265 576p/540p files for its app, and then there are these h264 (old devices) / h265 (new devices) 540p/720p/1080p source files that are not delivered anywhere but are exposed possibly just so that users can do duets/stitches on their devices instead of TikTok's servers
Testing more files from the user whose video was desaturated by FF, I cannot find any videos that don't get desaturated, although I only tested 15 or so. Added a few more examples to the mega folder
Pretty odd, especially considering the lack of watermark. Anyhow, I just found out that even disabling "Import HQ JPEG instead of PNGs" still isn't enough to fix the problem.
ffmpeg.exe -i "%9d.png" -vsync 0 -r 2901/50 -c:v libx264 -crf 10 -pix_fmt yuv420p10le -vf scale=out_color_matrix=bt709 -colorspace bt709 -color_primaries bt709 -color_range:v "pc" -threads 0 "test.mp4"
Putting aside converting is kinda stupid imo, you have to drop color_range for everything to finally work out.
Or force the media player to read the video using 16-235. Only the gods know wtf ffmpeg is doing.
I'm not sure what kind of platform would be delivering you H265 clips
You cannot be serious right now. TikTok (ByteDance Ltd.) has actually the best software proprietary hevc encoder, much better than x265 or mainconcept. See: https://www.compression.ru/video/codec_comparison/hevc_2019/download/4k_report_free_version/Free_4K_Video_Codec_Comparison_Report_2019_by_MSU.pdf
The videos looks like are encoded with apple core media though.
https://news.ycombinator.com/item?id=20036710
This was a very crazy scandal strory, but the end of it was simple: the author of that post on ycombinator does not understand how ffmpeg works. We did allow utilizing deprecated yuvj420p construct for h.265 (in https://github.com/FFmpeg/FFmpeg/commit/80757bed89c361fe2dc56eb61f4cb3b9f9b1aedc), but everone who uses ffmpeg should not have used it at all!! Anyway, since AV1 we no longer allow yuvj4xxp formats. I told everything I think about this genius guy here in av1 "bug" -- https://trac.ffmpeg.org/ticket/8455
You cannot be serious right now. TikTok (ByteDance Ltd.) has actually the best software proprietary hevc encoder, much better than x265 or mainconcept.
- People aren't born savvy
- Bytedance having some encoder doesn't mean they use it for tiktok
- Indeed.. of all the video download services I could find, none offered H265 (though none of them had HD quality either, and I discovered watermarks aren't actually hardcoded)
- It's not even the best encoder tbh, if you aren't just watching the 4K results
The video looks like are encoded with apple core media though.
That too. Especially odd considering they decided to use different codecs between the two videos. But maybe it has something to do with their date and/or whatever special method was used to retrieve them.
With this said, what the hell is going on here, with:
- extracting to jpeg (well, yuv I guess) doesn't respect the source bt709 flag and just assumes the usual bt601
- everything is treated like limited when converting to h264 (also the color_range behaviour perhaps, but I'm not sure)
If you extract to png though, everything works as expected.
It does not. See: exiftool.exe 121211.png
Gamma: 1.961
That is incorrect. Gamma is supposed to be 2.4. Another bug in ffmpeg, long time to be fixed, the PNG should use gAMA of EOTF not OETF that is 1.961 but with linear part from 0 to 0.0018. Case closed. You could have understood that if you used photoshop or ffplay that do not support color managment using gAMA. (And instead just assume sRGB curve (or 2.2 gamma, even worse) for both BT.709 OETF and for png images with any gAMA.)
@ValZapod do we have a ticket for this?
We do not, I forgot to open this. This is a bug in png encoder, ever since the gAMA chunk is actually written. Since https://github.com/FFmpeg/FFmpeg/commit/8d74baccff59192d395735036cd40a131a140391
Putting aside converting is kinda https://github.com/n00mkrad/flowframes/issues/52#issuecomment-1056936476 imo, you have to drop color_range for everything to finally work out.
Your command is wrong. swscale defaults to limited range. -vf scale=out_color_matrix=bt709:out_range=pc -color_range pc
I initially thought that the solution is to use:
-pix_fmt yuvj420p
That is deprecated. yuvj4xxp are deprecated.
despite clipping the color range from "full" to "limited"
Yes, swscale defaults to limited. Yes, you must tag limited or full by hand using -color_range pc. Yes. It is what it is a known bug (only in x264, range works in OpenH264, which are two different avc encoders). https://trac.ffmpeg.org/ticket/9167
this one doesn't do it
Is not tagged in any way. So...
You could have understood that if you used photoshop or ffplay that do not support color managment using gAMA.
I just checked with my own eyes the result tbh. But it turns out every goddamn program on my computer (except browsers ironically) was lying to me.
(And instead just assume sRGB curve (or 2.2 gamma, even worse) for both BT.709 OETF and for png images with any gAMA.)
2.2 seems good, at least with the png exported by madvr. But maybe that's already doing whatever needed internal conversion and consideration by itself.
We do not, I forgot to open this. This is a bug in png encoder, ever since the gAMA chunk is actually written.
The funny thing though, as far as the purposes of this program are concerned, is that if you take care of the jpg/png thing I explained above and the limited range factor.. everything works just fine and the exported final video is the same colouring of the original.
Yes, swscale defaults to limited. Yes, you must tag limited or full by hand using -color_range pc.
Maybe I'm missing something again.
But if (rgb->)yuv conversion uses limited out of the box, then isn't forcing color_range pointless? The input isn't some voodoo, we are kinda in control of it and it's just some standard png/jpg. Why interfering with ffmpeg's perennial hidden implicit assumptions that would be perfectly able to handle that content now, risking to ruin everything (as indeed happens)?
I mean.. perhaps vf scale=out_range=full could avoid a needless conversion, but it's kinda academical I guess.
It is what it is known bug (only in x264, range works in OpenH264, which are two different avc encoders). https://trac.ffmpeg.org/ticket/9167
Damn, I hadn't considered that's also true for all properties other than matrix too.
So (until that's fixed) every similar application should always like force -bsf:v h264_metadata=video_full_range_flag=0:matrix_coefficients=6:(colour_primaries=?:transfer_characteristics=?)?
GIMP does not lie anymore. BTW, on that issue I commented too ;) Valerii there :)
except browsers ironically
They just use turbo-jpeg and libpng by default that work correctly even for cHRM primaries color managment. There are still some problems in mobile though, it works, but colors may be not be all the same due to how rendering on Android works, it may render the same color differently, LOL. Not the first time.
2.2 seems good, at least with the png exported by madvr. But maybe that's already doing whatever needed internal conversion and consideration by itself.
It is slightly different compared to sRGB.
Also you cannot use 2.2 gamma or any pure gamma for encoding, since you just lose a lot of blacks.
perhaps vf scale=out_range=full could avoid a needless conversion, but it's kinda academical I guess.
JPEG does not need it since it only supports full and thus full is forced, and if you force limited it will warn that strict-std-complience should be changed to force limited into JPEG. You can think that jpeg output is tagged. That is the only exception though. In all other cases you need to understand what you want. With jpeg 2000 you can have two ranges too in one of 4 YCbCr formats.
So (until that's fixed) every similar application should always like force
Not in all cases. If all 3 values are in the source then the output will be properly tagged in all 3 values if you do not convert...
(rgb->)yuv conversion uses limited out of the box, then isn't forcing color_range pointless?
color_range is a global option that just forces the tag. It does not play any role in scale convertion. That is a different option. After yuvj4xxp pixel formats were deprecated the only way to output full range is out_range=pc. Okay? There is also -dest_range option that signals to swscale too but it does not work, I think. I never use it.
GIMP does not lie anymore. BTW, on that issue I commented too ;) Valerii there :)
Yes, I know. But alas poor normies don't install the dev version.
There are still some problems in mobile though
Don't even get me started.. and my god firefox and hdr are an absolute dumpster fire
JPEG does not need it since it only supports full and thus full is forced, and if you force limited it will warn that strict-std-complience should be changed to force limited into JPEG. You can think that jpeg output is tagged.
I was talking about the images->video step. Is there some sound logic that would forego conversion in that case, or would that happen again after the never-asked conversion to limited?
If all 3 values are in the source then the output will be properly tagged in all 3 values if you do not convert...
AI "enhancers" inherently have that passage.
color_range is a global option that just forces the tag.
Isn't it about forcing a certain "interpretation" of the input, as opposed to flagging the output?
After yuvj4xxp pixel formats
Forget that nightmare, I'm not OP. My target is glorious yuv420p10le if you didn't notice.
I was talking about the images->video step. Is there some sound logic that would forego conversion in that case, or would that happen again after the never-asked conversion to limited?
Oh. There is no double convertion. Except for the fact you already lost a lot of colors going into rgb24. First of all YCbCr is bigger, much bigger than rgb of the same transfer and primaries, so you clipped all of that permanently and second in the case of jpeg the chroma sample location in 420 is different from hevc or avc sample location. And different with bt.2020/bt.2100 sample location for that matter. That is always a bad idea. you can try 444 jpeg though.
Damn.. Well, the thing is only jpeg/png/webp are officially supported. Maybe you can even consider tga/bmp/hdr/exr/jp2/tiff depending on the specifics of the decoding library each tool uses.. but I fear they all work internally in RGB space at the end of the day. Thankfully I couldn't notice any artifact or loss (at least when handling mundane SDR material), but I see how that throws the "avoid needless conversions" argument out of the window.
EDIT: lol at d5d6048484217d5641ca375cdbf770c10286c877
but I fear they all work internally in RGB space at the end of the day.
Well, davinci resolve has the ability to preserve this superblack and superwhite data and so does ffmpeg. Also good tvs do have the ability to present them on display too even using internal player, most tvs do supports showing them when you accept ycbcr data.
consider tga/bmp/hdr/exr/jp2/tiff
Best way is lossless avif though, soon to be supported in ffmpeg. :)
Ngl, sometimes jpg is just good for its simplicity and speed. I can extract the whole clip in like a second or less, while png takes 5 seconds (on top of requiring almost ten times as much space)
..anyhow, enough with the chit chatting I almost lost the thread. You are right png are bugged, and I can see it now, and I'm dismayed. But (on top of, or at least parallel, to that) there still is the problem with extraction to jpeg giving no dams about the original colorspace.
You are right png are bugged, and I can see it now, and I'm dismayed.
You can fix it with simple pngcrush -m 40 -rem gAMA input.png withoutgAMA.png
But (on top of, or at least parallel, to that) there still is the problem with extraction to jpeg giving no dams about the original colorspace.
Now I am confused. Did you say that png is good? Well it was not... Does not it mean jpeg is correct? Or are you saying jpeg is also bad? Anyway, jpeg here works great! The color is the same. And mpv is also the same in sdr at least.
Yes, I mean jpeg is not correct here. At least with 2022-03-17-git-242c07982a.
As I already said it seems to always assume the source is bt601 (unlike pngs, which put gAMA aside, are fine eventually).
But that ain't good? That's what you get from the video when you decode it with bt601 rather than bt709.
But that ain't good? That's what you get from the video when you decode it with bt601 rather than bt709.
Oh, wow. Okay, yes you are right. I had to use https://github.com/IENT/YUView for it to be obvious. It thinks it is full range bt.601 while it is full range bt.709. ffplay at least is decoding correctly. WTF. LOL. You got me, congrats.
Okay, this is this patch here and it was not enough apparently. Goddamit. https://patchwork.ffmpeg.org/project/ffmpeg/patch/[email protected]/ https://github.com/FFmpeg/FFmpeg/commit/c523724c693ee031d8b5351aff6232a1f7d86876
This is correct png image with removed gAMA (will attach on trac too):
It appears the same as in YUView.
I have some slight difference here, especially around the contours of the light strip on the ceiling.. but it's not really about color hue, so I guess it's just madvr/EVR retouching the image a bit. Not sure why you'd need YUView though, everything should be already fairly evident from any normal image viewer (well provided you have the right png as a reference, at least)
Not sure why you'd need YUView though
Because it has this, supports hex view of 4:2:0 samples and has ffmpeg integration:
provided you have the right png as a reference
Actually you cannot for the changes in center chroma location. Compare ffmpeg.exe -i "this one does it.mp4" -frames:v 1 -vf zscale=min=709:m=170m:cin=left:c=center -q:v 1 3182811.jpg and ffmpeg.exe -i "this one does it.mp4" -frames:v 1 -vf zscale=min=709:m=170m -q:v 1 31828111.jpg
Fiest one is correct, so there is at least this workaround. Again zscale goes forth.
In "Beyound Compare" the file with jpeg with correct jpeg center chroma vs left chroma (most pixels are the same, blue pixels cannot be seen, not really):
Holy fucking abominable shit. I salute your courage and effort.
EDIT: as if the existential horror wasn't enough, now the comments from the most informed guy on the net disappeared https://archive.is/WtKhC