Confusing error message for pixel format in some cases

[redzic]

Currently, if the input video has a pixel format that is considered unsupported by --encoder, when we try to get the pixel format for scene detection, a confusing error message is given, saying that "<encoder> does not support <pixel format>", even though the encoder is not being invoked.

[alanorth]

Indeed, that's misleading. In my case the videos are YUVJ420P:

$ av1an -i input.mp4
Scene detection
⠙ 00:00:00 [#####################################################################################################]  0 frames (0 fps) 
Error: aom does not support YUVJ420P

So we need to use --sc-pix-format=yuv420p:

$ av1an -i sand.mp4 --sc-pix-format=yuv420p                                              
Scene detection
  00:00:04 [#####################################################################################################] 100%  307/307 (74.40 fps, eta 0s) 
Queue 2 Workers 1 Passes 2
...

[shssoichiro]

I think the error message is correct, it's just confusing because the Scene Detection bar also shows. In reality we should show the error message before drawing the progress bar for the first time. And adding a note to use a different format, or maybe the list of supported formats for that encoder, seems like a good idea as well. I don't want to auto-convert the input colorspace however, because that may not be what a user wants or expects.

We also probably shouldn't invoke this validation if --sc-only is passed. But in normal circumstances, I think erroring as early as possible is the correct move.

[redzic]

I think the error message is correct, it's just confusing because the Scene Detection bar also shows.

The error message itself is not incorrect (in that, aom does not support YUVJ420P in and of itself is a true statement), however the fact that it is being displayed in this particular case is incorrect (and removing the scene detection bar would not fix it).

The reason this error happens is because the scene detection calls get_format_bit_depth in order to call the correct scene detection function. However, this function is tied to a specific encoder. Currently, we call this function with the encoder specified by --encoder, just to give it some kind of value so that we can actually call it, even though we want to call this function in a general way, in that we just want to get the bit depth of the pixel format we asked for.

Let's say the user tries encoding a video with SVT-AV1 which has the pixel format YUV444P10LE (with just av1an -i <input> -e svt-av1). The correct behavior would be to perform scene detection with 16-bit precision, and during actual encoding, convert the colorspace to YUV420P10LE. However, because we happen to be using SVT-AV1 which does not support YUV444P10LE, get_format_bit_depth will just return an error, saying "svt-av1 does not support YUV444P10LE", even though we are not even trying to encode with YUV444P10LE at all. The only reason this happens is because get_format_bit_depth erroneously ties an encoder to a pixel format. In theory, we should have a separate function for checking if an encoder supports a certain pixel format, and a separate function for getting the bit depth of a pixel format that is not specific to any single encoder.

Additionally, this error message will not be displayed at all if --split-method none is specified, so it really is tied specifically to scene detection rather than being an early error message.