Quality of PNG images is lost in a video created with Media3 Transformer and colors are darker

[yuriikonovalov]

After creating a video with Transformer the quality of PNG images are somewhat lost. It's possible to see if zoom in. Besides it happens both when passing a PNG as a MediaItem or as a BitmapOverlay. The sizes of the output video and the input images are the same - 1080:1920.

The difference between the original PNG image (with black) and the frame of the exported video (with red/brown):

The unscaled frame of the exported video:

One of the original PNG imaged used to create a video - https://drive.google.com/file/d/1qDw0armszS_3TVZs4W4NfptrvwvaSr9w/view?usp=drive_link

https://github.com/androidx/media/assets/84384099/83754132-f66e-46d8-8813-1e03d1063496

The link to the created video - https://drive.google.com/file/d/1PQ5Ngp2zi9NNeibeiV6-xS8egPyKx8Ku/view?usp=drive_link Looks like Google Drive player does not play the video with the heighest quality so you need to download it and open locally.

Is there any explanation and a way to fix this? And also it's visible that colors in the exported video are darker?

[droid-girl]

Hi @yuriikonovalov, could you share your Transformer export settings, frame rate or any other parameters you set in Transformer?

[yuriikonovalov]

Hi @droid-girl, here are MediaItem and Transformer builders. Just in case it matters, the PNG frame is created via ffmpeg library from a webm video.

Media items:

    val backgroundMediaItem = MediaItem.Builder()
        .setUri(backgroundUri)
        .setMimeType(MimeTypes.IMAGE_PNG)
        .build()

    val backgroundEditedMediaItem = EditedMediaItem.Builder(backgroundMediaItem)
        .setDurationUs(6_000_000)
        .setFrameRate(30)
        .setEffects(
            Effects(
                listOf(),
                listOf(Presentation.createForWidthAndHeight(1080, 1920, Presentation.LAYOUT_SCALE_TO_FIT_WITH_CROP))
            )
        )
        .build()

    val frameMediaItem = MediaItem.Builder()
        .setUri(frameUri)
        .setMimeType(MimeTypes.IMAGE_PNG)
        .build()

    val frameEditedMediaItem = EditedMediaItem.Builder(frameMediaItem)
        .setDurationUs(6_000_000)
        .setFrameRate(30)
        .setEffects(
            Effects(
                listOf(),
                listOf(Presentation.createForWidthAndHeight(1080, 1920, Presentation.LAYOUT_SCALE_TO_FIT_WITH_CROP))
            )
        )
        .build()

Video composition:

    val backgroundSequence = EditedMediaItemSequence(backgroundEditedMediaItem)
    val frameSequence = EditedMediaItemSequence(frameEditedMediaItem)

    val composition = Composition.Builder(frameSequence, backgroundSequence)
        .setEffects(
            Effects(
                listOf(),
                listOf(
                    Presentation.createForWidthAndHeight(1080, 1920, Presentation.LAYOUT_SCALE_TO_FIT_WITH_CROP),
                    FrameDropEffect.createDefaultFrameDropEffect(30f),
                )
            )
        )
        .build()

Transfromer settings:

    val transformer = Transformer.Builder(context)
        .setVideoMimeType(MimeTypes.VIDEO_H264)
        .setAudioMimeType(MimeTypes.AUDIO_AAC)
        .setEncoderFactory(DefaultEncoderFactory.Builder(context.applicationContext).build())
        .addListener(transformerListener)
        .build()

    transformer.start(composition, outputFilePath)

[yuriikonovalov]

Here is also a video created from same background and frame PNG images but using ffmpeg.

https://github.com/androidx/media/assets/84384099/72fe2505-52e9-4a98-8a5e-b6acd8aacbc2

[droid-girl]

Thank you for sharing this information. There might be several reasons for this behaviour. @tof-tof: I wonder if downscaling is producing this artifact and a downscaling algorithm would solve this problem? or is there another reason?

[tof-tof]

Hmmm not completely sure. sounds a bit like #1050 which is being solved by upcoming work, we'll see if that solves this as well

[yuriikonovalov]

@tof-tof I guess #1050 fixes the color problem but not the quality problem. Maybe the reason for that is the bitrate of the output video? I compared the original video (size - 1080:1920, so scaling should not be the case) with the output video. The bitrate of the original video is 13815kbps. The bitrate of the output video is 8582kbps.

The original video and the output video.

[droid-girl]

As an experiment, you can try setting bitrate value as described here. Assigning this issue to @ychaparov for his input

[ychaparov]

Thanks @yuriikonovalov for the detailed bug report!

The problem you observed is caused by precision loss in the OpenGL coordinates we use for sampling the correct pixel from the input image.

Since we're sampling a high-res 1920x1080 image, we need OpenGL ES highp precision for the texture coordinates. However, this coordinate change in fragment shader was bumping us down to 10-bit mediump precision https://github.com/androidx/media/blob/d833d59124d795afc146322fe488b2c0d4b9af6a/libraries/effect/src/main/assets/shaders/fragment_shader_transformation_sdr_internal_es2.glsl#L137 -- 10-bit mediump float is insufficient to get the correct pixel from the 1920-pixel tall image you were using.

Because of this floating-point error, we saw the jagged line you reported.

We will work on a fix, thanks for catching this!

[yuriikonovalov]

@ychaparov Thanks for your response! If it's possible, could you provide some time estimations when it can potentially be fixed?

[ychaparov]

This is probably now fixed on https://github.com/androidx/media/tree/main (main branch) with https://github.com/androidx/media/commit/ae240606dbbfd8105d2a23084b33ad35e5b6be87 .

Please take a look if the issue is now fixed for you

[yuriikonovalov]

@ychaparov just tried that and it's fixed now