Video Mirroring Issue

[tomashined]

Hello. In AVPro Video - Core Edition (2.9.3), I am encountering an issue where the video I retrieve through mediaPlayer.TextureProducer.GetTexture() is mirrored both vertically and horizontally. In the MediaPlayer, the video is displayed in the correct orientation, but in the interface, everything is mirrored, even though the interface itself does not perform any flips. How can this be fixed? This occurs both in Unity Editor on Windows 11 and in the built application for iPad.

[Chris-RH]

Hi @tomashined

Are you able to send us a sample of the video that you are using so that we can confirm what is going on, please? Either here or to [email protected] with #2014 in the subject line Have you tried using the resolvetorendertexture component?

[tomashined]

@Chris-RH I have sent the video to your email, but there’s nothing particularly special in it. I am not using "Resolve To RenderTexture." I obtain the texture using mediaPlayer.TextureProducer.GetTexture() and pass it to a VisualElement (Unity UI Toolkit). However, for some reason, the texture is mirrored along both axes, even though the VisualElement has no transformations applied, and regular textures are displayed as I pass them. This means the texture is being flipped somewhere inside TextureProducer. How can this be fixed?

[Chris-RH]

Hi @tomashined,

OK, that added info makes a bit more sense. For info: mediaPlayer.TextureProducer.GetTexture() will return the texture from the native plugin as is. For performance reasons we do not correct the textures orientation as that would require further processing and memory usage for something that can be handled as transform in a shader. mediaPlayer.TextureProducer.GetAffineTransform() returns the required transform for correcting the texture orientation and is what our various components use to correct things. mediaPlayer.TextureProducer.GetTextureMatrix() returns the transform as a Matrix4x4 for ease of use with Unity's renderer, shaders, etc.

We don't have experience of Unity's UI Toolkit. There may be a different way to pass the transform on to it, but I couldn't say for sure. I suggest using our resolvetorendertexture component and apply the render texture to your UI component. That should orient the texture correctly.

[tomashined]

@Chris-RH You don’t need to have experience with Unity UI Toolkit; I just need help getting the Texture2D in the correct orientation (as shown in the video, as it appears in the VideoPlayer). I don't understand how to extract Texture2D from "Resolve To RenderTexture" if this component provides the correct texture orientation. Also, I don’t understand why the video is being mirrored along both axes when it was recorded correctly in the first place. Why is this rotation happening?

[MorrisRH]

Hi @tomashined

Thanks for sending on the video. Having tested with it I can see that it is vertically flipped. This is just how the frames are stored in the video and something that is corrected for when playing back using our components.

Here is some sample code for how to correct for this when applying to an Image UIToolKit component:

bool flip = _mediaPlayer.TextureProducer.RequiresVerticalFlip();
if (flip)
{
	// _video is an Image component, flip Y
	_video.style.scale = new Scale(new Vector2(1.0f, -1.0f));
	// _group is the child of _video and requires flipping also to put things back into the correct orientation
	_group.style.scale = new Scale(new Vector2(1.0f, -1.0f));
}

[tomashined]

@MorrisRH @Chris-RH

I have considered the option of flipping the VisualElement, but that doesn't work for me. I need to receive the Texture2D in the correct orientation because, for example, I also have processes that involve identifying specific elements at a given point. Therefore, it is crucial for me to have the Texture2D in the correct orientation for the video. Why is it that in your MediaPlayer component, the video is displayed correctly (just like in a regular player), but when rendered on a texture, it suddenly appears flipped along both axes? This is unacceptable to me, which is why I'm reaching out to you for a solution to this issue.

[MorrisRH]

Sounds like the Resolve To Render Texture component is your best bet then.

Attach one to the GameObject with the MediaPlayer component and attach the media player to it. I don't think you'll need any of the Resolve Flags so set this to Nothing. It should look something like this:

In a script you just need to assign ResolveToRenderTexture.TargetTexture to the Image, something like this:

	public ResolveToRenderTexture _resolve;
	private Image _video;

	void OnEnable()
	{
		UIDocument uiDocument = GetComponent<UIDocument>();
		_video = uiDocument.rootVisualElement.Q("video") as Image;
	}

	public void Update()
	{
		if (_resolve)
		{
			_video.image = _resolve.TargetTexture;
		}
	}

[tomashined]

I see that TargetTexture is a RenderTexture, but how can I obtain a Texture2D? I've mentioned several times that I need a Texture2D to manipulate the pixels. You still haven't explained why the texture is flipped, even though the video and MediaPlayer are in the correct orientation. @MorrisRH @Chris-RH

[MorrisRH]

You'll need to use Texture2D.ReadPixels() to get a copy of the texture in CPU accessible memory, Unity has an example in their documentation here.

The texture is flipped in the video. Video players and our components detect this and account for it. Because you are using the texture directly the flip hasn't been corrected for.

[tomashined]

@MorrisRH

Why are you saying the video is flipped initially, if it appears in the correct orientation when viewed through the default video player in Windows 11? I record the video using VideoKit (https://www.videokit.ai/), and the texture is in the correct orientation there. The video I sent you was recorded using their tool.

[MorrisRH]

Video frames are stored with their origin bottom left, whereas textures have their origin top left so a flip is required. This is normal and not specific to the video you're using. You can see this with our sample video in StreamingAssets/AVProVideoSamples/RenderHeads-1080p30-H264.mp4.

[tomashined]

@MorrisRH I'm having trouble implementing this method. Would you kindly create an example on your side where you obtain a Texture2D in the correct orientation?

[tomashined]

@MorrisRH @Chris-RH ?

[MorrisRH]

You can do something similar to the following coroutine to get a Texture2D from the resolved render texture:

	private IEnumerator GrabFrameAtTime(float time)
	{
		yield return new WaitForSeconds(time);
		yield return new WaitForEndOfFrame();

		Texture2D texture = new Texture2D(_resolve.TargetTexture.width, _resolve.TargetTexture.height, TextureFormat.RGBA32, false);
		RenderTexture currentRT = RenderTexture.active;
		RenderTexture.active = _resolve.TargetTexture;
		texture.ReadPixels(new Rect(0, 0, _resolve.TargetTexture.width, _resolve.TargetTexture.height), 0, 0);
		RenderTexture.active = currentRT;

		NativeArray<Color32> pixels = texture.GetRawTextureData<Color32>();
		// Do something with the pixels
	}

[tomashined]

@MorrisRH

Okay, it seems like this solution works for me at the moment. After initially obtaining the video texture, I call InitializeResolveToRenderTexture():

private Texture2D InitializeResolveToRenderTexture()
{
    if (videoTexture != null) Destroy(videoTexture);
    videoTexture = new Texture2D(resolveToRenderTexture.TargetTexture.width, resolveToRenderTexture.TargetTexture.height, TextureFormat.RGBA32, false);
    ResolveVideoTexture();
    return videoTexture;
}

Then, I call ResolveVideoTexture():

private void ResolveVideoTexture()
{
    renderTexture = RenderTexture.active;
    RenderTexture.active = resolveToRenderTexture.TargetTexture;
    videoTexture.ReadPixels(new Rect(0, 0, resolveToRenderTexture.TargetTexture.width, resolveToRenderTexture.TargetTexture.height), 0, 0);
    videoTexture.Apply();
    RenderTexture.active = renderTexture;
}

I invoke this in the Update method when Play() is active or directly from the seek function (if I'm displaying frame by frame). Feel free to correct me if there is a more optimized way to do this.

However, I would prefer if basic things, like the initial rotation of the video texture, were handled by you from the start, especially considering that the image is already displayed with the correct orientation in the MediaPlayer component. Why is this so complicated?

[Chris-RH]

The mediaplayer component already uses the resolve code to correct it. We use our shaders and components such as resolvetorendertexture to handle the rotation. This is the whole reason that we supply a range of components and shaders - to make it easy for devs.

[tomashined]

@Chris-RH

Hello. I'm following up on my previous issue (https://github.com/RenderHeads/UnityPlugin-AVProVideo/issues/2014) regarding the problem where mediaPlayer.TextureProducer.GetTexture() as Texture2D returns a vertically flipped video frame, while I need to obtain a correctly oriented Texture2D.

Using _resolveToRenderTexture is quite heavy, especially since I need the correctly oriented Texture2D on every application tick.

private void ResolveTexture()
{
    _renderTexture = RenderTexture.active;
    RenderTexture.active = _resolveToRenderTexture.TargetTexture;
    _videoTexture.ReadPixels(new Rect(0, 0, _resolveToRenderTexture.TargetTexture.width, _resolveToRenderTexture.TargetTexture.height), 0, 0);
    _videoTexture.Apply();
    RenderTexture.active = _renderTexture;
}

Yes, the RenderTexture from ResolveToRenderTexture is in the correct orientation, but what I really need is a Texture2D.

Is there a more optimized implementation?