Get every frame as bitmap and draw on it

[paulpogostick]

Hi. I need to get every frame that is being rendered as a System.Drawing.Bitmap (or something equivalent) so I can process it and then see this processed frame rendered (like printing letters on the bitmap). I have read other related posts, but I have not found a solution for this.

So far I know that in Meta.Vlc\Meta.Vlc.Wpf\VlcPlayer.Events.cs there is a method called VideoDisplayCallback which I think gets every frame in this.VideoSource, but if I set this.VideoSource = processBitmap(this.VideoSource);

only the first frame is shown constantly on the screen.

(I am doing something like this, but like I said, only the first frame is shown constantly)

void VideoDisplayCallback(IntPtr opaque, IntPtr picture)
{
	if (_context == null || DisplayThreadDispatcher == null)
	{
		return;
	}

	DisplayThreadDispatcher.BeginInvoke(System.Windows.Threading.DispatcherPriority.Background, new Action(() =>
	{
		this.VideoSource = processBitmap(this.VideoSource);

		_context.Display();

	}));

	try
	{
		TakeSnapshot();
	}
	catch
	{
		// ignored
	}
}

Is there a way to do this correctly (maybe converting the IntPtr picture parameter as a Bitmap?), and can it be donde without modifying the original source code, like with a callback?

Thanks in advance and congratulations for this very good library.

[devkanro]

IntPtr picture is a surface pointer. you can get the pixel data from it.

A 3x3 surface with RV32 packing data like this:

RGBX RGBX RGBX RGBX RGBX RGBX RGBX RGBX RGBX
|____________|->pitch(12bytes/3pixel)
|__|->pixel(4 bytes)
RGBX is RV32 packing, it is 4 bytes data for each pixel.
R is the red channel.
G is the green channel.
B is the blue channel.
X is non-using data. 
every channel is 1 byte

Pixel data is 3x3x4(36)bytes width = 3(px) height = 3(px) pitches = width x 4(bytes/px) = 12(bytes) lines = height x 1(line/px)

you can found those information in VideoFormatCallback

[paulpogostick]

Thank you. I tried assigning values to IntPtr picture but the rendered image was not modified.

I see that in VideoDisplayCallback this line is called:

_context.Display();

and in Display()

Image.Invalidate() is called in every iteration.

So, to modify every frame, Image must be modified? If so, how?

Or is it something completely different?

Thanks again.

[LarsWesselius]

You're getting raw pixel data back in the format specified. picture is vital in this. You can create a System.Drawing.Bitmap from it, if you like. Easiest way is using this constructor: https://msdn.microsoft.com/en-us/library/zy1a2d14(v=vs.110).aspx

Assuming 32bpp:

using (Bitmap myBmp = new Bitmap(myWidth, myHeight, myWidth * 4, PixelFormat.Format32bppArgb, picture))
{
}

The Meta.Vlc WPF control uses Image.Invalidate() to force an update. The data (image) itself is coming from MapView which is simply some reserved memory.

[paulpogostick]

Thank you! (sorry for the late reply, I was sick and I am recovering now :) ). I used your code and I process every bitmap with different effects. In this example I turn them as negative versions of the originals:

void VideoDisplayCallback(IntPtr opaque, IntPtr picture)
{
	if (_context == null || DisplayThreadDispatcher == null)
	{
		return;
	}

	System.Windows.Interop.InteropBitmap Image = _context.Image;
	if (Image != null)
	{
		Image.Dispatcher.Invoke(new Action(() =>
		{
			if (Image != null)
			{
				int myWidth = _context.Width;
				int myHeight = _context.Height;
				Bitmap image = new Bitmap(myWidth, myHeight, myWidth * 4,
 PixelFormat.Format32bppArgb, picture);

				image = CopyAsNegative(image);

				Image.Invalidate();
			}
		}));
	}

	try
	{
		TakeSnapshot();
	}
	catch
	{
		// ignored
	}
}

This is CopyAsNegative():

public static Bitmap CopyAsNegative(Bitmap bmpNew)
{
	BitmapData bmpData = bmpNew.LockBits(new Rectangle(0, 0, bmpNew.Width, bmpNew.Height), ImageLockMode.ReadOnly,
		PixelFormat.Format32bppArgb);

	IntPtr ptr = bmpData.Scan0;

	byte[] byteBuffer = new byte[bmpData.Stride * bmpNew.Height];

	System.Runtime.InteropServices.Marshal.Copy(ptr, byteBuffer, 0, byteBuffer.Length);
	byte[] pixelBuffer = null;

	int pixel = 0;

	for (int k = 0; k < byteBuffer.Length; k += 4)
	{
		pixel = ~BitConverter.ToInt32(byteBuffer, k);
		pixelBuffer = BitConverter.GetBytes(pixel);

		byteBuffer[k] = pixelBuffer[0];
		byteBuffer[k + 1] = pixelBuffer[1];
		byteBuffer[k + 2] = pixelBuffer[2];
	}

	System.Runtime.InteropServices.Marshal.Copy(byteBuffer, 0, ptr, byteBuffer.Length);

	bmpNew.UnlockBits(bmpData);

	bmpData = null;
	byteBuffer = null;

	return bmpNew;
}

with that, if you load and play a video, the images are rendered with the negative effect. BUT when playing a video (specially HD videos), in the top of the image sometimes appears a 'rectangle' with the original image (flickers). I tried different effects like CopyAsGrayscale(); but the the same happens.

Capture of a video with the 'rectangle' in the top of the image:

Original video: http://distribution.bbb3d.renderfarming.net/video/mp4/bbb_sunflower_1080p_30fps_normal.mp4

How can I remove that flickering? Does it have to do with the Image.Invalidate(); line? (maybe the invalidation occurs asyncronously with respect to the processed frame, so it must be changed to be updated syncronously?)

Thank you very much again.

[LarsWesselius]

Hope you get well soon!

What I suspect is happening is that the picture IntPtr is being updated before your code finishes updating the Image, causing tearing (as in, part of the frame buffer already contains new data). You might have to fiddle with locks or buffer video frames.

[mystery123sk]

I made a little example to use VLC to render video to bitmap, you can check it here:

https://github.com/mystery123sk/VLCRenderToBitmap