Clipboard support for bitmaps

[garyhertel]

Would it be possible to add bitmap support for the clipboard? It looks like right now only text is supported, although Avalonia does define a ClipboardFormat.CF_BITMAP enum value for this already.

I'm trying to add a screen capture tool for an Avalonia app. I've got the screen capture working already, but I can't copy the selected bitmaps to the clipboard since only text is currently supported.

[kekekeks]

OSX: NSPasteboardTypePNG X11: Need to add support for image/png, image/bmp, image/x-bmp, image/x-MS-bmp, image/jpeg TARGETS

[dfkeenan]

I would also very much like this. I am in the middle of trying to add support for Bitmap and SKBitmap to my app. I am trying to build it on top of the existing IClipboard.SetDataObjectAsync and IClipboard.GetDataAsync APIs.

I am only working on Windows at the moment. Though I am doing it in a way that adding support for other platforms might be possible. I have manged to get bitmaps from clipboard but my implementation doesn't handle all bitmap formats. Basically only 32 bit ones. I have been having some troubles trying to set clipboard though.

The SetDataObjectAsync appears to be erroring, well I am assuming that. The windows implementation loops infinity until it is successful which it never is. The Task never completes. So I don't know if the error is being caused by the data I am trying to pass in or there is something else wrong with the implementation of SetDataObjectAsync.

I was also curious why the DataObject methods use Ole rather than the User32 like the text methods do.

[danwalmsley]

@dfkeenan a recent PR was made to add custom clipboard formats...

https://github.com/AvaloniaUI/Avalonia/pull/4031

I think it would give a good guide on how to add bitmap support.

[jp2masa]

@dfkeenan I had a similar issue as well, probably the same. Try adding [STAThread] to Main, like this:

[STAThread]
static int Main (...)

[dfkeenan]

@danwalmsley , That's the API I am trying to use. I couldn't see any examples of usage though. Putting the [STAThread] on the entry point as @jp2masa suggested stopped the infinite loop. Thanks for the suggestion.

But my code for setting bitmap data doesn't work. ):

[dfkeenan]

So I have managed to put a SKBitmap into clipboard and get it back out in a different avalonia app. But I can't paste it into other apps like paint.net/aseprite.

[dfkeenan]

If you use WinForms to check the clipboard it does get different format names than Avalonia does. I tried using those names instead that didn't work either.

[kekekeks]

On Windows bitmaps has to be wrapped into HBITMAP object before going to the clipboard. So it requires special support from our clipboard implementation.

[dfkeenan]

I have tried putting a bitmap into clipboard using the below to create a HBITMAP. That does not seem to be working for me either.

[DllImport("gdi32.dll")]
private static extern unsafe IntPtr CreateBitmap(int nWidth, int nHeight, uint cPlanes, uint nBitCount, byte* lpvBits);

public static unsafe IntPtr CreateBitmap(int nWidth, int nHeight, uint cPlanes, uint nBitCount, Span<byte> lpvBits)
{
    fixed (byte* pbData = &MemoryMarshal.GetReference(lpvBits))
    {
        return CreateBitmap(nWidth, nHeight, cPlanes, nBitCount, pbData);
    }
}

Winforms finds Bitmap and System.Drawing.Bitmap formats. But if I try get the bitmap it returns null.

[dfkeenan]

I have also read that you need to allocate memory as movable using GlobalAlloc. I tried that but it didn't seem to work either.

[dfkeenan]

I wonder if we have to do something like DataObject.GetCompatiableBitmap

[dfkeenan]

Well I have tried using a System.Drawing.Bitmap and duplicating that GetCompatibleBitmap method linked in previous comment and try to put that into clipboard. That doesn't work either.

[dfkeenan]

I have created a standalone .NetStandard 2.1 project/library of all the code required to use System.Windows.Forms.Clipboard. I manged to get images in/out of a number of image editing software: Paint.Net, Aseprite, PyxelEdit. With varying degrees of success. i.e. the same as you get when using WinForms. I was converting between Avalonia.Media.Imaging.Bitmap and System.Drawing.Bitmap by saving/loading from a MemoryStream. Not sure if there is a better way.

I did this by keep adding files until it compiled. (I added them as links rather than copy.) It looks like a lot of code but every method DllImport etc. is in it's own file. I have not gone back through and removed everything that is not referenced. So It might be possible to cull a fair portion of it.

There were 3 classes I copied and changed because they had a bunch of unused methods that referenced types that I could not find. I also wanted to only include the bare minimum amount of the System.Windows.Forms.Application class. You could hardly call it an application anymore but I didn't change the name so I didn't have to change where it was referenced.

There were only a handful of things that required .NetStandard 2.1. I have not checked if they are required. But they looked easy enough to replace with something that is 2.0 compatible.

For the time being I also left in the code using BinaryFormatter to get/put .Net objects out/in the clipboard. Which i have been told you would not want in there.

[dfkeenan]

I have had another look at this today. I started by copying all the .cs files rather than using a linked reference.

• There are 3 things that only seem to be in NetStandard 2.1. No NuGet package for 2.0.
  • HashSet<T>(int capacity) constructor. - Could just replace that with HashSet<T>() I guess.
  • Stream.Read(Span<byte> buffer) method. - Looks simple enough to patch with an extension method.
  • Encoding.GetBytes(ReadOnlySpan<char> chars, Span<byte> bytes) method. - There appears to be an extension method but not in any NuGet package as far as I can tell. This looks to be a lot of code.
• Looking at the comments the serialization using BinaryFormatter is specifically for types like System.Drawing.Bitmap. There is logic there to "restrict" what it will serialize/deserialize //We are restricting serialization of formats that represent strings, bitmaps or OLE types.. Though to me it looks like the deserialization code would deserialize anything that is not in the list of restricted formats. Maybe someone smarter than me can have a look.
• There looks to be a bunch of code there just for debugging purposes. Would we want to keep this? I was attempting to change as little as possible so if there were fixes made to WinForms it would be easier to fix this copy.

Is it worth me to keep pursuing this?

[kekekeks]

Note that System.Drawing.Image implements ISerializable that just saves the bitmap data to a memory stream: https://github.com/dotnet/runtime/blob/6362ec357baaac66ae461b6a73c49cec5ea37470/src/libraries/System.Drawing.Common/src/System/Drawing/Image.cs#L83

[kekekeks]

Mkay, it seems that for bitmaps we need to:

1. use CreateDIBSection from binary data to get a DIB-backed HBITMAP
2. create a compatible DC and compatible bitmap
3. use BitBlt to copy DIB-backed bitmap to the standalone HBITMAP from (2)
4. use said HBITMAP for clipboard

That will require us to introduce IClipboard::SetBitmapAsync(Bitmap bmp) and Bitmap IClipboard:GetBitmapAsync() APIs and add some way to extract bitmap bits from the platform bitmap implementation in required format.

[dfkeenan]

Mkay, it seems that for bitmaps we need to:

1. use CreateDIBSection from binary data to get a DIB-backed HBITMAP
2. create a compatible DC and compatible bitmap
3. use BitBlt to copy DIB-backed bitmap to the standalone HBITMAP from (2)
4. use said HBITMAP for clipboard

I'm not 100% sure but I think that might be what this does. https://github.com/dotnet/winforms/blob/0ef3c13d76f1b2496ae1810d38a676e2e72adaf1/src/System.Windows.Forms/src/System/Windows/Forms/DataObject.cs#L119-L150

The current AValoniaUI Win32 Clipboard implementation does something with bitmaps. If you try to get the bitmap you get an byte[] rather then a handle to a bitmap. Which is why I started down the road of trying to read/write bitmaps using BITMAPINFOHEADER and BITMAPV5HEADER .

add some way to extract bitmap bits from the platform bitmap implementation in required format.

That's why I opened #4365

One downside of "compatible bitmaps" is I think you loose alpha channels. I would like to be able to copy images with alpha channels. I think windows has basically 2 main bitmap formats. DIB and DIBV5. I DIBV5 supports alpha channels. But I think if you want to use image data from other apps you basically need to workout/use what format they want. Aseprite, PyxelEdit, Paint.Net all seem to do there own thing for image data with alpha channels. ):

[dfkeenan]

Interestingly a lot of apps put a PNG format into clipboard which is just a png file. Which you can just read. But they ignore it if you put the same into clipboard. Seems they want either Bitmap or DeviceIndependentBitmap formats.

Something else that is weird about the current windows implementation is it reports things with format names like Unknown_Format_8 and Unknown_Format_17. I think Format 8 is what winforms calls Bitmap.

[garyhertel]

Where did we leave this feature? Have any recent changes made this any easier to implement?

At least for the app I'm working on, this is one of the top most useful features on the horizon. It would allow easy copy/pasting images into other apps like slack for discussions. While I can use other tools to do that, I also want to automatically upload the images and provide web links to add to tickets (currently possible). However, without clipboard access it's a usability nightmare since the same image can't be used for both.

[dfkeenan]

Where did we leave this feature? Have any recent changes made this any easier to implement?

I just had a bit of a look around the clipboard code on master. As far as I can tell no changes have been made.

[garyhertel]

I took a look through some of the links @dfkeenan mentioned earlier, and one of them had a pretty good clue on how to do this with bitmaps. So with a little work, I actually got it working copying a screenshot and pasting it into Paint. Here's what I ended up with, although it still needs some resource handling (those single line using statements in the new .net sure would be nice to have here). So what's our next step towards getting this added to Avalonia? :)

		public static async Task SetBitmapAsync(Bitmap bitmap)
		{
			if (bitmap == null)
				throw new ArgumentNullException(nameof(bitmap));

			// Convert from Avalonia Bitmap to System Bitmap
			var memoryStream = new MemoryStream(1000000);
			bitmap.Save(memoryStream); // this returns a png from Skia (we could save/load it from the system bitmap to convert it to a bmp first, but this seems to work well already)

			var systemBitmap = new System.Drawing.Bitmap(memoryStream);

			var hBitmap = systemBitmap.GetHbitmap();

			var screenDC = Win32UnmanagedMethods.GetDC(IntPtr.Zero);

			var sourceDC = Win32UnmanagedMethods.CreateCompatibleDC(screenDC);
			var sourceBitmapSelection = Win32UnmanagedMethods.SelectObject(sourceDC, hBitmap);

			var destDC = Win32UnmanagedMethods.CreateCompatibleDC(screenDC);
			var compatibleBitmap = Win32UnmanagedMethods.CreateCompatibleBitmap(screenDC, systemBitmap.Width, systemBitmap.Height);

			var destinationBitmapSelection = Win32UnmanagedMethods.SelectObject(destDC, compatibleBitmap);

			Win32UnmanagedMethods.BitBlt(
				destDC,
				0,
				0,
				systemBitmap.Width,
				systemBitmap.Height,
				sourceDC,
				0,
				0,
				0x00CC0020); // SRCCOPY

			try
			{
				await OpenClipboard();

				Win32UnmanagedMethods.EmptyClipboard();

				IntPtr result = Win32UnmanagedMethods.SetClipboardData(Win32UnmanagedMethods.ClipboardFormat.CF_BITMAP, compatibleBitmap);

				if (result == IntPtr.Zero)
				{
					int errno = Marshal.GetLastWin32Error();
				}
			}
			catch (Exception e)
			{

			}
			finally
			{
				Win32UnmanagedMethods.CloseClipboard();
			}
		}

        [DllImport("user32.dll", ExactSpelling = true)]
        public static extern IntPtr GetDC(IntPtr hWnd);


        [DllImport("gdi32.dll", ExactSpelling = true)]
        public static extern IntPtr CreateCompatibleDC(IntPtr hDC);


        [DllImport("gdi32.dll", ExactSpelling = true)]
        public static extern IntPtr CreateCompatibleBitmap(IntPtr hdc, int cx, int cy);

        [DllImport("gdi32.dll", SetLastError = true, ExactSpelling = true)]
        public static extern IntPtr SelectObject(IntPtr hdc, IntPtr h);

        [DllImport("gdi32.dll", SetLastError = true, ExactSpelling = true)]
        public static extern bool BitBlt(
            IntPtr hdc,
            int x,
            int y,
            int cx,
            int cy,
            IntPtr hdcSrc,
            int x1,
            int y1,
            uint rop);

[caesay]

This will not help people who need to process clipboard images on multiple platforms, however for anyone focused on Windows I have released a clipboard replacement library which has AvaloniaUI specific wrappers.

It's very easy to set and retrieve images on the clipboard:

// get image
Avalonia.Media.Imaging.Bitmap bitmap = ClipboardAvalonia.GetImage();

// set image
ClipboardAvalonia.SetImage(bitmap);

It is looks simple, but is very hard to get this right on windows. Using the CreateCompatibleBitmap method posted above, for example, is not recommended because the bitmap stored on the clipboard will be device dependent (eg. tied to current display) and will lack any transparency data. Storing and reading device independent bitmaps (CF_DIB or CF_DIBV5) is the preferred way, but pretty much every windows application has their own implementation - so if you want to do this well you need to handle those common inconsistencies.

The above is achieved in this library with a custom DIB parser and a lot of testing.

[kuiperzone]

Do we have a work around on Linux?

[CreateLab]

If we need some picture from clipboard, i test this code and it's work

        var formatsAsync = await Application.Current.Clipboard.GetFormatsAsync();
         object obj = null;
         if(formatsAsync.Contains("image/jpeg"))
            obj = (await Application.Current.Clipboard.GetDataAsync("image/jpeg"));
         if(formatsAsync.Contains("PNG"))
            obj = (await Application.Current.Clipboard.GetDataAsync("PNG"));
        
        var ms = new MemoryStream();
        Serializer.Serialize(ms, obj);
        byte[] data = ms.ToArray().Skip(4).ToArray();
        ms = new MemoryStream(data);
        

This work in widows with copy pase and with snipping tool and work with linux with just copy paste

it's dont work for specific task like copy from paint etc but i dont need it

[caesay]

This approach pasted above by @CreateLab should not be used on Windows. It will not work most of the time. Most Windows applications do not put JPEG or PNG on the clipboard. Even if they do, it could be stored under image/jpg, jpg, jfif etc - so if you want to read jpegs or png's you need to check lots of variations. Generally clipboard images are stored only as a DIB or DDB. And it's up to each application developer to interpret and implement the DIB spec, which is what leads to many differences and inconsistencies in how images are stored on Windows clipboard. This is especially true when handling transparency - as transparency was not supported in the original DIB spec. Checking whether an image has transparency, if it's premultiplied etc is a must for good compatibility. A few popular Windows applications do now also store images under the PNG format, so it is good to check this (and use it if available) however it's still very uncommon.

[CreateLab]

@caesay yep, u should understand with what app u wanna work with, that example work correctly with telegram app, with snipping tool For FireFox I need addition fix, like get data from FileNames and just get file from temp folder. It's not good idea but it mostly work.

For users if u wanna check u may get var formatsAsync = await Application.Current.Clipboard.GetFormatsAsync(); and find some information which app put there and try to parse it

[caesay]

You don't need to understand what app you're working with, you just need to use best practices when reading / writing to the clipboard which your code does not do. The library I shared will handle 99% of scenarios without any application specific code like the FireFox workaround you mentioned.

[CreateLab]

@caesay as i understand your lib works only in win os, am I right?

[kuiperzone]

@caesay Is your clipboard library still a Windows only solution? Or can it handle Linux now (or in the future)?

[caesay]

The linked library only handles Windows, but it does so well and handles the mentioned inconsistencies on the platform. I am not familiar with Linux, but as I understand it is common for Linux and Mac applications to store clipboard image data as a PNG, so it should be fairly easy to check if you are running on Windows or Linux. If running on Windows, use a library which can handle Windows DIB's and the inconsistencies that come with that. If running on Linux, retrieve PNG data as bytes and use your favorite parser.