ImageGrab.grab() doesn't ignore windows layered on top (tooltip, pop-ups, etc.)

[a-alak]

What did you do?

Run PIL.ImageGrab.grab() without arguments.

What did you expect to happen?

Until now this has not included pop-ups and tooltips in the screenshot. I just updated to windows 11. I can see in issue #2569 that it is the expected behavior to not include layered windows, and that the option to include them was added, but not as a default.

I suppose the underlying API has changed in windows 11. Anyone that can share some knowledge about this?

What are your OS, Python and Pillow versions?

• OS: Windows 11
• Python: 3.10.5
• Pillow: 10.4.0

--------------------------------------------------------------------
Pillow 10.4.0
Python 3.10.5 (tags/v3.10.5:f377153, Jun  6 2022, 16:14:13) [MSC v.1929 64 bit (AMD64)]
--------------------------------------------------------------------
Python executable is C:\Users\Public\SPEDA\env\Scripts\python.exe
Environment Python files loaded from C:\Users\Public\SPEDA\env
System Python files loaded from C:\Program Files\Python310
--------------------------------------------------------------------
Python Pillow modules loaded from C:\Users\Public\SPEDA\env\lib\site-packages\PIL
Binary Pillow modules loaded from C:\Users\Public\SPEDA\env\lib\site-packages\PIL
--------------------------------------------------------------------
--- PIL CORE support ok, compiled for 10.4.0
--- TKINTER support ok, loaded 8.6
--- FREETYPE2 support ok, loaded 2.13.2
--- LITTLECMS2 support ok, loaded 2.16
--- WEBP support ok, loaded 1.4.0
--- WEBP Transparency support ok
--- WEBPMUX support ok
--- WEBP Animation support ok
--- JPEG support ok, compiled for libjpeg-turbo 3.0.3
--- OPENJPEG (JPEG2000) support ok, loaded 2.5.2
--- ZLIB (PNG/ZIP) support ok, loaded 1.3.1
--- LIBTIFF support ok, loaded 4.6.0
*** RAQM (Bidirectional Text) support not installed
*** LIBIMAGEQUANT (Quantization method) support not installed
*** XCB (X protocol) support not installed
--------------------------------------------------------------------

[radarhere]

Could you attach an image to show what you're seeing?

[a-alak]

Yeah sorry, what i tested on was sensitive data, so i could not share. But here is an alternative example with the file explorer. Example 1 shows a tooltip with the url of the Outlook shortcut. Usually i would have expected the tooltip not to show in the screenshot like example 2.

[radarhere]

@nulano did you have any insight here?

[nulano]

I can confirm that tooltips are included with both include_layered_windows=False and include_layered_windows=True on Win 11 Pro version 23H2, but also on Win 10 Home version 22H2.

I just updated to windows 11.

Do you know which build of Win 10 you were running before you upgraded?

[a-alak]

Unfortunately not. And honestly I am more challenged by popups than tool tips, and I cannot show you an example pic of the pop up. It is running on a hospital system, where there often is weird pop up messages. They did not use to show up in ImageGrab, but now they do.

[radarhere]

Do these popups come from the same application that you are trying to see in your screenshot, or a different one? I'm wondering if the request of #4415 could be a potential solution.

[a-alak]

I think that might solve it! It is popups from another application. Some kind of background internal messaging service, that I can't disable.

[radarhere]

I've created #8516 to resolve this by adding a handle argument to ImageGrab.grab() that accepts a HDC.

With that, you could do something like

import win32gui
from PIL import ImageGrab
window = win32gui.FindWindow(None, "Insert window title here")
handle = win32gui.GetDC(window)
ImageGrab.grab(handle=handle)

[a-alak]

Amazing! Thank you @radarhere! Any idea when you can expect a release including PR #8516, how does pillow release cycles work? Sorry if the question is stupid.

[radarhere]

Pillow releases occur every three months - the next one is scheduled for January 2nd.

If you would like to try out the PR in the meantime, I've put together a wheel - pillow-11.1.0.dev0-cp310-cp310-win_amd64.whl.zip

[a-alak]

Amazing! Thanks @radarhere!

I am curious whether the tooltip issue can be solved as well? It is a minor problem for us, but I am just going to leave the issue open for that. If it is not possible or not something that people want in pillow I will close the issue 😇

[radarhere]

I expect it is something that people would want - I'm just not convinced that Windows makes that feature available.

https://learn.microsoft.com/en-us/windows/win32/api/wingdi/nf-wingdi-bitblt is the API that we use to get the screenshot, and CAPTUREBLT is the option that you can turn on and off with include_layered_windows that "Includes any windows that are layered on top". Nothing else on that page seems relevant.

https://github.com/python-pillow/Pillow/issues/8456#issuecomment-2427841869 was our attempt to test if there was a change between Windows 10 and 11, and we didn't find any difference.

[a-alak]

Thanks for sharing! I just tested out the wheel you send and I could not make it work. The screenshot is just a black screen. I run the exact same code you wrote on win 11 python 3.10.

[nulano]

I just tested out the wheel you send and I could not make it work. The screenshot is just a black screen. I run the exact same code you wrote on win 11 python 3.10.

Yep, that unfortunately matches what I found also, some applications don't seem to be able to be captured in this way: https://github.com/python-pillow/Pillow/pull/8516#pullrequestreview-2413911470

(worth noting that instead of using win32gui I used Microsoft Spy++; I saw that some applications have multiple windows and can only be captured by the outer-most one, but that is the one that has a title so it should also be selected by win32gui)

[radarhere]

I wonder if there's another way to solve your problem of these popups covering your application - by moving your application to the foreground? https://stackoverflow.com/questions/66164926/in-python-how-do-i-make-a-specific-window-stay-on-top

[a-alak]

Unfortunately the pop-up is somehow forced as foreground, so even though I try to pull target window to the front, it will not work.

Anyway, my colleague wrote this short script that works and does not have a black window:

import time
import ctypes
import win32gui
import win32ui
import win32con
from PIL import Image

# Window Title
window_title = 'SOME TITLE'

# Find the window by title
hwnd = win32gui.FindWindow(None, window_title)

if hwnd:
    # Get the window's dimensions
    left, top, right, bottom = win32gui.GetWindowRect(hwnd)
    width = right - left
    height = bottom - top

    # Get the window's device context (DC)
    window_dc = win32gui.GetWindowDC(hwnd)
    mfc_dc = win32ui.CreateDCFromHandle(window_dc)
    save_dc = mfc_dc.CreateCompatibleDC()

    # Create a bitmap object
    bitmap = win32ui.CreateBitmap()
    bitmap.CreateCompatibleBitmap(mfc_dc, width, height)
    save_dc.SelectObject(bitmap)

    # Use ctypes to call PrintWindow
    PW_RENDERFULLCONTENT = 2
    result = ctypes.windll.user32.PrintWindow(hwnd, save_dc.GetSafeHdc(), PW_RENDERFULLCONTENT)

    if result == 1:
        # Save the bitmap to a file
        bitmap.SaveBitmapFile(save_dc, 'screenshot.bmp')

        # Convert the bitmap to a PIL image and save as PNG
        bmp_info = bitmap.GetInfo()
        bmp_str = bitmap.GetBitmapBits(True)
        img = Image.frombuffer(
            'RGB',
            (bmp_info['bmWidth'], bmp_info['bmHeight']),
            bmp_str, 'raw', 'BGRX', 0, 1
        )
        img.save('screenshot.png')

        # Display the screenshot
        img.show()
    else:
        print("Failed to capture the window content.")

    # Clean up
    win32gui.DeleteObject(bitmap.GetHandle())
    save_dc.DeleteDC()
    mfc_dc.DeleteDC()
    win32gui.ReleaseDC(hwnd, window_dc)
else:
    print(f"No window found with title: {window_title}") 

I have a hard time figuring out what the difference between your C implementation and the above implementation is, but maybe this could help figuring out the reason for the black screens?

[nulano]

The difference is almost certainly the use of the PrintWindow function instead of BitBlt. But I cannot easily say how compatible that is with various programs IIUC BitBlt copies the (already painted) window data from the window manager buffer, and PrintWindow sends a message to the window requesting that it paint itself into a provided buffer; or at least that is what the documentation suggests, but it is possible that it is slightly outdated by now.

[radarhere]

The screenshot is just a black screen.

The black screen is the right size for the window though, yes?

Something that may or may not be related to your initial report that Windows 10 and Windows 11 behave differently - https://stackoverflow.com/a/54572219/4093019

this only occurs with the Windows update 1809 from late 2018. Apparently, Windows changed the way it handles clipping with that update so that the Device Context contents are no longer updated for parts of windows that are located offscreen.

But I expect you'll tell me that the window you're trying to capture is always entirely on-screen?

[nulano]

But I expect you'll tell me that the window you're trying to capture is always entirely on-screen?

That was what I observed, yes. A black box of the correct size while capturing a window that is visible and focused.

[radarhere]

If both BitBlt and PrintWindow are flawed, then my next idea would be for Pillow to run both when capturing a window - if one is non-black, then return that to the user. Otherwise just return the BitBlt image (it is possible the window is actually black).

Does that sound like a good idea, or it is too expensive/convoluted?

[radarhere]

Based on the lack of response, I'm going close this. #8516 is still open to resolve the generic scenario of #4415, but I don't think we can test the specifics of this issue without the OP verifying.