Allow for capturing obscured windows

[BanditTech]

General information:

• OS name: Windows
• OS version: 10
• OS architecture: 64 bits
• Resolutions:
  • Monitor 1: 1920x1080
  • Monitor 2: 1920x1080
• Python version: 3.8.3
• MSS version: 6.0.0

Description of the warning/error

This is not related to an error message. This has to do with capturing obscured windows.

Other details

In order to capture an obscured window, a library I have been using in AHK which also uses dll capture has figured out how to do it. I understand that this is not written in python or c, but this code may be an assistance in understanding the steps he has done in order to utilize the GetDCEx call.

Essentially he does what seems to be two captures, which is not ideal, but it works. I was hoping that it would be possible to collaborate on implementing this feature into your library, or if you would rather use this code as a starting point:

Bind Window function to set and remember the window

Some of the code here is not really necessary. Mostly it is just having a static value saved in the class which can be assigned to a window handle. When performing further screenshots, this handle will be used the get the DC instead of the screen.

BindWindow(window_id:=0, set_exstyle:=0, get:=0)
{
  static id, old, Ptr:=A_PtrSize ? "UPtr" : "UInt"
  if (get)
    return, id
  if (window_id)
  {
    id:=window_id, old:=0
    if (set_exstyle)
    {
      WinGet, old, ExStyle, ahk_id %id%
      WinSet, Transparent, 255, ahk_id %id%
      Loop, 30
      {
      Sleep, 100
      WinGet, i, Transparent, ahk_id %id%
      }
      Until (i=255)
    }
  }
  else
  {
    if (old)
      WinSet, ExStyle, %old%, ahk_id %id%
    id:=old:=0
  }
}

Capturing the screenshot, determining if a window handle is present and using that instead

if (hBM) and !(w<1 or h<1)
{
  win:=DllCall("GetDesktopWindow", Ptr)
  hDC:=DllCall("GetWindowDC", Ptr,win, Ptr)
  mDC:=DllCall("CreateCompatibleDC", Ptr,hDC, Ptr)
  oBM:=DllCall("SelectObject", Ptr,mDC, Ptr,hBM, Ptr)
  DllCall("BitBlt",Ptr,mDC,"int",x-zx,"int",y-zy,"int",w,"int",h
    , Ptr,hDC, "int",x, "int",y, "uint",0x00CC0020) ; |0x40000000)
  DllCall("ReleaseDC", Ptr,win, Ptr,hDC)
  if (id:=BindWindow(0,0,1))
    WinGet, id, ID, ahk_id %id%
  if (id)
  {
    WinGetPos, wx, wy, ww, wh, ahk_id %id%
    left:=x, right:=x+w-1, up:=y, down:=y+h-1
    left:=left<wx ? wx:left, right:=right>wx+ww-1 ? wx+ww-1:right
    up:=up<wy ? wy:up, down:=down>wy+wh-1 ? wy+wh-1:down
    x:=left, y:=up, w:=right-left+1, h:=down-up+1
  }
  if (id) and !(w<1 or h<1)
  {
    hDC2:=DllCall("GetDCEx", Ptr,id, Ptr,0, "int",3, Ptr)
    DllCall("BitBlt",Ptr,mDC,"int",x-zx,"int",y-zy,"int",w,"int",h
    , Ptr,hDC2, "int",x-wx, "int",y-wy, "uint",0x00CC0020) ; |0x40000000)
    DllCall("ReleaseDC", Ptr,id, Ptr,hDC2)
  }
  DllCall("SelectObject", Ptr,mDC, Ptr,oBM)
  DllCall("DeleteDC", Ptr,mDC)
}

Upvote & Fund

• We're using Polar.sh so you can upvote and help fund this issue.
• We receive the funding once the issue is completed & confirmed by you.
• Thank you in advance for helping prioritize & fund our backlog.

[BanditTech]

        self._cfactory(
            attr=self.user32, func="GetDCEx", argtypes=[HWND, HRGN, DWORD], restype=HDC
        )

So it seems the dll also will allow for returning the window handle as well, so you do not need any further dependencies:

FindWindow FindWindowEx ...

import win32gui

win2find = input('enter name of window to find')
whnd = win32gui.FindWindowEx(None, None, None, win2find)
if not (whnd == 0):
  print('FOUND!')

self.user32.GetDCEx(self._get_bound_capture_window(),0,3)

[TrInsanity]

This is definitely possible to achieve, just by changing the following line of _get_srcdc within windows.py:

        srcdc = MSS._srcdc_dict[cur_thread] = self.user32.GetWindowDC(0)

0 sets it to "fullscreen", but you can capture a specific window by passing in it's HWND.

In addition, you can change self.user32.GetWindowDC to self.user32.GetDC, which gets purely the client area of the window. See here for the difference.

I've hacked together a version that works for my purposes, with minimal modification, however it's probably not suitable for a pull request (I pass in the HWND when initialising mss).

This also would solve #158 , however this would be a Windows only solution.

[BoboTiG]

We could add 2 keyword arguments to the Windows class, something like window=0 and content_only=False.

WDYT? Better names in mind (knowing that they should be OS-agnostic)?

[BoboTiG]

The part dealing with the Windows handle retrieval should not be part of MSS. I would like to keep it simple and focused on screenshot stuff only.

[TrInsanity]

We could add 2 keyword arguments to the Windows class, something like window=0 and content_only=False.

WDYT? Better names in mind (knowing that they should be OS-agnostic)?

They sound suitable to me. I'm not sure of the implementation for Linux/MacOS, however OS-agnostic keywords make sense for future implementation.

The part dealing with the Windows handle retrieval should not be part of MSS. I would like to keep it simple and focused on screenshot stuff only.

Agreed, easy enough to acquire the HWND with win32gui.

[BoboTiG]

For now, let's focus on Windows only. Future implementations may come later for other OSes. Do you want to work on it @TrInsanity?

[TrInsanity]

@BoboTiG I've really got no experience with PRs & best practices etc. so wouldn't know where to start. I can give it a go but it'll likely need modification!

[BoboTiG]

Yeah, go ahead and I will help you :)

[BanditTech]

any update on this? :)

[lorcan2440]

Hi, was there any development on this? I need a solution which can capture hardware accelerated windows (mss already does this, solutions with win32ui don't work because it only has BitBlt) which are obscured by other windows.

[wchill]

I have the following which is working for some automation I'm doing. The window in question is obscured and also a game (thus hardware accelerated).

from PIL import Image

def capture_win_alt(convert: bool = False, window_name: Optional[str] = "MegaMan_BattleNetwork_LegacyCollection_Vol2"):
    # Adapted from https://stackoverflow.com/questions/19695214/screenshot-of-inactive-window-printwindow-win32gui
    global WIN_HANDLES

    from ctypes import windll

    import win32gui
    import win32ui

    if WIN_HANDLES is None:
        assert window_name is not None
        print("Acquiring window handle")
        windll.user32.SetProcessDPIAware()
        hwnd = win32gui.FindWindow(None, window_name)

        left, top, right, bottom = win32gui.GetClientRect(hwnd)
        w = right - left
        h = bottom - top
        print(f"Client rect: {left}, {top}, {right}, {bottom}")

        hwnd_dc = win32gui.GetWindowDC(hwnd)
        mfc_dc = win32ui.CreateDCFromHandle(hwnd_dc)
        save_dc = mfc_dc.CreateCompatibleDC()

        bitmap = win32ui.CreateBitmap()
        bitmap.CreateCompatibleBitmap(mfc_dc, w, h)

        WIN_HANDLES = (hwnd, hwnd_dc, mfc_dc, save_dc, bitmap)

    (hwnd, hwnd_dc, mfc_dc, save_dc, bitmap) = WIN_HANDLES
    save_dc.SelectObject(bitmap)

    # If Special K is running, this number is 3. If not, 1
    result = windll.user32.PrintWindow(hwnd, save_dc.GetSafeHdc(), 3)

    bmpinfo = bitmap.GetInfo()
    bmpstr = bitmap.GetBitmapBits(True)

    im = Image.frombuffer("RGB", (bmpinfo["bmWidth"], bmpinfo["bmHeight"]), bmpstr, "raw", "BGRX", 0, 1)

    if result != 1:
        win32gui.DeleteObject(bitmap.GetHandle())
        save_dc.DeleteDC()
        mfc_dc.DeleteDC()
        win32gui.ReleaseDC(hwnd, hwnd_dc)
        WIN_HANDLES = None
        raise RuntimeError(f"Unable to acquire screenshot! Result: {result}")

    open_cv_image = np.array(im)[:, :, ::-1].copy()
    return open_cv_image

I haven't checked the performance of this however. YMMV.

[lorcan2440]

@wchill Amazing, this worked, thank you so much! Been stuck on this for days. I'm going to make a few small changes for my purpose and post it here in case anyone in future finds it helpful.

• removed PIL.Image dependence
• removed unneeded copy of image array
• removed stray constant
• returned as contiguous array and drop alpha channel
• added utility viewing loop in OpenCV

import cv2
import numpy as np
from ctypes import windll
import win32gui
import win32ui


def capture_win_alt(window_name: str):
    # Adapted from https://stackoverflow.com/questions/19695214/screenshot-of-inactive-window-printwindow-win32gui

    windll.user32.SetProcessDPIAware()
    hwnd = win32gui.FindWindow(None, window_name)

    left, top, right, bottom = win32gui.GetClientRect(hwnd)
    w = right - left
    h = bottom - top

    hwnd_dc = win32gui.GetWindowDC(hwnd)
    mfc_dc = win32ui.CreateDCFromHandle(hwnd_dc)
    save_dc = mfc_dc.CreateCompatibleDC()
    bitmap = win32ui.CreateBitmap()
    bitmap.CreateCompatibleBitmap(mfc_dc, w, h)
    save_dc.SelectObject(bitmap)

    # If Special K is running, this number is 3. If not, 1
    result = windll.user32.PrintWindow(hwnd, save_dc.GetSafeHdc(), 3)

    bmpinfo = bitmap.GetInfo()
    bmpstr = bitmap.GetBitmapBits(True)

    img = np.frombuffer(bmpstr, dtype=np.uint8).reshape((bmpinfo["bmHeight"], bmpinfo["bmWidth"], 4))
    img = np.ascontiguousarray(img)[..., :-1]  # make image C_CONTIGUOUS and drop alpha channel

    if not result:  # result should be 1
        win32gui.DeleteObject(bitmap.GetHandle())
        save_dc.DeleteDC()
        mfc_dc.DeleteDC()
        win32gui.ReleaseDC(hwnd, hwnd_dc)
        raise RuntimeError(f"Unable to acquire screenshot! Result: {result}")

    return img


def main():

    WINDOW_NAME = "PS Remote Play"
    while cv2.waitKey(1) != ord('q'):
        screenshot = capture_win_alt(WINDOW_NAME)
        cv2.imshow('Computer Vision', screenshot)


if __name__ == '__main__':
    main()

[wchill]

@lorcan2440 make sure you don't leak handles. I stored them in a global variable for this reason.