Fix get_process_window to return main window instead of arbitrary window handle

Open Copilot opened this issue 5 months ago • 0 comments

Problem

The get_process_window function was returning arbitrary window handles belonging to the current process instead of specifically returning the main application window. This caused issues when:

A process has multiple top-level windows (e.g., main window + console window)
A process has child controls or dialogs
A DLL is injected and creates additional windows (like a terminal for debugging)

This was particularly problematic for DLL injection scenarios where spawning a terminal on Windows would cause the function to potentially return the console window handle instead of the main application window handle needed for DirectX hooking.

Solution

Enhanced the window filtering logic to properly identify main windows by:

Checking for window owners: Windows that have an owner (checked via GetWindow(hwnd, GW_OWNER)) are child dialogs or controls, not main windows. These are now skipped.
Checking visibility: Invisible windows are typically message-only windows or hidden system windows. Only visible windows are now considered as potential main windows.
Code quality improvements:
- Changed HWND(0) to HWND::default() for better clarity
- Changed output.0 == 0 to output.is_invalid() for better readability
- Simplified error handling

Changes

Added imports: GetWindow, IsWindowVisible, GW_OWNER
Added owner check to filter out child windows and dialogs
Added visibility check to filter out hidden windows
Improved HWND initialization and validation

This ensures the function returns the actual main application window, which is critical for render engine function discovery and hooking.

Fixes the issue described in the problem statement and provides more robust behavior for DirectX9/10/11/12 method discovery when used in DLL injection scenarios.

Original prompt

This section details on the original issue you should resolve

<issue_title>get_process_window not working correctly / get_process_window returns arbitrary window handle instead of main window</issue_title> <issue_description>I expect the get_process_window function to return the main window handle of the current process, not just any window handle belonging to the current process.

The current implementation of get_process_window does not work correctly if a process contains multiple windows or controls, as it may return a handle to a child control instead of the main application window.
pub(crate) fn get_process_window() -> Option<HWND> {
    extern "system" fn enum_windows_callback(hwnd: HWND, l_param: LPARAM) -> BOOL {
        let mut wnd_proc_id: u32 = 0;
        unsafe {
            GetWindowThreadProcessId(hwnd, Some(&mut wnd_proc_id));
            if std::process::id() != wnd_proc_id {
                return true.into();
            }
            // windows has an owner => not a main window
            if GetWindow(hwnd, GW_OWNER).is_ok() {
                return true.into();
            }

            *(l_param.0 as *mut HWND) = hwnd;
        }
        false.into()
    }

    let mut output: HWND = HWND::default();
    unsafe {
        let _ = EnumWindows(
            Some(enum_windows_callback),
            std::mem::transmute::<_, LPARAM>(&mut output as *mut HWND),
        );
    };

    match output.is_invalid() {
        true => None,
        false => Some(output),
    }
}
```</issue_description>

## Comments on the Issue (you are @copilot in this section)

<comments>
<comment_new><author>@ohchase</author><body>
Thank you very much! I'm familiar with this issue. This will be a pain point for those who want to spawn a terminal on windows when injecting a DLL. Your suggested fix looks ideal, I can work on this during the weekend. If you'd like pull requests are welcome as well! 
</body></comment_new>
</comments>

Fixes ohchase/shroud#12

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

Oct 02 '25 23:10 Copilot