Visual Studio Crash When Invoking NativeMethods.SetWindowPos Method

[Malini-SF4235]

Environment

Visual Studio professional 2026 - version 18.0.0

.NET version

Facing this issue in .NetCore

Did this work in a previous version of Visual Studio and/or previous .NET release?

No

Issue Description

I have a Boolean property named NativeCheck. During the property processing, the SetWindowPos method is invoked. Visual Studio crashes when this method is executed. Below is the method definition: [DllImport("USER32.dll"); internal static extern bool SetWindowPos(IntPtr hWnd, IntPtr hWndInsertAfter, int X, int Y, int cx, int cy, int uFlags)

Demo Link

CustomControl_Demo.zip

Issue replication video

https://github.com/user-attachments/assets/884c2559-dc96-4e7c-b6f8-66788485120b

Steps to reproduce

1. Open the Designer.
2. Select the CustomButton and navigate to the Properties window.
3. Set the NativeCheck property to True.

Expected Behavior: The value should be properly set to the NativeCheck property without any issues. Observed Behavior: When setting the NativeCheck property to True, Visual Studio crashes.

[SimonZhao888]

@Malini-SF4235 I think you are missing some details with your custom button, the private method ShowToolTip() will pop up a native window, but this behavior needs to occur at runtime, not the in the design mode, because in designer mode, handles to controls may not have been created or may not be available, causing API calls to fail or block. Maybe you need update the following codes:

[Browsable(true)]
public bool NativeCheck
{
    get { return nativeCheck; }
    set
    {
        if (value && !DesignMode)
            ShowToolTip();
        nativeCheck = value;
    }
}

Hope this can solve your problem.

[Malini-SF4235]

@SimonZhao888 - Thank you for your suggestion! I understand your point regarding the ShowToolTip() method and its behavior in design mode. You mentioned that the native window pop-up should only occur at runtime because, in design mode, control handles may not be created, which can cause API calls to fail or block.

However, I have a doubt: in the .NET Framework version, this issue did not occur even though the same native methods were called. I have attached a framework sample for your reference.

Could you please clarify why this difference exists between .NET Framework and .NET Core? Is there any specific change in the design-time behavior or handle creation process in .NET Core that causes this issue?

Your insights would be helpful in understanding the scenario. We look forward to your response.

CustomControl_Framework_Sample.zip

[SimonZhao888]

@Malini-SF4235 - In .NET Framework WinForms the designer loads your control inside the main Visual Studio process and most native User32 calls (CreateWindowEx via CreateHandle, SetWindowPos, etc.) succeed early, even during property setters invoked at design-time. The parent window hierarchy is already established enough that creating auxiliary tooltip windows rarely fails. Design-time instantiation is more permissive and the timing of Site assignment vs. handle creation is slightly different. In .NET (Core / 5+), the WinForms designer was reworked (out‑of‑process / isolated). Differences that make a design-mode check necessary for code like your ShowToolTip:

1. Timing: Your property setter can run before the control’s underlying window handle exists. Forcing CreateHandle for a secondary native window (tooltip) may fail or throw in the designer, or leak resources.
2. Reliability of Control.DesignMode: In constructors and early property setters DesignMode is often false because Site is not yet assigned. You need a more robust composite check.
3. Native interop restrictions: Some P/Invoke calls during design-time in .NET can hit security / threading / parenting issues (invalid parent HWND, z-order issues).
4. Resource leaks: Repeated reloads of the designer (code edits) can accumulate undisposed native windows if created unconditionally.
5. Separation: Out-of-process designer reduces guarantees that a created tooltip window will behave; it may float oddly or crash the designer.

[dotnet-policy-service[bot]]

This submission has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 14 days.

It will be closed if no further activity occurs within 7 days of this comment.