Crash while opening VST instrument plugin UI

[pen-siv]

trafficstars

System Info:

• NixOS: building from source using the nix package definition recently merged into nixpkgs-unstable
• Wayland window backend (hyprland with XWayland enabled)

Problem: ossia-score crashes entirely when I click to open the plugin UI of any VST plugin that's been added to the timeline.

Running from a terminal, I get the following message printed at the moment of crash.

X Error of failed request:  BadWindow (invalid Window parameter)
  Major opcode of failed request:  1 (X_CreateWindow)
  Resource id in failed request:  0x4
  Serial number of failed request:  54
  Current serial number in output stream:  56

Being NixOS, I'm aware that this may be a misconfiguration on my part, but would appreciate advice, if so.

[jcelerier]

Hm, and I imagine this works in other VST hosts ? Which plugins did you try ? XCreateWindow failing may be would be caused by hyprland refusing to create it, I can try to install it on my end and see why it fails

[jcelerier]

hm, tried Hyprland and on ArchLinux it seems to work (even if it applies an absolutely awful UI scaling)

[jcelerier]

Would you be able to give a stack trace of where the crash happen ? maybe I can at least make it not crash in this case...

[pen-siv]

Yes, other hosts work as expected with VSTs.

This is the full console output while running ossia up until the crash. There's no stack trace exactly, but the 3 distinctly different sections of logging are from starting up, starting a new project, then adding Dexed VST to the timeline, then trying to open its UI.

QVariant::load: unknown user type with name TimeVal.
pd 0.55.0
bonk version 1.5
fiddle version 1.1 TEST4
pd~ version 0.54
pique 0.1 for PD version 23
qt.qml.typeregistration: Invalid QML element name "Type"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "Access"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "Bounding"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "Repetitions"; value type names should begin with a lowercase letter
qt.qml.typeregistration: Invalid QML element name "Duration"; value type names should begin with a lowercase letter
Debug: "https://raw.githubusercontent.com/ossia/score-addons/master/remote.json"   "404: Not Found" (:0)
Debug: Could not save device (:0)
Debug: Audio device emits changed: Dataflow::AudioDevice(0x55d473f59870) (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "states", "State.1", "comment")  -- last found:  states 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "states", "State.1", "label")  -- last found:  states 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "states", "State.1", "trigger")  -- last found:  states 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "syncs", "Sync.1", "comment")  -- last found:  syncs 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "syncs", "Sync.1", "label")  -- last found:  syncs 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "syncs", "Sync.1", "trigger")  -- last found:  syncs 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "events", "Event.1", "comment")  -- last found:  events 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "events", "Event.1", "label")  -- last found:  events 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "events", "Event.1", "expression")  -- last found:  events 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "states", "State.2", "comment")  -- last found:  states 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "states", "State.2", "label")  -- last found:  states 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "states", "State.2", "trigger")  -- last found:  states 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "intervals", "Interval.1", "comment")  -- last found:  intervals 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "intervals", "Interval.1", "label")  -- last found:  intervals 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "intervals", "Interval.1", "min")  -- last found:  intervals 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "intervals", "Interval.1", "max")  -- last found:  intervals 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "intervals", "Interval.1", "default")  -- last found:  intervals 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "intervals", "Interval.1", "percentage")  -- last found:  intervals 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "intervals", "Interval.1", "speed")  -- last found:  intervals 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "intervals", "Interval.1", "muted")  -- last found:  intervals 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "syncs", "Sync.2", "comment")  -- last found:  syncs 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "syncs", "Sync.2", "label")  -- last found:  syncs 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "syncs", "Sync.2", "trigger")  -- last found:  syncs 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "events", "Event.2", "comment")  -- last found:  events 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "events", "Event.2", "label")  -- last found:  events 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "events", "Event.2", "expression")  -- last found:  events 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "states", "State.3", "comment")  -- last found:  states 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "states", "State.3", "label")  -- last found:  states 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "states", "State.3", "trigger")  -- last found:  states 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "intervals", "Interval.2", "comment")  -- last found:  intervals 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "intervals", "Interval.2", "label")  -- last found:  intervals 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "intervals", "Interval.2", "min")  -- last found:  intervals 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "intervals", "Interval.2", "max")  -- last found:  intervals 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "intervals", "Interval.2", "default")  -- last found:  intervals 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "intervals", "Interval.2", "percentage")  -- last found:  intervals 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "intervals", "Interval.2", "speed")  -- last found:  intervals 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "intervals", "Interval.2", "muted")  -- last found:  intervals 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "intervals", "Interval.wand39", "processes", "Dexed", "comment")  -- last found:  intervals 
 (:0)
Debug: looking for QList("Untitled.hock27", "processes", "Scenario.1", "intervals", "Interval.wand39", "processes", "Dexed", "label")  -- last found:  intervals 
 (:0)
X Error of failed request:  BadWindow (invalid Window parameter)
  Major opcode of failed request:  1 (X_CreateWindow)
  Resource id in failed request:  0x6
  Serial number of failed request:  54
  Current serial number in output stream:  56

I'll do some trial-and-error with different configurations today and report here, too.

[jcelerier]

could you run ossia-score under gdb, and when the crash happens type "bt" and paste me what it says ? it will help pinpoint the error. from a terminal:

$ gdb ossia-score
... gdb starts ...
(gdb) r
... app launches ...
... crash happens ...
(gdb) bt

[pen-siv]

...
[New Thread 0x7fff8d3fe6c0 (LWP 16941)]
[New Thread 0x7fff8cbfd6c0 (LWP 16942)]
[Thread 0x7fff8d3fe6c0 (LWP 16941) exited]
[New Thread 0x7fff8d3fe6c0 (LWP 16943)]
X Error of failed request:  BadWindow (invalid Window parameter)
  Major opcode of failed request:  1 (X_CreateWindow)
  Resource id in failed request:  0x5
  Serial number of failed request:  54
  Current serial number in output stream:  56
[Thread 0x7fffc9ffb6c0 (LWP 16902) exited]
[Thread 0x7fffe11be6c0 (LWP 16876) exited]
[Thread 0x7fffe91ff6c0 (LWP 16869) exited]
[Thread 0x7fff8cbfd6c0 (LWP 16942) exited]
[Thread 0x7fff8dbff6c0 (LWP 16939) exited]
[Thread 0x7fff8effd6c0 (LWP 16932) exited]
[Thread 0x7fffa4db66c0 (LWP 16931) exited]
[Thread 0x7fff8ffff6c0 (LWP 16930) exited]
[Thread 0x7fff8f7fe6c0 (LWP 16929) exited]
[Thread 0x7fffa5db86c0 (LWP 16928) exited]
[Thread 0x7fffa65b96c0 (LWP 16919) exited]
[Thread 0x7fffa6dba6c0 (LWP 16918) exited]
[Thread 0x7fffa75bb6c0 (LWP 16917) exited]
[Thread 0x7fffc8ff96c0 (LWP 16907) exited]
[Thread 0x7fffc97fa6c0 (LWP 16906) exited]
[Thread 0x7fffca7fc6c0 (LWP 16901) exited]
[Thread 0x7fffcaffd6c0 (LWP 16900) exited]
[Thread 0x7fffcb7fe6c0 (LWP 16899) exited]
[Thread 0x7fffe09586c0 (LWP 16898) exited]
[Thread 0x7fffcbfff6c0 (LWP 16897) exited]
[Thread 0x7fffe19bf6c0 (LWP 16875) exited]
[Thread 0x7fffe21c06c0 (LWP 16874) exited]
[Thread 0x7fffe29c16c0 (LWP 16873) exited]
[Thread 0x7fffe31c26c0 (LWP 16872) exited]
[Thread 0x7fffe3fff6c0 (LWP 16871) exited]
[Thread 0x7fffe89726c0 (LWP 16870) exited]
[Thread 0x7fffe96c52c0 (LWP 16852) exited]
[Thread 0x7fff8d3fe6c0 (LWP 16943) exited]
[New process 16852]
[Inferior 1 (process 16852) exited with code 01]
(gdb) bt
No stack.

[jcelerier]

hmm, I see that this call is happening inside the pug-ins - on my side I just do the following:

ERect* vstRect{};
fx.dispatcher(&fx, effEditGetRect, 0, 0, &vstRect, 0.f);
fx.dispatcher(&fx, effEditOpen, 0, 0, (void*)winId(), 0); // the crash happens here which is a call to within the VST

[pen-siv]

Yeah, it is curious that only VSTs lead to this issue. I tried running without xwayland to see what other issues might occur.

In this case (no xwayland), the program crashes when I try to add VSTs to the timeline. Other processes attach just fine, but when adding a VST the app closes with the message: Failed to connect to the X Server. (Not unsurprising, but special)

[pen-siv]

Further explorations:

running ossia-score with an invalid $WAYLAND_DISPLAY (WAYLAND_DISPLAY=invalid ossia-score)

• new message line printed in the console along with the usual messages: qt.qpa.plugin: Could not load the Qt platform plugin "wayland" in "" even though it was found.
• Program is functional, but I cannot click-and-drag Processes onto the timeline. Other click-drag operations, like branching the timeline, work fine.
• However, if I recover or load a project which already has a VST attached, then the UI will load without crash

running ossia-score with QT_QPA_PLATFORM=xcb ossia-score

• acts the same as invalid $WAYLAND_DISPLAY as above, but without the qt.qpa.plugin message

running from AppImage (appimage-run ossia.score-3.2.4-linux-amd64.AppImage)

• acts the same as invalid $WAYLAND_DISPLAY as above, but with the following additional messages

[2024-07-09 10:22:37.813] [ossia] [error] No NDI library found on the system.
Please install an NDI library, for instance https://code.videolan.org/jbk/libndi
UltraLeap not found. Make sure LeapC.dll / dylib is in the right path.
...
Player: listening on  40591
...
Warning: Numeric mode unsupported in the posix collation implementation (:0)

[jcelerier]

okay, I can reproduce the crash locally, as well as the drag'n'drop not working with Hyprland. Thanks for the investigation!

ossia built against ArchLinux Qt libraries :

• QT_QPA_PLATFORM=xcb : drag'n'drop works
• QT_QPA_PLATFORM=xcb : VST gui works, no crash
• QT_QPA_PLATFORM=xcb : mouse cursor scales badly when going over the open vst icon
• QT_QPA_PLATFORM=xcb : absolutely terrible automatic scaling by the compositor
• QT_QPA_PLATFORM=wayland : drag'n'drop works
• QT_QPA_PLATFORM=wayland : vst crash with the XCreateWindow error.
  • This is caused by X11 not managing to create the window (makes sense as we're running under wayland), which calls XDefaultError which seems to call exit on the whole program, without leaving any chance for recovery.
• QT_QPA_PLATFORM=wayland : 2x scaling for no good reason applied by the compositor but at least it's crisp, not pixelated

ossia built against Nix Qt libraries :

• QT_QPA_PLATFORM=xcb : drag'n'drop fails
• QT_QPA_PLATFORM=xcb : VST gui works, no crash
• QT_QPA_PLATFORM=xcb : mouse cursor scales correctly
• QT_QPA_PLATFORM=xcb : absolutely terrible automatic scaling by the compositor
• QT_QPA_PLATFORM=wayland : drag'n'drop works
• QT_QPA_PLATFORM=wayland : vst crash with the XCreateWindow error - same as above, expected

[jcelerier]

Drag'n'drop seems to be broken in Hyprland : https://github.com/hyprwm/Hyprland/issues/1083 Seems to be a solution given here : https://github.com/hyprwm/Hyprland/issues/1969

[jcelerier]

Note that VST2 / VST3 plug-ins GUIs will only ever work with QT_QPA_PLATFORM=xcb, not wayland as the plug-ins themselves expect a X11 host.

[jcelerier]

I will add some checking code that will disable the plug-in GUI button if we happen to be under wayland as there's no way to make this work short of running the entire VST in an external process (with the expected performance cost and implementation complexity)

[pen-siv]

Drag'n'drop seems to be broken in Hyprland : hyprwm/Hyprland#1083 Seems to be a solution given here : hyprwm/Hyprland#1969

I wonder if this also relates to intermittent dragging issues that I have in Bitwig with hyprland which I didn't have in sway. I'll try the suggested solution. If nothing works ideally, I'll try in sway.

Platform xcb understood. My session-wide $QT_QPA_PLATFORM is actually set to QT_QPA_PLATFORM=wayland;xcb but ossia-score doesn't seem to recognize that as a fallback for the VST.

[pen-siv]

Btw, I don't have any scaling issues. Using a standard HD monitor (external) and built-in HiDPI (Steam Deck).

[jcelerier]

I don't think any fallback is possible, you have to pass a pointer to the host X11 display when you create a vst. Or if it is possible in any way, I'm very interested in the code that allows it :)

I found this recent bugfix in Qt : https://codereview.qt-project.org/c/qt/qtbase/+/563851 which should fix drag'n'drop. I will update the Qt version score builds against soon.

[pen-siv]

My experience with C++ is extremely limited, but if I do find a workaround I'll let you know.

[jcelerier]

it would be worth seeing how e.g. qtractor and lmms do it as they both are Qt-based apps able to load plug-ins

[jcelerier]

ikewise for Carla

[pen-siv]

Well, I loaded Carla in LMMS and was able to open the UI for Dexed there with no issue.

[pen-siv]

Seems like Carla sets the env var explicitly to support the action https://github.com/falkTX/Carla/blob/main/source/frontend/carla-plugin#L606

This could probably be handled within the nix package without changing any of the app code.

[jcelerier]

well, that's what I do in ossia too.. so I don't know why you ended up having ossia start with wayland:

https://github.com/ossia/score/blob/e0676c9fe340df3975ccde2a098cfa15610fff3f/src/app/main.cpp#L116

this function forces QT_QPA_PLATFORM to xcb unless the user did set it explicitly : https://github.com/ossia/score/blob/e0676c9fe340df3975ccde2a098cfa15610fff3f/src/app/main.cpp#L148

[pen-siv]

Ah, yeah, this variable is set for my session, so it wouldn't override it. I'm rebuilding with the following added to the nix pkg I have locally

  postInstall = ''
    wrapProgram "$out/bin/ossia-score" --set QT_QPA_PLATFORM "xcb"
  '';

This should let me run it no matter what I have set for the rest of the session.

[pen-siv]

I'm happy to report that adding the postInstall worked for ensuring the proper platform, and that adding windowrulev2 = "noinitialfocus,xwayland:1" to my hyprland config seems to have fixed the dragging issues in both ossia-score and Bitwig!

Thank you very much for digging into this with me. I can make a PR to the nix package if you think that's worth it, too.

for any future searchers: the hyprland mouse fix is declared like so if you're using home-manager:

users.*.wayland.windowManager.hyprland.settings.windowrulev2 = [ "noinitialfocus,xwayland:1" ];

to force noinitialfocus for xwayland apps.

[jcelerier]

great :) a PR to the nix package would be awesome !

[jcelerier]

on ossia side I added detection of wayland platform plug-in to disable the VST / VST3 UI so that at least it won't cause crashes to the user

[pen-siv]

Sorry I never put in a PR since I figured that just wrapping the output works well enough for my specific case--and, I ended up dropping hyprland in-favor of Gnome anyhow :)

Closing with one last example for any future issue-havers:

ossia-score = (pkgs.symlinkJoin {
  name = "ossia-score";
  paths = [ pkgs.ossia-score ];
  buildInputs = [ pkgs.makeWrapper ];
  postBuild = ''
    wrapProgram $out/bin/ossia-score \
    --set QT_QPA_PLATFORM "xcb" \
    --set GDK_BACKEND "x11"
  '';
});

Thanks again.

[jcelerier]

great, I'll add the GDK_BACKEND env. var. to the scripts here too I didn't know about it and I think it could solve a few problems I've seen