qubes-issues qubes.WaitForSession processes keep accumulating with preloaded DispVMs

Qubes OS release

r4.3

Brief summary

qubes.WaitForSession processes keep accumulating with preloaded DispVMs, consuming CPU cycles quite noticeably after a point.

Steps to reproduce

Launch and shut down many preloaded DispVMs in succession.

Expected behavior

At most N WaitForSession processes, where N is the amount of paused DispVMs.

Actual behavior

It seems that in the case of my system (in-place upgraded from r4.2), qubes.WaitForSession processes just never die after preloaded dispVMs are created and then shut down.

user@dom0 ~ % pgrep -f qubes.WaitForSession | wc -l
49

user@dom0 ~ % ps aux | grep WaitForSession
user        6265  0.2  0.0 231068  3624 ?        S    Nov30   2:23 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user        6732  0.3  0.0 231068  3556 ?        S    Nov30   2:26 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user        6863  0.2  0.0 231068  3704 ?        S    Nov30   2:21 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user        9097  0.3  0.0 231068  3548 ?        S    Nov30   2:26 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user       10856  0.3  0.0 231068  3504 ?        S    Nov30   2:27 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user       19430  0.2  0.0 231068  3592 ?        S    Nov30   2:22 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user       91126  0.3  0.0 231068  3612 ?        S    00:00   2:15 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user      162332  0.2  0.0 231068  3700 ?        S    01:00   1:57 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user      240638  0.2  0.0 231068  3616 ?        S    02:00   1:47 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user      325954  0.2  0.0 231068  3744 ?        S    03:00   1:35 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user      336899  0.2  0.0 231068  3556 ?        S    03:07   1:36 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user      424706  0.2  0.0 231068  3616 ?        S    04:00   1:24 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user      517814  0.2  0.0 231068  3596 ?        S    04:52   1:13 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user      519546  0.2  0.0 231068  3524 ?        S    04:53   1:14 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user      520134  0.2  0.0 231068  3532 ?        S    04:53   1:13 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user      520541  0.2  0.0 231068  3428 ?        S    04:53   1:12 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user      521147  0.2  0.0 231068  3752 ?        S    04:54   1:13 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user      523254  0.2  0.0 231068  3616 ?        S    04:54   1:12 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user      524657  0.2  0.0 231068  3692 ?        S    04:55   1:12 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user      536324  0.2  0.0 231068  3592 ?        S    05:00   1:11 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user      699367  0.2  0.0 231068  3484 ?        S    06:00   0:59 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user      869480  0.2  0.0 231068  3616 ?        S    07:00   0:47 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user     1046521  0.2  0.0 231068  3576 ?        S    08:00   0:34 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user     1230678  0.1  0.0 231068  3544 ?        S    09:00   0:23 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user     1370061  0.1  0.0 231068  3516 ?        S    09:43   0:14 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user     1379141  0.1  0.0 231068  3752 ?        S    09:45   0:14 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user     1386614  0.1  0.0 231068  3540 ?        S    09:47   0:14 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user     1440324  0.1  0.0 231068  3492 ?        S    10:00   0:12 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user     1457882  0.1  0.0 231068  3472 ?        S    10:04   0:12 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user     1459206  0.1  0.0 231068  3612 ?        S    10:04   0:12 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user     1460821  0.1  0.0 231068  3648 ?        S    10:04   0:12 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user     1574371  0.1  0.0 231068  3648 ?        S    10:28   0:09 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user     1577553  0.1  0.0 231068  3756 ?        S    10:29   0:10 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user     1598899  0.1  0.0 231068  3596 ?        S    10:33   0:09 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user     1698953  0.1  0.0 231068  3420 ?        S    10:53   0:08 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user     1703588  0.1  0.0 231068  3660 ?        S    10:54   0:07 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user     1705115  0.1  0.0 231068  3620 ?        S    10:54   0:07 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user     1706148  0.1  0.0 231068  3680 ?        S    10:54   0:07 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user     1707542  0.1  0.0 231068  3516 ?        S    10:55   0:07 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user     1712583  0.1  0.0 231068  3648 ?        S    10:55   0:07 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user     1715736  0.1  0.0 231068  3652 ?        S    10:56   0:07 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user     1735875  0.1  0.0 231068  3612 ?        S    11:00   0:07 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user     1844016  0.1  0.0 231068  3504 ?        S    11:18   0:06 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user     1979269  0.1  0.0 231068  3656 ?        S    11:40   0:03 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user     1980881  0.1  0.0 231068  3456 ?        S    11:40   0:03 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user     2026380  0.1  0.0 231068  3648 ?        S    11:47   0:02 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user     2035088  0.1  0.0 231068  3664 ?        S    11:49   0:02 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user     2037280  0.1  0.0 231068  3596 ?        S    11:49   0:02 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession
user     2104307  0.1  0.0 231068  3776 ?        S    12:00   0:02 /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession

After launching and shutting down 2 more:

user@dom0 ~ % pgrep -f qubes.WaitForSession | wc -l
51

Additional information

CPU usage difference after killing all the processes (which also screws up the rest of the preloaded DispVMs, sadly):

Dropped from around 15% per-core at 50 processes.

Dec 01 '25 11:12 Atrate

@ben-grande

Dec 01 '25 11:12 Atrate

Launch and shut down many preloaded DispVMs in succession.

Can you do the following please:

Reboot
Launch and shut down many preloaded DispVMs in succession. - (please specify how you are doing this, qvm-run via app menu?)
pgrep -f qubes.WaitForSession | wc -l

Because I cannot see that service lingering on my machine.

At most N WaitForSession processes, where N is the amount of paused DispVMs.

This service is not called to preload disposables. It is called by qvm-run.

Dec 01 '25 11:12 ben-grande

Once I have time to do a reboot, I'll report back. As far as my launching workflow is concerned, I use the preloading service and, when needed, launch and stop VMs using these i3 keybinds:

Launching an app using a preloaded dispvm:

bindsym i exec "qvm-run -q -a --service --dispvm=default-dvm -- qubes.StartApp+Alacritty"

Shutting down a VM:

bindsym $mod+$sup+Shift+Ctrl+q exec --no-startup-id qvm-shutdown -- "$(xdotool getactivewindow getwindowclassname | cut -d ':' -f1)"

Dec 01 '25 12:12 Atrate

This service is not called to preload disposables.

Actually, it might. With https://github.com/QubesOS/qubes-notification-proxy/pull/13, (autostarted) notification proxy will trigger qubes.WaitForSession on dom0 (note the qubes.WaitForSession here is running in dom0, not VM).

Dec 01 '25 13:12 marmarek

pgrep -f qubes.WaitForSession | xargs -rL1 pstree -Ulspa | grep --color=auto qrexec-daemon -e WaitFor -e disp

There are more than one per disposable since https://github.com/QubesOS/qubes-notification-proxy/pull/13. It seems to be related to it failing to start qubes-notification-agent before the preload can be used. It seems to linger until it can start qubes-notification-agent.

Dec 01 '25 15:12 ben-grande

It seems to linger until it can start qubes-notification-agent.

It might be related to the domain pausing and it can't end the connection. When I manually unpause a preloaded disposable, the processes ends.

Dec 01 '25 16:12 ben-grande

If I used the following patch:

https://github.com/QubesOS/qubes-core-admin/pull/757

Therefore, it runs on the disposable:

await self.run_service_for_stdio("qubes.WaitForSession")

Which does complete (successfully and fast if I comment wait-for-session on dom0 /etc/qubes/rpc-config/qubes.Notifications) -> https://github.com/QubesOS/qubes-notification-proxy/pull/13.

But on dom0, I still see a process /usr/bin/bash /etc/qubes-rpc/qubes.WaitForSession, possibly qubes-core-qrexec/libqrexec/exec.c exec_wait_for_session() being called somewhere, but also lingering as there is no guid file.

Dec 01 '25 16:12 ben-grande

I don't really know what changed, but after updating to r4.3-rc4 I haven't been able to reproduce this issue yet.

Dec 07 '25 09:12 Atrate

Temporary fix:

https://github.com/QubesOS/qubes-notification-proxy/commit/290cb4e4f3a77e77b0ce2645a82f486042a267eb

On Sun, Dec 7, 2025, 10:53 AM Atrate @.***> wrote:

Atrate left a comment (QubesOS/qubes-issues#10443) https://github.com/QubesOS/qubes-issues/issues/10443#issuecomment-3621871678

I don't really know what changed, but after updating to r4.3-rc4 I haven't been able to reproduce this issue yet.

— Reply to this email directly, view it on GitHub https://github.com/QubesOS/qubes-issues/issues/10443#issuecomment-3621871678, or unsubscribe https://github.com/notifications/unsubscribe-auth/BCE2O4LWZMR5SDVC5KUN6MT4AP2KZAVCNFSM6AAAAACNVIVWTSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTMMRRHA3TCNRXHA . You are receiving this because you were assigned.Message ID: @.***>

Dec 07 '25 11:12 ben-grande