Xpra-org
Long sessions get very slow
Describe the bug Xpra gets really slow on a session that has been running for a few weeks.
To Reproduce Steps to reproduce the behavior:
XPRA_USE_NATIVE_TRAY=0 xpra start ssh://[email protected]/ --ssh=ssh --tray=yes --dpi=96 --border=red,2 --opengl=no --start=/usr/bin/xfce4-terminal- Work on the session (heavily) every day for three weeks.
System Information (please complete the following information):
- Server OS: Ubuntu 22.04
- Client OS: Xubuntu 24.04
- Xpra Server Version 6.2.4
- Xpra Client Version 6.2.4
Additional context It seems that there is a memory leak, as much as you can have a memory leak in Python. Sessions start at about 500mb, and by the three-week mark get to 4gb.
as much as you can have a memory leak in Python
Most of the codecs are Cython glued to assembly.
by the three-week mark get to 4gb.
Please include xpra info after 3 weeks.
Which side has the memory leak? Client or server?
I wish I had known about that command. I have just restarted my session because it was too slow. I will comment with the output of that command when it gets slow again.
@molecularentropy please see the other edited questions.
If your connection is local, you could run with --encodings=rgb,webp to limit the codecs to the bare minimum.
I've also had this issue recently with the same symptoms (unusual memory usage, session slow to the point that you can watch typed characters appear one by one), so I assume it's the same issue. Run time was something over 6 weeks.
Note the affected Xpra server was running a beta version (v6.4-r38239), on Fedora 40. Here's a screenshot showing top on the server:
As a temporary fix, I restarted and replaced the session:
/usr/bin/xpra upgrade :10
Yep. I'm almost there. My session is already slow. Going to wait another week for it to get unbearable and then I will supply both the pre and post xpra info output
Okay, the session is now unbearably slow.
@totaam, can you give me a list of things you need from that log? The only reason is that there is a lot of data in that output, and some of it is sensitive.
Anything that shows memory usage, threads, all the windows, etc Process list, with shared memory usage and total. That sort of thing.
Let me know if you need anything more, as I will have to restart the session very soon, as it's practically unusable now.
Yep, that's totally fine, I'll have to restart tomorrow morning, so about 12 hours from now.
Okay, have to restart. I guess any further information will have to wait until the new session gets slow again. Some observations by me:
Delay has significantly increased:
It is not possible for there to be bandwidth contention. The cable between the two machines is direct (no switches or anything), and Xpra is the only connection through the cable.
Not sure what load() is, but it's significantly higher:
Here's the time it takes to get slow:
You're right, this does look like a memory leak, it explains the load average as the kernel ends up swapping like crazy.
You should be able to see with top when that happens.
The session has seen at least 2448 windows:
$ grep class-instance xpra_session_slow_sanitised.txt
windows.1.class-instance=('xfce4-terminal', 'Xfce4-terminal')
windows.6.class-instance=('emacs', 'Emacs')
windows.40.class-instance=('VirtualBox Manager', 'VirtualBox Manager')
windows.979.class-instance=('Navigator', 'firefox_firefox')
windows.981.class-instance=('Navigator', 'firefox_firefox')
windows.1405.class-instance=('firefox_firefox', 'firefox_firefox')
windows.1448.class-instance=('emacs', 'Emacs')
windows.1661.class-instance=('zoom', 'zoom')
windows.2448.class-instance=('zoom', 'zoom')
I don't think that this is a huge number, but I guess it's possible that each window consumes memory somehow. (v6.4 will completely drop Gtk - which will remove one major potential source of leaks)
There's no way of knowing which encoding was used for the windows that have already been closed. But for the ones still active, we have:
$ grep window xpra_session_slow_sanitised.txt | grep total_frames
client.window.1.total_frames.jpega=36
client.window.1.total_frames.scroll=1
client.window.1.total_frames.webp=94
client.window.6.total_frames.rgb24=62051
client.window.6.total_frames.scroll=1111
client.window.6.total_frames.webp=25855
client.window.40.total_frames.webp=3
client.window.979.total_frames.jpeg=13
client.window.979.total_frames.rgb24=7899
client.window.979.total_frames.scroll=306
client.window.979.total_frames.webp=20481
client.window.981.total_frames.rgb24=24
client.window.981.total_frames.scroll=5
client.window.981.total_frames.webp=113
client.window.1405.total_frames.rgb24=1
client.window.2448.total_frames.jpega=103
client.window.2448.total_frames.rgb24=9359
client.window.2448.total_frames.scroll=132
client.window.2448.total_frames.webp=5370
Again, these numbers aren't particularly high - I would expect to reach 60K frames in under an hour if the framerate is high enough.
I am somewhat suprised not to see a video codec in that list - why is that?
Also, scroll encoding is disabled in newer versions: #4596
Maybe you can run your next session with --encodings=all,-webp to eliminate this codec?
If your new session is already running, run this on your server (ie: for DISPLAY=:10 here):
xpra exit :10
xpra start --use-display=yes --encodings=all,-webp,-scroll :10
Then re-connect.
And please try a more up-to-date version - ie: 6.3.2 or even 6.4 beta.
You should be able to see with top when that happens.
Yes, in general Xpra consumes more memory and CPU as it goes.
I am somewhat suprised not to see a video codec in that list - why is that?
I didn't remove that from the log. I can have a look for you again in the unsanitised version if you give me the key you're looking for, but that doesn't sound like anything I would have removed.
Maybe you can run your next session with --encodings=all,-webp to eliminate this codec? And please try a more up-to-date version - ie: 6.3.2 or even 6.4 beta.
I will try all of these in the coming weeks, but given this takes quite a while to test, I'll gather up the findings that I can and put them up here when I can confirm.
that doesn't sound like anything I would have removed.
If not removed then perhaps not installed to begin with?
What xpra packages are installed? xpra-codecs?
I see all the video codecs when I run xpra encoding, with more colourspace details when running xpra video.
Codecs package is installed
$ dpkg -l | grep -i xpra
ii xpra 6.2.4-r0-1 amd64 tool to detach/reattach running X programs
ii xpra-audio 6.2.4-r0-1 amd64 Components required for xpra audio forwarding,
ii xpra-client 6.2.4-r0-1 amd64 tool to detach/reattach running X programs,
ii xpra-client-gtk3 6.2.4-r0-1 amd64 tool to detach/reattach running X programs,
ii xpra-codecs 6.2.4-r0-1 amd64 Picture and video codecs
ii xpra-codecs-extras 6.2.4-r0-1 amd64 Extra picture and video codecs
ii xpra-common 6.2.4-r0-1 amd64 tool to detach/reattach running X programs,
ii xpra-html5 13-r1569-1 all html5 xpra client
ii xpra-server 6.2.4-r0-1 amd64 tool to detach/reattach running X programs,
ii xpra-x11 6.2.4-r0-1 amd64 tool to detach/reattach running X programs,
$
$ xpra encoding
some GStreamer elements are missing or unavailable on this system:
vah264lpenc, vah264enc
Error: frame queue timeout after 3000ms
on 'buffer' of size 24576
of gstreamer-vaapih265enc(NV12 - 128x128)
version : (5, 0)
frames : 0
width : 128
height : 128
encoding : hevc
colorspace : NV12
dst_formats : ('YUV420P',)
Warning: gstreamer 'vaapih265enc' encoder failed
hevc compression failed on image 1 of 5
Error setting up the pipeline:
gst_parse_error: no element "nvh264dec" (1)
GStreamer pipeline for:
appsrc name=src emit-signals=1 block=0 is-live=1 do-timestamp=1 stream-type=0 format=2 caps=video/x-h264,width=128,height=128,profile=(string)main,stream-format=(string)byte-stream,alignment=(string)au ! \
nvh264dec name=decoder ! \
appsink name=sink emit-signals=1 max-buffers=10 drop=False sync=False async=True qos=False caps=video/x-raw,width=128,height=128,format=(string)NV12
Error creating context h264 128x128 YUV420P
gstreamer: h264 decoding failed: failed to setup gstreamer pipeline
modules found:
* csc_cython : /usr/lib/python3/dist-packages/xpra/codecs/csc_cython/converter.cpython-310-x86_64-linux-gnu.so
* csc_libyuv : No module named 'xpra.codecs.libyuv'
* dec_avif : /usr/lib/python3/dist-packages/xpra/codecs/avif/decoder.cpython-310-x86_64-linux-gnu.so
* dec_gstreamer : /usr/lib/python3/dist-packages/xpra/codecs/gstreamer/decoder.py
* dec_jpeg : /usr/lib/python3/dist-packages/xpra/codecs/jpeg/decoder.cpython-310-x86_64-linux-gnu.so
* dec_nvjpeg : No module named 'xpra.codecs.nvidia'
* dec_openh264 : /usr/lib/python3/dist-packages/xpra/codecs/openh264/decoder.cpython-310-x86_64-linux-gnu.so
* dec_pillow : /usr/lib/python3/dist-packages/xpra/codecs/pillow/decoder.py
* dec_spng : No module named 'xpra.codecs.spng'
* dec_vpx : /usr/lib/python3/dist-packages/xpra/codecs/vpx/decoder.cpython-310-x86_64-linux-gnu.so
* dec_webp : /usr/lib/python3/dist-packages/xpra/codecs/webp/decoder.cpython-310-x86_64-linux-gnu.so
* drm : /usr/lib/python3/dist-packages/xpra/codecs/drm/drm.cpython-310-x86_64-linux-gnu.so
* enc_avif : /usr/lib/python3/dist-packages/xpra/codecs/avif/encoder.cpython-310-x86_64-linux-gnu.so
* enc_gstreamer : /usr/lib/python3/dist-packages/xpra/codecs/gstreamer/encoder.py
* enc_jpeg : /usr/lib/python3/dist-packages/xpra/codecs/jpeg/encoder.cpython-310-x86_64-linux-gnu.so
* enc_nvjpeg : No module named 'xpra.codecs.nvidia'
* enc_openh264 : /usr/lib/python3/dist-packages/xpra/codecs/openh264/encoder.cpython-310-x86_64-linux-gnu.so
* enc_pillow : /usr/lib/python3/dist-packages/xpra/codecs/pillow/encoder.py
* enc_rgb : /usr/lib/python3/dist-packages/xpra/codecs/argb/encoder.py
* enc_spng : No module named 'xpra.codecs.spng'
* enc_vpx : /usr/lib/python3/dist-packages/xpra/codecs/vpx/encoder.cpython-310-x86_64-linux-gnu.so
* enc_webp : /usr/lib/python3/dist-packages/xpra/codecs/webp/encoder.cpython-310-x86_64-linux-gnu.so
* enc_x264 : /usr/lib/python3/dist-packages/xpra/codecs/x264/encoder.cpython-310-x86_64-linux-gnu.so
* evdi : No module named 'xpra.codecs.evdi'
* nvdec : No module named 'xpra.codecs.nvidia'
* nvenc : No module named 'xpra.codecs.nvidia'
* nvfbc : No module named 'xpra.codecs.nvidia'
* v4l2 : /usr/lib/python3/dist-packages/xpra/codecs/v4l2/virtual.cpython-310-x86_64-linux-gnu.so
codecs versions:
* avif : 0.9.3
* cython : 4.2
* drm : 4.4
* gstreamer : 5.0
* jpeg : 1.0
* numpy : 2.0.1
* openh264 : 2.2.0
* pillow : 9.0.1
* rgb : 6.0
* v4l2 : 1.0
* vpx : 1.11.0
* webp : 1.2.2
* x264 : 163
$
$ xpra video
some GStreamer elements are missing or unavailable on this system:
vah264lpenc, vah264enc
Error: frame queue timeout after 3000ms
on 'buffer' of size 24576
of gstreamer-vaapih265enc(NV12 - 128x128)
version : (5, 0)
frames : 0
width : 128
height : 128
encoding : hevc
colorspace : NV12
dst_formats : ('YUV420P',)
Warning: gstreamer 'vaapih265enc' encoder failed
hevc compression failed on image 1 of 5
Error setting up the pipeline:
gst_parse_error: no element "nvh264dec" (1)
GStreamer pipeline for:
appsrc name=src emit-signals=1 block=0 is-live=1 do-timestamp=1 stream-type=0 format=2 caps=video/x-h264,width=128,height=128,profile=(string)main,stream-format=(string)byte-stream,alignment=(string)au ! \
nvh264dec name=decoder ! \
appsink name=sink emit-signals=1 max-buffers=10 drop=False sync=False async=True qos=False caps=video/x-raw,width=128,height=128,format=(string)NV12
Error creating context h264 128x128 YUV420P
gstreamer: h264 decoding failed: failed to setup gstreamer pipeline
* csc
- BGRX_to_YUV420P : cython
- BGRX_to_YUV444P : cython
- BGR_to_YUV420P : cython
- BGR_to_YUV444P : cython
- GBRP10_to_r210 : cython
- GBRP_to_BGRX : cython
- GBRP_to_RGBX : cython
- RGBX_to_YUV420P : cython
- RGBX_to_YUV444P : cython
- RGB_to_YUV420P : cython
- RGB_to_YUV444P : cython
- YUV420P_to_BGR : cython
- YUV420P_to_BGRX : cython
- YUV420P_to_RGB : cython
- YUV420P_to_RGBX : cython
- YUV444P10_to_r210 : cython
- YUV444P_to_BGRX : cython
- YUV444P_to_RGBX : cython
- r210_to_BGR48 : cython
- r210_to_YUV420P : cython
- r210_to_YUV444P10 : cython
* decoding
- av1_to_YUV420P : gstreamer
- h264_to_YUV420P : openh264
- hevc_to_YUV420P : gstreamer
- vp8_to_YUV420P : vpx, gstreamer
- vp9_to_YUV420P : vpx, gstreamer
- vp9_to_YUV444P : vpx
* encoding
- BGR48_to_h264 : x264
- BGRA_to_jpega : jpeg
- BGRA_to_webp : webp
- BGRX_to_h264 : x264
- BGRX_to_jpeg : jpeg
- BGRX_to_webp : webp
- BGR_to_jpeg : jpeg
- BGR_to_webp : webp
- NV12_to_h264 : gstreamer-vaapih264enc
- RGBA_to_jpega : jpeg
- RGBA_to_webp : webp
- RGBX_to_jpeg : jpeg
- RGBX_to_webp : webp
- RGB_to_jpeg : jpeg
- RGB_to_webp : webp
- XBGR_to_jpeg : jpeg
- XRGB_to_jpeg : jpeg
- YUV420P_to_av1 : gstreamer-av1enc
- YUV420P_to_h264 : openh264, gstreamer-x264enc, gstreamer-openh264enc, x264
- YUV420P_to_jpeg : jpeg
- YUV420P_to_vp8 : vpx, gstreamer-vp8enc
- YUV420P_to_vp9 : vpx
- YUV422P_to_h264 : x264
- YUV422P_to_jpeg : jpeg
- YUV444P10_to_vp9 : vpx
- YUV444P_to_av1 : gstreamer-av1enc
- YUV444P_to_h264 : gstreamer-x264enc, x264
- YUV444P_to_jpeg : jpeg
- YUV444P_to_vp9 : vpx, gstreamer-vp9enc
- csc-module
- cython : active
- libyuv : active
- video-encoder
- gstreamer : active
- jpeg : active
- nvenc : active
- nvjpeg : active
- openh264 : active
- vpx : active
- webp : active
- x264 : active
* gpu
- csc : ()
- decodings : ()
- encodings : h264
$
Then maybe the client is missing them?
I would have expected at least some h264 or vp8 in the frame totals.
Client also has them:
$ xpra video
* csc
- BGRX_to_YUV420P : cython
- BGRX_to_YUV444P : cython
- BGR_to_YUV420P : cython
- BGR_to_YUV444P : cython
- GBRP10_to_r210 : cython
- GBRP_to_BGRX : cython
- GBRP_to_RGBX : cython
- RGBX_to_YUV420P : cython
- RGBX_to_YUV444P : cython
- RGB_to_YUV420P : cython
- RGB_to_YUV444P : cython
- YUV420P_to_BGR : cython
- YUV420P_to_BGRX : cython
- YUV420P_to_RGB : cython
- YUV420P_to_RGBX : cython
- YUV444P10_to_r210 : cython
- YUV444P_to_BGRX : cython
- YUV444P_to_RGBX : cython
- r210_to_BGR48 : cython
- r210_to_YUV420P : cython
- r210_to_YUV444P10 : cython
* decoding
- h264_to_YUV420P : openh264
- vp8_to_YUV420P : vpx
- vp9_to_YUV420P : vpx
- vp9_to_YUV444P : vpx
* encoding
- BGR48_to_h264 : x264
- BGRA_to_jpega : jpeg
- BGRA_to_webp : webp
- BGRX_to_h264 : x264
- BGRX_to_jpeg : jpeg
- BGRX_to_webp : webp
- BGR_to_jpeg : jpeg
- BGR_to_webp : webp
- RGBA_to_jpega : jpeg
- RGBA_to_webp : webp
- RGBX_to_jpeg : jpeg
- RGBX_to_webp : webp
- RGB_to_jpeg : jpeg
- RGB_to_webp : webp
- XBGR_to_jpeg : jpeg
- XRGB_to_jpeg : jpeg
- YUV420P_to_h264 : x264, openh264
- YUV420P_to_jpeg : jpeg
- YUV420P_to_vp8 : vpx
- YUV420P_to_vp9 : vpx
- YUV422P_to_h264 : x264
- YUV422P_to_jpeg : jpeg
- YUV444P10_to_vp9 : vpx
- YUV444P_to_h264 : x264
- YUV444P_to_jpeg : jpeg
- YUV444P_to_vp9 : vpx
- csc-module
- cython : active
- libyuv : active
- video-encoder
- gstreamer : active
- jpeg : active
- nvenc : active
- nvjpeg : active
- openh264 : active
- vpx : active
- webp : active
- x264 : active
* gpu
- csc : ()
- decodings : ()
- encodings : ()
$
Well, that's weird. I would really have expected one of the video codecs to kick in.
Maybe run with --video-encoders=openh264,vpx,x264 to ensure gstreamer doesn't get used.
Was there anything in your server log? Any warnings or errors?
In any case, I don't think this is the cause of the memory leak - though it may well make things worse if the leak is in the jpeg or webp encoders, since they will end up being used more.
I'm going to have to find the time to run Ubuntu somewhere, because the problem may well be tied to the library versions on that platform as I haven't seen any leaks on Fedora - even with long running sessions.
Was there anything in your server log? Any warnings or errors?
I start the server using "xpra start ssh://...". Am I able to somehow retroactively bring up the server log?
$ xpra info | grep -i server.log-file
server.log-file=/run/user/1000/xpra/1/server.log
Or you can even download it to your client from the system tray menu:
2025-08-26 08:39:14,869 Warning: gstreamer 'vaapih265enc' encoder failed
I have this? No errors about vp8, though.
Okay, so this:
--video-encoders=openh264,vpx,x264
Results in an absolute tragedy of a rendering output. Artifacts everywhere.
And now I'm running with this:
--encodings=all,-webp,-scroll
Which is working fine so far. Will have to wait a long time to see if it gets slow.
However, refreshing in emacs seems to be worse than before. This has always been an issue, but not a dealbreaker. xpra never correctly refreshed emacs sometimes. As in, emacs pops up a completion window or something, and it just won't appear on the client, or half of it will appear, or something else like that. Or if I paste lines, the bottom part of the window won't update and I have to C+l to get the whole window to refresh sometimes.
--video-encoders=openh264,vpx,x264Artifacts everywhere.
Expected since:
Xpra Server Version 6.2.4
refreshing in emacs seems to be worse than before
Also a known issue, many existing tickets about this: https://github.com/Xpra-org/xpra/issues/670#issuecomment-3139166061, #3660 Should be better with the current versions that include fa20fb86f5e678741fc50ac0659a609ef263cacb
Okay, I will upgrade soon. Maybe even tomorrow. Can you please recommend a specific version that you believe will be the best for me?
Can you please recommend a specific version that you believe will be the best for me?
Seeing the current memory improvements from #3871, I would go with the current beta channel to get 6.4RC
Otherwise the stable repo.
Will have to be 6.3.2, unfortunately. My server is on jammy, which does not have beta pacakges for 6.4. I'll do it soon.
My server is on jammy, which does not have beta packages for 6.4.
Found the cause: https://github.com/Xpra-org/xpra/issues/4618#issuecomment-3256937148 Then I had to wait for the Ubuntu mirrors to finish their scheduled maintenance. And even after that was finished... their servers were still very flaky and breaking the build.
But finally now the 6.4 beta builds for Ubuntu Jammy are available. (at 6.4-r39150 to be precise)
I have upgraded. Unfortunately there was no parity between betas on the two respective ubuntu flavours, so
Server: xpra 6.4-r39306-1 Client: xpra 6.4-r39262-1
I will have to run this for a long time, of course, but the rendering in emacs is now even worse. Disabling that double buffer extension seemed not to do it for me. If I tab-complete an M-x command, the completion window at the bottom is not rendered until i mouse-over.