Xpra-org
help users configure GStreamer
As per https://github.com/Xpra-org/xpra/issues/3750#issuecomment-1543814606, #3872, #3706 and #3843. We need a tool to help users. This tool can test various pipelines and the user should be able to save the ones that work and offer good performance.
One major difficulty is going to be runtime management of these pipelines. Xpra normally adjusts framerate, quality and speed to the bandwidth conditions but this is much harder to do when we don't really know what the pipeline is made of without introspecting it..
To follow up on the referenced ticket, I confirm that Gstreamer VAAPI is failing on AMDGPU (using 5825U with Renoir/Barcelo Graphics) on any distribution I tested (Ubuntu 22.04, 22.04, 23.10 and Fedora 38). Issue occurs with both open source and "pro" version of the driver, on kernel 6.2, 6.4 and 6.5 version.
FFMPEG is working fine to convert video with VAAPI using command line so it is unfortunate it is not available for XPRA (anyway even with previous version it did not work with xpra because it was detected as "h264+mp4" rather than just "h264" so it was not being used by XPRA).
@belotv This is evidently related to: https://discussion.fedoraproject.org/t/proprietary-video-codecs-are-no-longer-hardware-accelerated-by-default-on-amd-gpus-since-fedora-37/73723 https://baronhk.wordpress.com/2022/10/05/the-facts-about-video-codec-acceleration-on-fedora-37/ https://news.itsfoss.com/fedora-drops-vaapi-codec/ https://discussion.fedoraproject.org/t/why-is-fedora-38-missing-the-essential-gpu-driver-component-to-have-hw-acceleration-enabled-by-default/87945/3
But why it doesn't work even in AMDGPU Pro drivers is unknown.
No, I am aware that codecs are not included in Fedora anymore, I enabled all codecs by installing from rpmfusion (and even tried compiling mesa/gstreamer manually). Codecs are properly detected by vainfo, and can be used for video encoding using ffmpeg VAAPi hardware acceleration successfully (5 times quicker than CPU encoding, so it definitely works).
Running "xpra video" causes a ring vcn_enc0 timeout and reset the GPU same as #3843.
It seems that xpra is using some gstreamer parameters that amdgpu vaapi does not like.
@ehfd / @belotv One of the key things that we do different in xpra is that we try to tune the video encoders dynamically to adapt to the content, client performance and to the bandwidth available. This is more difficult with gstreamer elements than with the native codecs we normally use, but this makes a massive difference to performance and perceived quality. It's quite possible that our default settings are wrong for some encoders, though I would argue that locking up the system is not helpful!
Yes not very practical to reset the GPU when an encoding fails, however I guess they had no other option as they cannot guarantee the operations after the ring has failed.
I may try to debug that over the next 2 months if I have time (but it is unlikely I will).
Unrelated but you added amf encoding for Gstreamer for Linux only, which does not make sense 😊. The gstreamer implementation of AMF is Windows only as it relies on d3d11 (https://github.com/GStreamer/gstreamer/blob/57bfbf51b2ca8a82d04070a1445ece3ee14077e0/subprojects/gst-plugins-bad/sys/amfcodec/meson.build#L10)
libva should be good, it just doesn't work reliably enough.
The point of this ticket is to tell users quite clearly that even trying to run the pipeline might completely crash their system (as it did with mine again just a couple of days ago) and we can enable what doesn't crash.
That said, it might crash at some later point due to a library / kernel update.. but at least we will try to give them a way to enable it safely.
vais the way forward for AMD GPUs in Linux.
Ah, I meant vah264enc, vaav1enc, and all other GStreamer plugins in the family.
vaapih264enc is set to be deprecated.
So ... given
... can I just disable gstreamer?
Amazing benefits don't warrant instability 😕
Amazing benefits don't warrant instability 😕
No one is making you run this tool.
No, ofc not 😅
Should we then ... disable gstreamer when connecting? I'm not sure how this and
2023-11-11 22:15:40,676 Error setting up the pipeline:
2023-11-11 22:15:40,676 gst_parse_error: could not link decoder to sink (3)
2023-11-11 22:15:40,676 GStreamer pipeline for:
2023-11-11 22:15:40,676 appsrc name=src emit-signals=1 block=0 is-live=1 do-timestamp=1 stream-type=0 format=2 caps=video/x-h264,width=128,height=128,profile=(string)main,stream-format=(string)byte-stream,alignment=(string)au ! \
2023-11-11 22:15:40,677 d3d11h264dec name=decoder ! \
2023-11-11 22:15:40,677 appsink name=sink emit-signals=1 max-buffers=10 drop=false sync=false async=true qos=false caps=video/x-raw,width=128,height=128,format=(string)I420
2023-11-11 22:15:40,677 Error creating context h264 128x128 YUV420P
2023-11-11 22:15:40,677 gstreamer: h264 decoding failed: failed to setup gstreamer pipeline
that interface with each other 😕
Have you been able to fix that error with setting up the Pipeline. I am having the same issue.
@Vkhark + @stdedos these are harmless, we're checking if the d3d11h264dec element can be used.
Nothing to worry about.
As of today, d3d11 is causing bigger problems: not loading and later popping up alert dialogs. Ouch.
https://github.com/Xpra-org/xpra/issues/4100#issuecomment-1994488334
Encoding
Platform-agnostic notes
If nvidia hardware is found, we should always recommend nvfbc and nvenc - though there are known compatibility issues with drivers / CUDA / nvidia APIs..
Let's try to move away from environment variables for configuration, perhaps overload the mode to add options there? ie: xpra shadow,backend=nvfbc,cuda=1,push=0,device=GTX1070
X11
We currently only enable gstreamer CaptureAndEncode here:
https://github.com/Xpra-org/xpra/blob/cc39c2ee7c60631428122a9fd47a69b949e80680/xpra/server/window/video_compress.py#L619-L625
Ideally, we would make it easier to enable it, at least for desktop mode as this is much more likely to benefit from hardware accelerated stream encoders, seamless may suffer from slow gstreamer pipeline starts (especially when hardware contexts are involved).
For monitor mode, we would need to start a pipeline for each monitor and ensure the geometry used with ximagesrc matches.
The rate control is currently non-existent as the data callback:
https://github.com/Xpra-org/xpra/blob/cc39c2ee7c60631428122a9fd47a69b949e80680/xpra/server/window/video_compress.py#L652
Just ends up queing the data in a new draw packet:
https://github.com/Xpra-org/xpra/blob/cc39c2ee7c60631428122a9fd47a69b949e80680/xpra/server/window/compress.py#L2759-L2770
The pipeline object should be able to handle our speed and quality tuning:
https://github.com/Xpra-org/xpra/blob/cc39c2ee7c60631428122a9fd47a69b949e80680/xpra/server/window/compress.py#L1312
https://github.com/Xpra-org/xpra/blob/cc39c2ee7c60631428122a9fd47a69b949e80680/xpra/server/window/compress.py#L1377
Though the mapping to encoder attributes is going to be "interesting".
And perhaps also the bandwidth-limit. Not sure about b-frames and other more subtle attributes.
The colorspace (and eventually scaling?) chosen could re-use most of the logic we use in
https://github.com/Xpra-org/xpra/blob/cc39c2ee7c60631428122a9fd47a69b949e80680/xpra/server/window/video_compress.py#L1374
(perhaps just filter and only use gstreamer pipeline options from this call)
For shadow mode, there are no nvfbc elements in gstreamer, so hardware encoders will always need to upload pixel data to the GPU.
Wayland
(and potentially also some X11 DEs) via RemoteDesktop API:
The RemoteDesktop backend just adds input support to PortalShadow.
Once we have negotiated the dbus freedesktop dance to get a pipewire handle:
https://github.com/Xpra-org/xpra/blob/72b841343ff34a9be40f555c16d8bcf64b9943c6/xpra/platform/posix/fd_portal_shadow.py#L245
Will create a pipewire CaptureAndEncode
This needs to be smarter about choosing the best encoder element.
The problem is that it is difficult to test reliably without a pipewire handle... (we want to stay on the GPU and a videotestsrc produces data in main memory)
It's not clear yet where the encoder tuning should be hooked up:
https://github.com/Xpra-org/xpra/blob/72b841343ff34a9be40f555c16d8bcf64b9943c6/xpra/platform/posix/fd_portal_shadow.py#L40-L46
is not a good fit, and the pipewire gstreamer pipeline nature of shadow sessions on Wayland don't map well to the damage events trigger we normally use with all the other servers.
MS Windows
Just try each element in this exact order? https://github.com/Xpra-org/xpra/blob/72b841343ff34a9be40f555c16d8bcf64b9943c6/xpra/platform/win32/shadow_server.py#L55 Apparently, d3d11convert can scale, so we should be good to go with any software / hardware encoder? Using a d3d11upload with sources that don't provide d3d11 memory. Do we need a d3d11download if we use a GPU encoder? Probably? For accelerated encoding, mediafoundation but obviously there is a snag: MediaFoundation Plugin not found: The mediafoundation plugin requires MSVC. MinGW does not ship the necessary wrappers. You can download the 1.18 MSVC binaries from here: https://gstreamer.freedesktop.org/download/ That's a major showstopper right there - licensing, crt and packaging issues aplenty.
See also gstreamer nvcodec...:
new encoders nv{cuda,d3d11,autogpu}{h264,h265}enc in 1.24 release contain the new preset support
MacOS
Needs testing with avfvideosrc:
https://github.com/Xpra-org/xpra/blob/72b841343ff34a9be40f555c16d8bcf64b9943c6/xpra/platform/darwin/shadow_server.py#L27
No idea what support we get from the applemedia elements, or how reliable they are.
Testing this with virtualized hardware is going to be a major PITA.
Decoding
To be updated. Less of a priority as our software decoders aren't usually a bottleneck and they are very reliable.
Open questions
- do we care about jpeg? meh (browsers?)
- run test pipelines and measure? latency, compression ratio, etc.
@totaam
Since VA-API and NVIDIA both provide hardware JPEG encoders, it might be worth looking at. Since JPEG streams can be sent pixel-by-pixel compared to video which isn't, it might work for low frame rate high latency scenarios (the traditional VNC scenario with XDamage on).
https://gstreamer.freedesktop.org/documentation/nvcodec/nvjpegenc.html?gi-language=python https://gstreamer.freedesktop.org/documentation/qsv/qsvjpegenc.html?gi-language=python
@ehfd we've had hardware jpeg encoding and decoding in xpra for quite some time now, this is about gstreamer. Doing jpeg via gstreamer probably is not worth it.
Encoding
Should I add those in the Wiki / Docs under a developer diary or something?
Seems a lot of compiled information 😅
Should I add those in the Wiki / Docs under a developer diary or something?
Not yet! This is just a recap of where we are at so that I can try to come up with a plan. Once the changes are implemented, a lot of this information will be obsolete.
Worth remembering that not having python3-gstreamer1 installed can manifest itself as:
TypeError: unknown type GstValueList
Sent to stderr... And missing encoder information in get_encoder_info
Not obvious, is it!
As of the latest commit, here's how I test desktop or monitor mode with gstreamer hardware accel (Intel only):
XPRA_STREAM_MODE=gstreamer xpra start --bind-tcp=0.0.0.0:10000 --start=xterm :10 -d compress,gstreamer
Then attach:
xpra attach --no-mmap --encoding=stream
@totaam Do you have VA-API accelerated jpeg encode in your non-GStreamer backend?
@ehfd no vaapi, nvjpeg instead.
Could do it via gstreamer I guess, but it's a pain to start a pipeline for what is one-shot encodes (jpeg).
It could be possible to use libva to do it outside of GStreamer just like how you do it with nvjpeg.
libva is a generic api, which makes it a nightmare to code properly for. I've tried.
v6 will have this GUI disabled for now and we automatically enabled encoding=stream and gstreamer streaming mode when appropriate: 509af1a4cf7124076a6e9693742561fd435fc2c6
I am disabling GStreamer again following server crashes I have seen at random, and other reports - ie from IRC:
Xorg-for-Xpra-43: ../../../include/privates.h:121: dixGetPrivateAddr: Assertion `key->initialized' failed.
(EE)
(EE) Backtrace:
(EE) 0: /usr/libexec/Xorg-for-Xpra-43 (xorg_backtrace+0x89) [0x56094a14d739]
(EE) 1: /usr/libexec/Xorg-for-Xpra-43 (0x560949f77000+0x1df269) [0x56094a156269]
(EE) 2: /lib64/libc.so.6 (0x7f0274800000+0x3e6f0) [0x7f027483e6f0]
(EE) 3: /lib64/libc.so.6 (0x7f0274800000+0x8b94c) [0x7f027488b94c]
(EE) 4: /lib64/libc.so.6 (raise+0x16) [0x7f027483e646]
(EE) 5: /lib64/libc.so.6 (abort+0xd3) [0x7f02748287f3]
(EE) 6: /lib64/libc.so.6 (0x7f0274800000+0x2871b) [0x7f027482871b]
(EE) 7: /lib64/libc.so.6 (0x7f0274800000+0x37386) [0x7f0274837386]
(EE) 8: /usr/libexec/Xorg-for-Xpra-43 (0x560949f77000+0x18d405) [0x56094a104405]
(EE) 9: /usr/libexec/Xorg-for-Xpra-43 (0x560949f77000+0x18d8cc) [0x56094a1048cc]
(EE) 10: /usr/libexec/Xorg-for-Xpra-43 (0x560949f77000+0x196b1b) [0x56094a10db1b]
(EE) 11: /usr/libexec/Xorg-for-Xpra-43 (0x560949f77000+0x4caf7) [0x560949fc3af7]
(EE) 12: /lib64/libc.so.6 (0x7f0274800000+0x29590) [0x7f0274829590]
(EE) 13: /lib64/libc.so.6 (__libc_start_main+0x80) [0x7f0274829640]
(EE) 14: /usr/libexec/Xorg-for-Xpra-43 (_start+0x25) [0x560949fc3ea5]
(EE)
(EE)
Fatal server error:
(EE) Caught signal 6 (Aborted). Server aborting
(EE)
(EE)
This was solved by --video-encoders=all,-gstreamer --video-decoders=all,-gstreamer.
One more crasher bug: https://github.com/Xpra-org/xpra/issues/4330#issuecomment-2308760220
Is there any configuration where even probing GStreamer is safe?
Too many problems, xpra 6.3 will default to no-gstreamer for video codecs again: d6b60eda68675dcdc1b194af7aff0b3040a15019