xrdp
xrdp copied to clipboard
neutrinolabs
VNC backend fails to work on FIPS-enabled machines
We are required to run FIPS enabled on all of our systems. However, we are unable to RDP into RHEL 9 systems when FIPS is enabled on RHEL 9 regardless of the FIPS setting on Windows. I believe this is due to RHEL 9 utilizing FIPS 140-3 standards while Windows uses 140-2. From the documentation I've found, when configured for FIPS, XRDP uses 140-1. 140-3 doesn't allow ciphers that are approved for 140-1 while 140-2 does. Are there any plans of making XRDP compliant with FIPS 140-2 or 140-3 even?
Thanks.
I can see what's happening here.
The connection to the RHEL 9 machine is working fine. However, you're using the VNC backend, and that (currently) uses VNC authentication which is based on DES. Open SSL is refusing to use DES to encrypt the password on our side. If we send it, it's highly likely that tigervnc will refuse to decrypt it, as that's the point of FIPS compliance.
Consider installing xorgxrdp and using that as your backend. There are some advantages to this - performance is generally better. However, you will find some slight differences in your way of working. Another disadvantage of this backend is xorgxrdp is tied in with the version of the Xorg X server. That might make point-releases of RHEL harder to deal with, as you might find xorgxrdp refuses to run until it is rebuilt. Running point-releases in a pre-production environment (which you are probably doing anyway) will mitigate this.
The obvious longer-term solution here is to have an option for the VNC backend to use UNIX domain sockets. That avoids not only FIPS issues, but also avoids this problem in the future for other encryption types, and removes the VNC socket from the IP stack where it can be affected by firewalls.
We need to retain the ability for the VNC backend to connect to standard VNC TCP ports for other use-cases.
I'm going to re-title this issue and change it to a feature request.