protocol error with very wide screen

[rowlap]

There seems to be a problem with screen layouts exceeding 8192 pixels wide overall, for example 3x 3840x2160 monitors in a row.

If the connection parameters are chosen to obtain RemoteFX (i.e. 32-bit color depth, LAN mode) then after successful authentication, mstsc.exe (Windows 10 20H2) quits with the message

Because of a protocol error, this session will be disconnected. Please try connecting to the remote computer again.

Nothing stands out in xrdp / sesman debug logs as being a protocol error when mstsc disconnects.

If I request a specific height/width on the commandline, both xrdp and Windows servers provide max 8192 in "windowed mode", but Windows-to-Windows does work with Full Screen "Use all my monitors".

[matt335672]

Thanks for reporting this.

May I ask what is the output of xrdp -v? This will tell us the flags xrdp is compiled with.

The RDP protocol has limits of 8192 x 8192 for any one monitor, and 32766 x 32766 for the overall desktop. See here and also [MS-RDPBCGR] Appendix A notes <45> and <46>

[rowlap]

$ xrdp -v
xrdp 0.9.17
  A Remote Desktop Protocol Server.
  Copyright (C) 2004-2020 Jay Sorg, Neutrino Labs, and all contributors.
  See https://github.com/neutrinolabs/xrdp for more information.

  Configure options:
      --build=x86_64-redhat-linux-gnu
      --host=x86_64-redhat-linux-gnu
      --program-prefix=
      --disable-dependency-tracking
      --prefix=/usr
      --exec-prefix=/usr
      --bindir=/usr/bin
      --sbindir=/usr/sbin
      --sysconfdir=/etc
      --datadir=/usr/share
      --includedir=/usr/include
      --libdir=/usr/lib64
      --libexecdir=/usr/libexec
      --localstatedir=/var
      --sharedstatedir=/var/lib
      --mandir=/usr/share/man
      --infodir=/usr/share/info
      --enable-pixman
      --enable-painter
      --enable-vsock
      --with-socketdir=/run/xrdp
      build_alias=x86_64-redhat-linux-gnu
      host_alias=x86_64-redhat-linux-gnu
      CFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1  -m64 -mtune=generic
      LDFLAGS=-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld
      PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig

  Compiled with OpenSSL 1.0.2k-fips  26 Jan 2017

I appreciate that 3x 4k monitors are not a typical screen setup, but it's consistently reproducible with the right hardware. The per-monitor limits should be OK (and indeed it works during the xrdp login prompt).

[matt335672]

This is likely to be a duplicate of #2068

Once we've verified a patch for that we can see if it indeed fixes this one too.

[matt335672]

@rowlap - we've released v0.9.18 which may resolve this.

Are you in a position to try this release, or are you depending on a particular distro to package it for you?

Thanks.

[rowlap]

Looks like https://src.fedoraproject.org/rpms/xrdp already has the new release in testing (and it's easier to test with those packages).

[matt335672]

Yes, you're quite right.

There's a small chance the RHEL 7/8 ones may get replaced with updates for #2118, but these won't affect your platform.

[rowlap]

Tested against 0.9.18-5 Fedora RPM, but the same output is seen from mstsc "Because of a protocol error, this session will be disconnected."

[matt335672]

OK - thanks.

[labmonkey42]

Chiming in with an Ubuntu 22.04 VM doing this when I attempt to use all three 4k monitors (horizontally aligned).

xrdp 0.9.17
  A Remote Desktop Protocol Server.
  Copyright (C) 2004-2020 Jay Sorg, Neutrino Labs, and all contributors.
  See https://github.com/neutrinolabs/xrdp for more information.

  Configure options:
      --enable-ipv6
      --enable-jpeg
      --enable-fuse
      --enable-rfxcodec
      --enable-opus
      --enable-painter
      --enable-vsock
      --build=x86_64-linux-gnu
      --prefix=/usr
      --includedir=${prefix}/include
      --mandir=${prefix}/share/man
      --infodir=${prefix}/share/info
      --sysconfdir=/etc
      --localstatedir=/var
      --disable-silent-rules
      --libdir=${prefix}/lib/x86_64-linux-gnu
      --libexecdir=${prefix}/lib/x86_64-linux-gnu
      --disable-maintainer-mode
      --disable-dependency-tracking
      --with-socketdir=/run/xrdp/sockdir
      build_alias=x86_64-linux-gnu
      CFLAGS=-g -O2 -ffile-prefix-map=/build/xrdp-asBei3/xrdp-0.9.17=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security
      LDFLAGS=-Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -Wl,-z,now -Wl,--as-needed
      CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2 -Wno-error=deprecated-declarations
      PKG_CONFIG_PATH=/build/xrdp-asBei3/xrdp-0.9.17/pkgconfig

  Compiled with OpenSSL 3.0.2 15 Mar 2022

[metalefty]

8192x8192 per monitor is the limit when using RemotFX. Just a guess, xrdp (librfxcodec) might process 3 monitors as 1 monitor.

[metalefty]

I could reproduce the issue at fafd7bf

[metalefty]

[20221007-14:54:50] [ERROR] xrdp_sec_send_fastpath: xrdp_fastpath_send failed
[20221007-14:54:50] [ERROR] xrdp_rdp_send_fastpath: xrdp_sec_send_fastpath failed
[20221007-14:54:50] [ERROR] libxrdp_fastpath_send_frame_marker: xrdp_rdp_send_fastpath failed
[20221007-14:54:50] [ERROR] xrdp_iso_send: trans_write_copy_s failed
[20221007-14:54:50] [DEBUG] Closed socket 12 ([::ffff:192.168.24.5]:3389)

When I turned off fast path, the protocol error doesn't occur but screen is not drawn. Something is going wrong in screen encoding but RDP session is alive and other things are wokring (keyboard, mouse input, sound redirection). It works like that I'm operating computer with eyes closed.

There might be two issues:

• Fast path issue
• RFX screen encoding