Nesting recognition is broken

[uli42]

in hw/nxagent/Atoms.c there's a check in nxagentQueryAtoms if the nxagent is running nested. This check makes assumptions about atom ids that do not seem to work as indented. E.g.

  /*
   * Value of nxagentAtoms[8] is "NX_AGENT_SIGNATURE".
   *
   * This atom is created internally by the agent server at
   * startup to let other agents determine if they are run
   * nested. If agent is run nested, in fact, at the time it
   * will create the NX_AGENT_SIGNATURE atom on the real X
   * server it will find the existing atom with a value less
   * than any NX_IDENTITY created but itself.
   */

  if (nxagentAtoms[8] > nxagentAtoms[0])
  {
    nxagentAtoms[8] = None;
  }

  if (nxagentAtoms[8] != None)
  {
    /*
     * We are running nested in another agent
     * server.
     */

    nxagentChangeOption(Nested, 1);

    /*
     * Avoid the image degradation caused by
     * multiple lossy encoding.
     */

    fprintf(stderr, "Warning: Disabling use of lossy encoding in nested mode.\n");

    nxagentPackMethod = nxagentPackLossless;
  }

But during my tests with nested nxagents (both started manually (without nxcomp) and with x2goclient (with nxcomp)) I always could see that this test cannot work as the TEST output will always look like this for a nested nxagent:

nxagentPrintAtomMapInfo: Map at [0x28170e0] size [15] number of entry [256] auto increment [256].
[    0] local:     69 - remote:     69 - [0x851f8b] NX_IDENTITY
[    1] local:     70 - remote:     70 - [0x84c60f] WM_PROTOCOLS
[    2] local:     71 - remote:     71 - [0x84f9e7] WM_DELETE_WINDOW
[    3] local:     72 - remote:     72 - [0x851f97] WM_NX_READY
[    4] local:     73 - remote:     73 - [0x851fa3] MCOPGLOBALS
[    5] local:     74 - remote:     74 - [0x851faf] NX_CUT_BUFFER_SERVER
[    6] local:     75 - remote:     75 - [0x851fc4] TARGETS
[    7] local:     76 - remote:     76 - [0x851fcc] TEXT
[    8] local:     77 - remote:     77 - [0x851ecd] NX_AGENT_SIGNATURE
[    9] local:     78 - remote:     78 - [0x851fd1] NXDARWIN
[   10] local:     79 - remote:     79 - [0x83b998] CLIPBOARD
[   11] local:     80 - remote:     80 - [0x850cce] TIMESTAMP
[   12] local:     81 - remote:     81 - [0x850bee] UTF8_STRING
[   13] local:     82 - remote:     82 - [0x851fda] _NET_WM_STATE
[   14] local:     83 - remote:     83 - [0x851fe8] _NET_WM_STATE_FULLSCREEN
---------------------------------------------
nxagentQueryAtoms: Created intern atom [NX_IDENTITY] with id [69].
nxagentQueryAtoms: Created intern atom [WM_PROTOCOLS] with id [70].
nxagentQueryAtoms: Created intern atom [WM_DELETE_WINDOW] with id [71].
nxagentQueryAtoms: Created intern atom [WM_NX_READY] with id [72].
nxagentQueryAtoms: Created intern atom [MCOPGLOBALS] with id [73].
nxagentQueryAtoms: Created intern atom [NX_CUT_BUFFER_SERVER] with id [74].
nxagentQueryAtoms: Created intern atom [TARGETS] with id [75].
nxagentQueryAtoms: Created intern atom [TEXT] with id [76].
nxagentQueryAtoms: Created intern atom [NX_AGENT_SIGNATURE] with id [0].
nxagentQueryAtoms: Created intern atom [NXDARWIN] with id [0].
nxagentQueryAtoms: Created intern atom [CLIPBOARD] with id [79].
nxagentQueryAtoms: Created intern atom [TIMESTAMP] with id [80].
nxagentQueryAtoms: Created intern atom [UTF8_STRING] with id [81].
nxagentQueryAtoms: Created intern atom [_NET_WM_STATE] with id [82].
nxagentQueryAtoms: Created intern atom [_NET_WM_STATE_FULLSCREEN] with id [83].
nxagentQueryAtoms: Ending time is [3707320919] reference latency is [0] Ms.
--------------- Atom map in context [nxagentQueryAtoms: Exiting] ----------------------

The first nxagent (client to :0) has this output:

-------------- Atom map in context [nxagentInitAtomMap: Exiting] ----------------------
nxagentPrintAtomMapInfo: Map at [0xe63710] size [15] number of entry [256] auto increment [256].
[    0] local:     69 - remote:    618 - [0x851f4b] NX_IDENTITY
[    1] local:     70 - remote:    332 - [0x84c5cf] WM_PROTOCOLS
[    2] local:     71 - remote:    330 - [0x84f9a7] WM_DELETE_WINDOW
[    3] local:     72 - remote:    619 - [0x851f57] WM_NX_READY
[    4] local:     73 - remote:    620 - [0x851f63] MCOPGLOBALS
[    5] local:     74 - remote:    621 - [0x851f6f] NX_CUT_BUFFER_SERVER
[    6] local:     75 - remote:    459 - [0x851f84] TARGETS
[    7] local:     76 - remote:    486 - [0x851f8c] TEXT
[    8] local:     77 - remote:    622 - [0x851e8d] NX_AGENT_SIGNATURE
[    9] local:     78 - remote:    623 - [0x851f91] NXDARWIN
[   10] local:     79 - remote:    457 - [0x83b958] CLIPBOARD
[   11] local:     80 - remote:    461 - [0x850c8e] TIMESTAMP
[   12] local:     81 - remote:    328 - [0x850bae] UTF8_STRING
[   13] local:     82 - remote:    346 - [0x851f9a] _NET_WM_STATE
[   14] local:     83 - remote:    349 - [0x851fa8] _NET_WM_STATE_FULLSCREEN
---------------------------------------------
nxagentQueryAtoms: Created intern atom [NX_IDENTITY] with id [618].
nxagentQueryAtoms: Created intern atom [WM_PROTOCOLS] with id [332].
nxagentQueryAtoms: Created intern atom [WM_DELETE_WINDOW] with id [330].
nxagentQueryAtoms: Created intern atom [WM_NX_READY] with id [619].
nxagentQueryAtoms: Created intern atom [MCOPGLOBALS] with id [620].
nxagentQueryAtoms: Created intern atom [NX_CUT_BUFFER_SERVER] with id [621].
nxagentQueryAtoms: Created intern atom [TARGETS] with id [459].
nxagentQueryAtoms: Created intern atom [TEXT] with id [486].
nxagentQueryAtoms: Created intern atom [NX_AGENT_SIGNATURE] with id [0].
nxagentQueryAtoms: Created intern atom [NXDARWIN] with id [0].
nxagentQueryAtoms: Created intern atom [CLIPBOARD] with id [457].
nxagentQueryAtoms: Created intern atom [TIMESTAMP] with id [461].
nxagentQueryAtoms: Created intern atom [UTF8_STRING] with id [328].
nxagentQueryAtoms: Created intern atom [_NET_WM_STATE] with id [346].
nxagentQueryAtoms: Created intern atom [_NET_WM_STATE_FULLSCREEN] with id [349].
nxagentQueryAtoms: Ending time is [3704700082] reference latency is [2] Ms.
--------------- Atom map in context [nxagentQueryAtoms: Exiting] ----------------------

As you can see neither of them prints out the "nested" message.

I think the logic is totally broken. We should probably simply check if the remote side has NX_AGENT_SIGNATURE right at the beginning.

[uli42]

I looks to me as the main problem is that nxagent is creating ALL atoms on the real X server, so there's no way to distinguish an nxagent from a real X server by checking the existing atoms.

[uli42]

I think the whole approach of using atoms (and their order) is wrong as long as nxagent creates the atoms on the remote side itself. This will create the atoms on a real X server or an nxagent, does not matter. So after the first time nx has been run against a real X server it is undistinguishable from an nxagent because it now has all the atoms, too (and forever!).

[uli42]

Either check for the NXAGENT_VERSION property on the root window of the X server or prevent forwarding of the NXAGENT_SIGNATURE and/or NX_IDENTITY atom.