Sockstat connection resolution unreliable

[atagar]

Migrated from ticket 23057.

Recently our Jenkins, which run Stem's tests, hosts upgraded their Debian distribution. Doing so caused our test_connections_by_sockstat to start failing...

======================================================================
FAIL: test_connections_by_sockstat
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/jenkins-workspace/workspace/stem-tor-ci/test/integ/util/connection.py", line 55, in test_connections_by_sockstat
    self.check_resolver(Resolver.SOCKSTAT)
  File "/srv/jenkins-workspace/workspace/stem-tor-ci/test/require.py", line 58, in wrapped
    return func(self, *args, **kwargs)
  File "/srv/jenkins-workspace/workspace/stem-tor-ci/test/integ/util/connection.py", line 37, in check_resolver
    self.fail('Unable to find our controller connection with %s (%s). Connections found were...\n\n%s\n\nCommand output was...\n\n%s' % (resolver, resolver_command, '\n'.join(map(str, connections)), resolver_output))
AssertionError: Unable to find our controller connection with sockstat (sockstat). Connections found were...

Connection(local_address=u'127.0.0.1', local_port=1024, remote_address=u'127.0.0.1', remote_port=38974, protocol=u'tcp', is_ipv6=False)
Connection(local_address=u'127.0.0.1', local_port=1024, remote_address=u'127.0.0.1', remote_port=38966, protocol=u'tcp', is_ipv6=False)

Command output was...

[u'USER     PROCESS              PID      PROTO  SOURCE ADDRESS            FOREIGN ADDRESS           STATE', u'jenkins  tor                  4759     tcp4   127.0.0.1:1024            *:*                       LISTEN', u'jenkins  tor                  4759     tcp4   127.0.0.1:1024            *:*                       LISTEN', u'jenkins  tor                  4759     tcp4   127.0.0.1:1024            127.0.0.1:38974           ESTABLISHED', u'jenkins  tor                  4759     tcp4   127.0.0.1:1024            127.0.0.1:38966           ESTABLISHED', u'jenkins  sockstat             4930     tcp4   127.0.0.1:38912           127.0.0.1:1111            ESTABLISHED', u'jenkins  sockstat             4930     tcp4   127.0.0.1:38912           127.0.0.1:1111            ESTABLISHED', u'jenkins  python               18063    tcp4   127.0.0.1:38912           127.0.0.1:1111            ESTABLISHED', u'jenkins  python               18063    tcp4   127.0.0.1:38912           127.0.0.1:1111            ESTABLISHED']

Here's the sockstat output...

USER     PROCESS              PID      PROTO  SOURCE ADDRESS            FOREIGN ADDRESS           STATE
jenkins  python               18337    tcp4   127.0.0.1:41728           127.0.0.1:1111            ESTABLISHED
jenkins  python               18337    tcp4   127.0.0.1:41728           127.0.0.1:1111            ESTABLISHED
jenkins  tor                  20588    tcp4   127.0.0.1:1024            *:*                       LISTEN
jenkins  tor                  20588    tcp4   127.0.0.1:1024            *:*                       LISTEN
jenkins  tor                  20588    tcp4   127.0.0.1:1024            127.0.0.1:41814           ESTABLISHED
jenkins  tor                  20588    tcp4   127.0.0.1:1024            127.0.0.1:41806           ESTABLISHED
jenkins  sockstat             20594    tcp4   127.0.0.1:41728           127.0.0.1:1111            ESTABLISHED
jenkins  sockstat             20594    tcp4   127.0.0.1:41728           127.0.0.1:1111            ESTABLISHED

Note that our socksport (1024) is listed twice, but our controlport (1111) isn't among the tor process entries at all. However, we're showing connections to the controlport.

Did some searching around but stumped. If we can fix sockstat to once again work on the jenkins hosts I'm all ears - otherwise we'll drop this connection resolution method in Stem 2.0.0.