"ConnectionResetError: [Errno 104] Connection reset by peer" from worker_1

[unzvfu]

Running the entity service locally:

$ docker-compose -f tools/docker-compose.yml up

and

port=`docker ps | grep nginx | sed 's/^.*0.0.0.0:\([0-9]\+\)->.*$/\1/'`
if [ "$port" -lt 32000 ]; then
   echo "Can't determine port"
   exit 2
fi

docker run -i \
       -e ENTITY_SERVICE_URL=http://localhost:${port}/api/v1 \
       -e ENTITY_SERVICE_PERMUTATION=1 \
       -e ENTITY_SERVICE_TEST_SIZE=10000 \
       --net=host \
       quay.io/n1analytics/entity-app python - < backend/test_service.py

results in a "Connection reset by peer" error in worker_1 every time. There's no error for TEST_SIZE equal to 100, 1000, or 5000.

Error log:

$ docker logs tools_worker_1
[HIL: Snip]
01:03:02   DEBUG    Computing similarity for a chunk of filters
01:03:02   DEBUG    Checking that the resource exists (in case of job being canceled)
01:03:02   DEBUG    Trying to connect to postgres db
01:03:02   DEBUG    Task accepted: async_worker.compute_filter_similarity[6e7de736-fed9-469f-a0e9-c752e824464e] pid:10
01:03:02   INFO     Task async_worker.compute_similarity[ce4fa95d-d8ad-48df-8def-db2a8d5bd574] succeeded in 0.08996254298835993s: None
01:03:02   DEBUG    Fetching and deserializing chunk of filters for dataprovider 1
01:03:02   DEBUG    Connected to minio
01:03:02   DEBUG    Starting new HTTP connection (1): minio
01:03:02   DEBUG    http://minio:9000 "GET /entityservice?location HTTP/1.1" 200 None
01:03:02   DEBUG    http://minio:9000 "HEAD /entityservice/ HTTP/1.1" 200 0
01:03:02   DEBUG    Starting new HTTP connection (2): minio
01:03:02   DEBUG    http://minio:9000 "GET /entityservice/raw-clks/1e402fc1c3ead19b8ac76e381d6ba0b88e561aff29d86cd0.bin HTTP/1.1" 206 1340000
01:03:02   DEBUG    Fetching and deserializing chunk of filters for dataprovider 2
01:03:02   DEBUG    Connected to minio
01:03:02   DEBUG    Starting new HTTP connection (1): minio
01:03:02   DEBUG    http://minio:9000 "GET /entityservice?location HTTP/1.1" 200 None
01:03:02   DEBUG    http://minio:9000 "HEAD /entityservice/ HTTP/1.1" 200 0
01:03:02   DEBUG    Starting new HTTP connection (2): minio
01:03:02   DEBUG    http://minio:9000 "GET /entityservice/raw-clks/0635e32454bb9c68e98b3a0f4765d45514d0c174ded1c41a.bin HTTP/1.1" 206 1340000
01:03:03   DEBUG    Calculating filter similarity
01:03:13   INFO     Timings: Prep: 0.1591 + 0.1477, Solve: 10.3933, Total: 10.7001  Comparisons: 100000000
01:03:13   DEBUG    Progress. Compared 100000000 CLKS
01:03:13   DEBUG    Offset DP1 by: 0, DP2 by: 0
01:04:13   WARNING  /usr/local/lib/python3.6/site-packages/celery/app/trace.py:549: RuntimeWarning: Exception raised outside body: OperationalError('Error 104 while writing to socket. Connection reset by peer.',):
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 543, in send_packed_command
    self._sock.sendall(item)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 572, in execute_command
    connection.send_command(*args)
  File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 563, in send_command
    self.send_packed_command(self.pack_command(*args))
  File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 556, in send_packed_command
    (errno, errmsg))
redis.exceptions.ConnectionError: Error 104 while writing to socket. Connection reset by peer.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 543, in send_packed_command
    self._sock.sendall(item)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 414, in _reraise_as_library_errors
    yield
  File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 494, in _ensured
    return fun(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/kombu/messaging.py", line 203, in _publish
    mandatory=mandatory, immediate=immediate,
  File "/usr/local/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 608, in basic_publish
    return self._put(routing_key, message, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", line 767, in _put
    client.lpush(self._q_for_pri(queue, pri), dumps(message))
  File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 1227, in lpush
    return self.execute_command('LPUSH', name, *values)
  File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 578, in execute_command
    connection.send_command(*args)
  File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 563, in send_command
    self.send_packed_command(self.pack_command(*args))
  File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 556, in send_packed_command
    (errno, errmsg))
redis.exceptions.ConnectionError: Error 104 while writing to socket. Connection reset by peer.

During handling of the above exception, another exception occurred:
[HIL: Snip]

[unzvfu]

Further investigation shows that the error occurs inconsistently for test sizes around 7975, and consistently for test sizes above 8000.

[hardbyte]

Entirely unhelpful with regard to solving the problem...

As impressive as the grep/sed command is you can probably use docker port tools_nginx_1 "8851" to get the port number you want.