zimfarm
zimfarm copied to clipboard
openzim
StackExchanger watcher regularly restarts after connection reset by peer errors
StackExchanger watcher regularly restarts after connection reset by peer errors
It looks like intermittent issue on an upstream (archive.org? wasabi?)
Sample occurence:
Common labels: {"cluster":"kiwix-prod","container":"zimfarm-watcher","flags":"F","job":"zimfarm/zimfarm-watcher","namespace":"zimfarm","pod":"watcher-deployment-54cf77666f-k79zm","service_name":"zimfarm-watcher"}
Line limit: "1000 (379 displayed)"
Total bytes processed: "33.4 kB"
2024-06-03 08:56:03.893 [2024-06-03 06:56:03,762::INFO] Getting list of SE domains
2024-06-03 08:56:07.699 [2024-06-03 06:56:07,699::INFO] Grabbing online versions and comparing with S3…
2024-06-03 09:01:18.439 [2024-06-03 07:01:18,439::ERROR] FAILED. An error occurred: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2024-06-03 09:01:19.269 [2024-06-03 07:01:18,439::ERROR] ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2024-06-03 09:01:19.269 Traceback (most recent call last):
2024-06-03 09:01:19.269 File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 793, in urlopen
2024-06-03 09:01:19.269 response = self._make_request(
2024-06-03 09:01:19.269 ^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.269 File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 491, in _make_request
2024-06-03 09:01:19.269 raise new_e
2024-06-03 09:01:19.269 File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 467, in _make_request
2024-06-03 09:01:19.269 self._validate_conn(conn)
2024-06-03 09:01:19.269 File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
2024-06-03 09:01:19.269 conn.connect()
2024-06-03 09:01:19.269 File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 653, in connect
2024-06-03 09:01:19.270 sock_and_verified = _ssl_wrap_socket_and_match_hostname(
2024-06-03 09:01:19.270 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270 File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 806, in _ssl_wrap_socket_and_match_hostname
2024-06-03 09:01:19.270 ssl_sock = ssl_wrap_socket(
2024-06-03 09:01:19.270 ^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270 File "/usr/local/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 465, in ssl_wrap_socket
2024-06-03 09:01:19.270 ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
2024-06-03 09:01:19.270 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270 File "/usr/local/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 509, in _ssl_wrap_socket_impl
2024-06-03 09:01:19.270 return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
2024-06-03 09:01:19.270 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270 File "/usr/local/lib/python3.12/ssl.py", line 455, in wrap_socket
2024-06-03 09:01:19.270 return self.sslsocket_class._create(
2024-06-03 09:01:19.270 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270 File "/usr/local/lib/python3.12/ssl.py", line 1042, in _create
2024-06-03 09:01:19.270 self.do_handshake()
2024-06-03 09:01:19.270 File "/usr/local/lib/python3.12/ssl.py", line 1320, in do_handshake
2024-06-03 09:01:19.270 self._sslobj.do_handshake()
2024-06-03 09:01:19.270 ConnectionResetError: [Errno 104] Connection reset by peer
2024-06-03 09:01:19.270
2024-06-03 09:01:19.270 During handling of the above exception, another exception occurred:
2024-06-03 09:01:19.270
2024-06-03 09:01:19.270 Traceback (most recent call last):
2024-06-03 09:01:19.270 File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 564, in send
2024-06-03 09:01:19.270 resp = conn.urlopen(
2024-06-03 09:01:19.270 ^^^^^^^^^^^^^
2024-06-03 09:01:19.270 File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 847, in urlopen
2024-06-03 09:01:19.270 retries = retries.increment(
2024-06-03 09:01:19.270 ^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270 File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 470, in increment
2024-06-03 09:01:19.270 raise reraise(type(error), error, _stacktrace)
2024-06-03 09:01:19.270 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270 File "/usr/local/lib/python3.12/site-packages/urllib3/util/util.py", line 38, in reraise
2024-06-03 09:01:19.270 raise value.with_traceback(tb)
2024-06-03 09:01:19.270 File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 793, in urlopen
2024-06-03 09:01:19.270 response = self._make_request(
2024-06-03 09:01:19.270 ^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270 File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 491, in _make_request
2024-06-03 09:01:19.270 raise new_e
2024-06-03 09:01:19.270 File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 467, in _make_request
2024-06-03 09:01:19.270 self._validate_conn(conn)
2024-06-03 09:01:19.270 File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
2024-06-03 09:01:19.270 conn.connect()
2024-06-03 09:01:19.270 File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 653, in connect
2024-06-03 09:01:19.270 sock_and_verified = _ssl_wrap_socket_and_match_hostname(
2024-06-03 09:01:19.270 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270 File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 806, in _ssl_wrap_socket_and_match_hostname
2024-06-03 09:01:19.270 ssl_sock = ssl_wrap_socket(
2024-06-03 09:01:19.270 ^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270 File "/usr/local/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 465, in ssl_wrap_socket
2024-06-03 09:01:19.270 ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
2024-06-03 09:01:19.270 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270 File "/usr/local/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 509, in _ssl_wrap_socket_impl
2024-06-03 09:01:19.270 return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
2024-06-03 09:01:19.270 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270 File "/usr/local/lib/python3.12/ssl.py", line 455, in wrap_socket
2024-06-03 09:01:19.270 return self.sslsocket_class._create(
2024-06-03 09:01:19.270 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270 File "/usr/local/lib/python3.12/ssl.py", line 1042, in _create
2024-06-03 09:01:19.270 self.do_handshake()
2024-06-03 09:01:19.270 File "/usr/local/lib/python3.12/ssl.py", line 1320, in do_handshake
2024-06-03 09:01:19.270 self._sslobj.do_handshake()
2024-06-03 09:01:19.270 urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2024-06-03 09:01:19.270
2024-06-03 09:01:19.270 During handling of the above exception, another exception occurred:
2024-06-03 09:01:19.270
2024-06-03 09:01:19.270 Traceback (most recent call last):
2024-06-03 09:01:19.270 File "/usr/local/bin/watcher", line 583, in entrypoint
2024-06-03 09:01:19.270 sys.exit(runner.run())
2024-06-03 09:01:19.270 ^^^^^^^^^^^^
2024-06-03 09:01:19.270 File "/usr/local/bin/watcher", line 494, in run
2024-06-03 09:01:19.270 self.check_and_go()
2024-06-03 09:01:19.270 File "/usr/local/bin/watcher", line 427, in check_and_go
2024-06-03 09:01:19.270 last_modified = get_last_modified_for(url)
2024-06-03 09:01:19.270 ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270 File "/usr/local/bin/watcher", line 68, in get_last_modified_for
2024-06-03 09:01:19.270 with requests.head(url, allow_redirects=True) as resp:
2024-06-03 09:01:19.270 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270 File "/usr/local/lib/python3.12/site-packages/requests/api.py", line 100, in head
2024-06-03 09:01:19.270 return request("head", url, **kwargs)
2024-06-03 09:01:19.270 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270 File "/usr/local/lib/python3.12/site-packages/requests/api.py", line 59, in request
2024-06-03 09:01:19.270 return session.request(method=method, url=url, **kwargs)
2024-06-03 09:01:19.270 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270 File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
2024-06-03 09:01:19.270 resp = self.send(prep, **send_kwargs)
2024-06-03 09:01:19.270 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270 File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
2024-06-03 09:01:19.270 r = adapter.send(request, **kwargs)
2024-06-03 09:01:19.270 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270 File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 579, in send
2024-06-03 09:01:19.270 raise ConnectionError(err, request=request)
2024-06-03 09:01:19.270 requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2024-06-03 09:01:55.223 [entrypoint] exposing s3_url secret as S3_URL
2024-06-03 09:01:55.356 [entrypoint] exposing zimfarm_username secret as ZIMFARM_USERNAME
2024-06-03 09:01:55.357 [entrypoint] exposing zimfarm_password secret as ZIMFARM_PASSWORD
2024-06-03 09:02:16.063 [2024-06-03 07:02:16,063::INFO]
2024-06-03 09:02:16.063 __ _ _
2024-06-03 09:02:16.063 / _| __ _ _ __ _ __ ___ __ ____ _| |_ ___| |__ ___ _ __
2024-06-03 09:02:16.063 | |_ / _` | '__| '_ ` _ \ ____\ \ /\ / / _` | __/ __| '_ \ / _ \ '__|
2024-06-03 09:02:16.063 | _| (_| | | | | | | | |_____\ V V / (_| | || (__| | | | __/ |
2024-06-03 09:02:16.063 |_| \__,_|_| |_| |_| |_| \_/\_/ \__,_|\__\___|_| |_|\___|_|
2024-06-03 09:02:16.063
2024-06-03 09:02:16.063
2024-06-03 09:02:16.063 [2024-06-03 07:02:16,063::INFO] Testing S3 credentials
2024-06-03 09:02:40.652 [2024-06-03 07:02:40,652::INFO] Testing Zimfarm credentials with https://api.farm.openzim.org/v1…
2024-06-03 09:02:42.048 [2024-06-03 07:02:42,048::INFO] Starting watcher:
2024-06-03 09:02:42.048 with zimfarm username: stackwatcher
2024-06-03 09:02:42.048 using cache: s3.us-west-1.wasabisys.com
2024-06-03 09:02:42.048 with bucket: org-kiwix-stackexchange
2024-06-03 09:02:42.048 [2024-06-03 07:02:42,048::INFO] Getting list of SE domains
2024-06-03 09:02:44.595 [2024-06-03 07:02:44,594::INFO] Grabbing online versions and comparing with S3…
2024-06-03 09:03:30.530 [2024-06-03 07:03:30,530::ERROR] FAILED. An error occurred: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2024-06-03 09:03:30.549 [2024-06-03 07:03:30,530::ERROR] ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2024-06-03 09:03:30.549 Traceback (most recent call last):
2024-06-03 09:03:30.549 File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 793, in urlopen
2024-06-03 09:03:30.549 response = self._make_request(
2024-06-03 09:03:30.549 ^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.549 File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 491, in _make_request
2024-06-03 09:03:30.549 raise new_e
2024-06-03 09:03:30.549 File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 467, in _make_request
2024-06-03 09:03:30.549 self._validate_conn(conn)
2024-06-03 09:03:30.549 File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
2024-06-03 09:03:30.549 conn.connect()
2024-06-03 09:03:30.549 File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 653, in connect
2024-06-03 09:03:30.549 sock_and_verified = _ssl_wrap_socket_and_match_hostname(
2024-06-03 09:03:30.549 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.549 File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 806, in _ssl_wrap_socket_and_match_hostname
2024-06-03 09:03:30.549 ssl_sock = ssl_wrap_socket(
2024-06-03 09:03:30.549 ^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.549 File "/usr/local/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 465, in ssl_wrap_socket
2024-06-03 09:03:30.549 ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
2024-06-03 09:03:30.550 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550 File "/usr/local/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 509, in _ssl_wrap_socket_impl
2024-06-03 09:03:30.550 return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
2024-06-03 09:03:30.550 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550 File "/usr/local/lib/python3.12/ssl.py", line 455, in wrap_socket
2024-06-03 09:03:30.550 return self.sslsocket_class._create(
2024-06-03 09:03:30.550 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550 File "/usr/local/lib/python3.12/ssl.py", line 1042, in _create
2024-06-03 09:03:30.550 self.do_handshake()
2024-06-03 09:03:30.550 File "/usr/local/lib/python3.12/ssl.py", line 1320, in do_handshake
2024-06-03 09:03:30.550 self._sslobj.do_handshake()
2024-06-03 09:03:30.550 ConnectionResetError: [Errno 104] Connection reset by peer
2024-06-03 09:03:30.550
2024-06-03 09:03:30.550 During handling of the above exception, another exception occurred:
2024-06-03 09:03:30.550
2024-06-03 09:03:30.550 Traceback (most recent call last):
2024-06-03 09:03:30.550 File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 564, in send
2024-06-03 09:03:30.550 resp = conn.urlopen(
2024-06-03 09:03:30.550 ^^^^^^^^^^^^^
2024-06-03 09:03:30.550 File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 847, in urlopen
2024-06-03 09:03:30.550 retries = retries.increment(
2024-06-03 09:03:30.550 ^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550 File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 470, in increment
2024-06-03 09:03:30.550 raise reraise(type(error), error, _stacktrace)
2024-06-03 09:03:30.550 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550 File "/usr/local/lib/python3.12/site-packages/urllib3/util/util.py", line 38, in reraise
2024-06-03 09:03:30.550 raise value.with_traceback(tb)
2024-06-03 09:03:30.550 File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 793, in urlopen
2024-06-03 09:03:30.550 response = self._make_request(
2024-06-03 09:03:30.550 ^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550 File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 491, in _make_request
2024-06-03 09:03:30.550 raise new_e
2024-06-03 09:03:30.550 File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 467, in _make_request
2024-06-03 09:03:30.550 self._validate_conn(conn)
2024-06-03 09:03:30.550 File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
2024-06-03 09:03:30.550 conn.connect()
2024-06-03 09:03:30.550 File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 653, in connect
2024-06-03 09:03:30.550 sock_and_verified = _ssl_wrap_socket_and_match_hostname(
2024-06-03 09:03:30.550 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550 File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 806, in _ssl_wrap_socket_and_match_hostname
2024-06-03 09:03:30.550 ssl_sock = ssl_wrap_socket(
2024-06-03 09:03:30.550 ^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550 File "/usr/local/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 465, in ssl_wrap_socket
2024-06-03 09:03:30.550 ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
2024-06-03 09:03:30.550 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550 File "/usr/local/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 509, in _ssl_wrap_socket_impl
2024-06-03 09:03:30.550 return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
2024-06-03 09:03:30.550 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550 File "/usr/local/lib/python3.12/ssl.py", line 455, in wrap_socket
2024-06-03 09:03:30.550 return self.sslsocket_class._create(
2024-06-03 09:03:30.550 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550 File "/usr/local/lib/python3.12/ssl.py", line 1042, in _create
2024-06-03 09:03:30.550 self.do_handshake()
2024-06-03 09:03:30.550 File "/usr/local/lib/python3.12/ssl.py", line 1320, in do_handshake
2024-06-03 09:03:30.550 self._sslobj.do_handshake()
2024-06-03 09:03:30.550 urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2024-06-03 09:03:30.550
2024-06-03 09:03:30.550 During handling of the above exception, another exception occurred:
2024-06-03 09:03:30.550
2024-06-03 09:03:30.550 Traceback (most recent call last):
2024-06-03 09:03:30.550 File "/usr/local/bin/watcher", line 583, in entrypoint
2024-06-03 09:03:30.550 sys.exit(runner.run())
2024-06-03 09:03:30.550 ^^^^^^^^^^^^
2024-06-03 09:03:30.550 File "/usr/local/bin/watcher", line 490, in run
2024-06-03 09:03:30.550 self.check_and_go()
2024-06-03 09:03:30.550 File "/usr/local/bin/watcher", line 427, in check_and_go
2024-06-03 09:03:30.550 last_modified = get_last_modified_for(url)
2024-06-03 09:03:30.550 ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550 File "/usr/local/bin/watcher", line 68, in get_last_modified_for
2024-06-03 09:03:30.550 with requests.head(url, allow_redirects=True) as resp:
2024-06-03 09:03:30.550 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550 File "/usr/local/lib/python3.12/site-packages/requests/api.py", line 100, in head
2024-06-03 09:03:30.550 return request("head", url, **kwargs)
2024-06-03 09:03:30.550 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550 File "/usr/local/lib/python3.12/site-packages/requests/api.py", line 59, in request
2024-06-03 09:03:30.550 return session.request(method=method, url=url, **kwargs)
2024-06-03 09:03:30.550 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550 File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
2024-06-03 09:03:30.550 resp = self.send(prep, **send_kwargs)
2024-06-03 09:03:30.550 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550 File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
2024-06-03 09:03:30.551 r = adapter.send(request, **kwargs)
2024-06-03 09:03:30.551 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.551 File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 579, in send
2024-06-03 09:03:30.551 raise ConnectionError(err, request=request)
2024-06-03 09:03:30.551 requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2024-06-03 09:04:19.656 [entrypoint] exposing s3_url secret as S3_URL
2024-06-03 09:04:19.657 [entrypoint] exposing zimfarm_username secret as ZIMFARM_USERNAME
2024-06-03 09:04:19.658 [entrypoint] exposing zimfarm_password secret as ZIMFARM_PASSWORD
2024-06-03 09:04:22.099 [2024-06-03 07:04:22,098::INFO]
2024-06-03 09:04:22.298 __ _ _
2024-06-03 09:04:22.298 / _| __ _ _ __ _ __ ___ __ ____ _| |_ ___| |__ ___ _ __
2024-06-03 09:04:22.298 | |_ / _` | '__| '_ ` _ \ ____\ \ /\ / / _` | __/ __| '_ \ / _ \ '__|
2024-06-03 09:04:22.298 | _| (_| | | | | | | | |_____\ V V / (_| | || (__| | | | __/ |
2024-06-03 09:04:22.298 |_| \__,_|_| |_| |_| |_| \_/\_/ \__,_|\__\___|_| |_|\___|_|
2024-06-03 09:04:22.298
2024-06-03 09:04:22.298
2024-06-03 09:04:22.298 [2024-06-03 07:04:22,099::INFO] Testing S3 credentials
2024-06-03 09:04:27.539 [2024-06-03 07:04:27,539::INFO] Testing Zimfarm credentials with https://api.farm.openzim.org/v1…
2024-06-03 09:04:27.651 [2024-06-03 07:04:27,651::INFO] Starting watcher:
2024-06-03 09:04:27.651 with zimfarm username: stackwatcher
2024-06-03 09:04:27.651 using cache: s3.us-west-1.wasabisys.com
2024-06-03 09:04:27.651 with bucket: org-kiwix-stackexchange
2024-06-03 09:04:27.651 [2024-06-03 07:04:27,651::INFO] Getting list of SE domains
2024-06-03 09:04:28.826 [2024-06-03 07:04:28,826::INFO] Grabbing online versions and comparing with S3…
2024-06-03 09:10:17.017 [2024-06-03 07:10:17,017::ERROR] FAILED. An error occurred: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2024-06-03 09:10:17.534 [2024-06-03 07:10:17,017::ERROR] ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2024-06-03 09:10:17.534 Traceback (most recent call last):
2024-06-03 09:10:17.534 File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 793, in urlopen
2024-06-03 09:10:17.534 response = self._make_request(
2024-06-03 09:10:17.534 ^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534 File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 491, in _make_request
2024-06-03 09:10:17.534 raise new_e
2024-06-03 09:10:17.534 File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 467, in _make_request
2024-06-03 09:10:17.534 self._validate_conn(conn)
2024-06-03 09:10:17.534 File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
2024-06-03 09:10:17.534 conn.connect()
2024-06-03 09:10:17.534 File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 653, in connect
2024-06-03 09:10:17.534 sock_and_verified = _ssl_wrap_socket_and_match_hostname(
2024-06-03 09:10:17.534 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534 File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 806, in _ssl_wrap_socket_and_match_hostname
2024-06-03 09:10:17.534 ssl_sock = ssl_wrap_socket(
2024-06-03 09:10:17.534 ^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534 File "/usr/local/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 465, in ssl_wrap_socket
2024-06-03 09:10:17.534 ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
2024-06-03 09:10:17.534 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534 File "/usr/local/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 509, in _ssl_wrap_socket_impl
2024-06-03 09:10:17.534 return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
2024-06-03 09:10:17.534 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534 File "/usr/local/lib/python3.12/ssl.py", line 455, in wrap_socket
2024-06-03 09:10:17.534 return self.sslsocket_class._create(
2024-06-03 09:10:17.534 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534 File "/usr/local/lib/python3.12/ssl.py", line 1042, in _create
2024-06-03 09:10:17.534 self.do_handshake()
2024-06-03 09:10:17.534 File "/usr/local/lib/python3.12/ssl.py", line 1320, in do_handshake
2024-06-03 09:10:17.534 self._sslobj.do_handshake()
2024-06-03 09:10:17.534 ConnectionResetError: [Errno 104] Connection reset by peer
2024-06-03 09:10:17.534
2024-06-03 09:10:17.534 During handling of the above exception, another exception occurred:
2024-06-03 09:10:17.534
2024-06-03 09:10:17.534 Traceback (most recent call last):
2024-06-03 09:10:17.534 File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 564, in send
2024-06-03 09:10:17.534 resp = conn.urlopen(
2024-06-03 09:10:17.534 ^^^^^^^^^^^^^
2024-06-03 09:10:17.534 File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 847, in urlopen
2024-06-03 09:10:17.534 retries = retries.increment(
2024-06-03 09:10:17.534 ^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534 File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 470, in increment
2024-06-03 09:10:17.534 raise reraise(type(error), error, _stacktrace)
2024-06-03 09:10:17.534 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534 File "/usr/local/lib/python3.12/site-packages/urllib3/util/util.py", line 38, in reraise
2024-06-03 09:10:17.534 raise value.with_traceback(tb)
2024-06-03 09:10:17.534 File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 793, in urlopen
2024-06-03 09:10:17.534 response = self._make_request(
2024-06-03 09:10:17.534 ^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534 File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 491, in _make_request
2024-06-03 09:10:17.534 raise new_e
2024-06-03 09:10:17.534 File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 467, in _make_request
2024-06-03 09:10:17.534 self._validate_conn(conn)
2024-06-03 09:10:17.534 File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
2024-06-03 09:10:17.534 conn.connect()
2024-06-03 09:10:17.534 File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 653, in connect
2024-06-03 09:10:17.534 sock_and_verified = _ssl_wrap_socket_and_match_hostname(
2024-06-03 09:10:17.534 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534 File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 806, in _ssl_wrap_socket_and_match_hostname
2024-06-03 09:10:17.534 ssl_sock = ssl_wrap_socket(
2024-06-03 09:10:17.534 ^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534 File "/usr/local/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 465, in ssl_wrap_socket
2024-06-03 09:10:17.534 ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
2024-06-03 09:10:17.534 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534 File "/usr/local/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 509, in _ssl_wrap_socket_impl
2024-06-03 09:10:17.534 return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
2024-06-03 09:10:17.534 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534 File "/usr/local/lib/python3.12/ssl.py", line 455, in wrap_socket
2024-06-03 09:10:17.534 return self.sslsocket_class._create(
2024-06-03 09:10:17.534 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534 File "/usr/local/lib/python3.12/ssl.py", line 1042, in _create
2024-06-03 09:10:17.534 self.do_handshake()
2024-06-03 09:10:17.534 File "/usr/local/lib/python3.12/ssl.py", line 1320, in do_handshake
2024-06-03 09:10:17.534 self._sslobj.do_handshake()
2024-06-03 09:10:17.534 urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2024-06-03 09:10:17.534
2024-06-03 09:10:17.534 During handling of the above exception, another exception occurred:
2024-06-03 09:10:17.534
2024-06-03 09:10:17.534 Traceback (most recent call last):
2024-06-03 09:10:17.534 File "/usr/local/bin/watcher", line 583, in entrypoint
2024-06-03 09:10:17.534 sys.exit(runner.run())
2024-06-03 09:10:17.534 ^^^^^^^^^^^^
2024-06-03 09:10:17.534 File "/usr/local/bin/watcher", line 490, in run
2024-06-03 09:10:17.534 self.check_and_go()
2024-06-03 09:10:17.534 File "/usr/local/bin/watcher", line 427, in check_and_go
2024-06-03 09:10:17.534 last_modified = get_last_modified_for(url)
2024-06-03 09:10:17.534 ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534 File "/usr/local/bin/watcher", line 68, in get_last_modified_for
2024-06-03 09:10:17.534 with requests.head(url, allow_redirects=True) as resp:
2024-06-03 09:10:17.534 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534 File "/usr/local/lib/python3.12/site-packages/requests/api.py", line 100, in head
2024-06-03 09:10:17.534 return request("head", url, **kwargs)
2024-06-03 09:10:17.534 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534 File "/usr/local/lib/python3.12/site-packages/requests/api.py", line 59, in request
2024-06-03 09:10:17.534 return session.request(method=method, url=url, **kwargs)
2024-06-03 09:10:17.534 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534 File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
2024-06-03 09:10:17.534 resp = self.send(prep, **send_kwargs)
2024-06-03 09:10:17.534 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534 File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
2024-06-03 09:10:17.534 r = adapter.send(request, **kwargs)
2024-06-03 09:10:17.534 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.535 File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 579, in send
2024-06-03 09:10:17.535 raise ConnectionError(err, request=request)
2024-06-03 09:10:17.535 requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2024-06-03 09:11:02.072 [entrypoint] exposing s3_url secret as S3_URL
2024-06-03 09:11:02.074 [entrypoint] exposing zimfarm_username secret as ZIMFARM_USERNAME
2024-06-03 09:11:02.075 [entrypoint] exposing zimfarm_password secret as ZIMFARM_PASSWORD
2024-06-03 09:11:19.567 [2024-06-03 07:11:19,567::INFO]
2024-06-03 09:11:19.567 __ _ _
2024-06-03 09:11:19.567 / _| __ _ _ __ _ __ ___ __ ____ _| |_ ___| |__ ___ _ __
2024-06-03 09:11:19.567 | |_ / _` | '__| '_ ` _ \ ____\ \ /\ / / _` | __/ __| '_ \ / _ \ '__|
2024-06-03 09:11:19.567 | _| (_| | | | | | | | |_____\ V V / (_| | || (__| | | | __/ |
2024-06-03 09:11:19.567 |_| \__,_|_| |_| |_| |_| \_/\_/ \__,_|\__\___|_| |_|\___|_|
2024-06-03 09:11:19.567
2024-06-03 09:11:19.567
2024-06-03 09:11:19.567 [2024-06-03 07:11:19,567::INFO] Testing S3 credentials
2024-06-03 09:11:35.117 [2024-06-03 07:11:35,116::INFO] Testing Zimfarm credentials with https://api.farm.openzim.org/v1…
2024-06-03 09:11:35.275 [2024-06-03 07:11:35,275::INFO] Starting watcher:
2024-06-03 09:11:35.275 with zimfarm username: stackwatcher
2024-06-03 09:11:35.275 using cache: s3.us-west-1.wasabisys.com
2024-06-03 09:11:35.275 with bucket: org-kiwix-stackexchange
2024-06-03 09:11:35.275 [2024-06-03 07:11:35,275::INFO] Getting list of SE domains
2024-06-03 09:11:37.900 [2024-06-03 07:11:37,899::INFO] Grabbing online versions and comparing with S3…
2024-06-03 09:21:05.560 [2024-06-03 07:21:05,463::INFO] All synced up.
2024-06-03 09:21:05.560 [2024-06-03 07:21:05,560::INFO] Done.
Not sure something is needed to be done since the system recover nicely ; maybe just avoid to restart completely the process.
List of all restarts in the last 30 days (not filtered by cause, some of them are expected due to updates typically)
2024-05-07 17:29:05
2024-05-13 10:38:41
2024-05-14 09:56:57
2024-05-14 11:17:08
2024-05-14 12:42:25
2024-05-14 13:52:54
2024-05-14 14:08:04
2024-05-15 04:08:13
2024-05-27 04:30:34
2024-05-28 08:12:46
2024-05-29 08:14:52
2024-05-30 08:16:38
2024-05-31 08:19:12
2024-06-01 08:20:42
2024-06-01 08:22:34
2024-06-01 08:24:48
2024-06-01 08:34:15
2024-06-01 08:44:18
2024-06-01 08:45:28
2024-06-01 08:55:54
2024-06-03 09:02:16
2024-06-03 09:04:22
2024-06-03 09:11:19
Still happening. Looks like this matches Wasabi S3 unavaibility. No change needed, systems recovers properly and automatically.