chia-blockchain icon indicating copy to clipboard operation
chia-blockchain copied to clipboard

Published 20 hours ago • verified publisher icon Chia-Network

[Bug] Ubuntu harvester terminates itself after any disruption

Open benjfranklin100 opened this issue 2 years ago • 16 comments

What happened?

I have two remote harvesters, and if I restart my farmer, temporarily lose network connection, use a different Ethernet port, etc., one of them (Ubuntu) goes offline and disappears from my farmer’s remote harvesters section. The other one (Windows) stays on and reconnects without any issues. Both are on same network, connected to the same icx6610 switch, etc. This has been going on the time I updated everything to 1.3.5 or so and has yet to resolve even now on 1.5.1. I have had this continue to occur after installing the gui client and cli-only client and using different installation methods. Everything runs just fine asides from the permanent termination of the harvester. Only solution is to manually restart the harvester.

Version

1.5.1, 1.6.0

What platform are you using?

Linux

What ui mode are you using?

CLI

Relevant log output

2022-09-02T19:12:36.058 harvester chia.harvester.harvester: INFO     16 plots were eligible for farming 7f7d2a6216... Found 0 proofs. Time: 0.47556 s. Total 6171 plots
2022-09-02T19:12:40.273 harvester harvester_server        : INFO     Connection closed: 192.168.1.22, node id: b84dd1c8a9d8610c799a98f51ccc02692887146e357810438ea96
2022-09-02T19:12:40.274 harvester chia.harvester.harvester: INFO     peer disconnected {'host': '192.168.1.22’, 'port': 8447}
2022-09-02T19:13:23.045 daemon chia.daemon.server         : ERROR    Ping error
Traceback (most recent call last):
  File "/home/user/chia-blockchain/chia/daemon/server.py", line 275, in ping_task
    await socket.ping()
  File "/home/user/chia-blockchain/venv/lib/python3.10/site-packages/aiohttp/web_ws.py", line 295, in ping
    await self._writer.ping(message)
  File "/home/user/chia-blockchain/venv/lib/python3.10/site-packages/aiohttp/http_websocket.py", line 676, in ping
    await self._send_frame(message, WSMsgType.PING)
  File "/home/user/chia-blockchain/venv/lib/python3.10/site-packages/aiohttp/http_websocket.py", line 653, in _send_frame
    self._write(header + message)
  File "/home/user/chia-blockchain/venv/lib/python3.10/site-packages/aiohttp/http_websocket.py", line 663, in _write
    raise ConnectionResetError("Cannot write to closing transport")
ConnectionResetError: Cannot write to closing transport
2022-09-02T19:13:23.047 daemon chia.daemon.server         : ERROR    Ping failed, connection closed. 

<the above is the last thing that was logged after I had restarted the farmer. Below is what was logged after restarting the harvester> 

2022-09-02T19:16:05.489 daemon chia.daemon.server         : INFO     sending term signal to chia_harvester
2022-09-02T19:16:20.512 daemon chia.daemon.server         : INFO     sending kill signal to chia_harvester
2022-09-02T19:16:20.959 daemon chia.daemon.server         : INFO     process chia_harvester returned -9
2022-09-02T19:16:20.961 daemon chia.daemon.server         : INFO     Daemon Server stopping, Services stopped: ['chia_harvester']
2022-09-02T19:16:20.964 daemon chia.daemon.server         : INFO     chia daemon exiting
2022-09-02T19:16:20.965 daemon chia.daemon.server         : INFO     Daemon WebSocketServer closed 
2022-09-02T19:16:23.479 daemon chia.daemon.server         : INFO     chia-blockchain version: 1.5.1
2022-09-02T19:16:23.532 daemon chia.daemon.server         : INFO     Starting Daemon Server
2022-09-02T19:16:25.528 daemon chia.daemon.server         : INFO     ConnectionClosed. Closing websocket with Unknown
2022-09-02T19:16:26.242 harvester chia.harvester.harvester: INFO     Using plots_refresh_parameter: {'batch_size': 300, 'batch_sleep_milliseconds': 1, 'interval_seconds': 120, 'retry_invalid_seconds': 1200}
2022-09-02T19:16:26.242 harvester harvester               : INFO     chia-blockchain version: 1.5.1
2022-09-02T19:16:26.243 harvester harvester_server        : INFO     Service capabilities: [(1, '1'), (2, '1'), (3, '1')]
2022-09-02T19:16:26.245 harvester harvester               : INFO     Started harvester service on network_id: mainnet
2022-09-02T19:16:26.245 harvester harvester               : INFO     Reconnecting to peer {'host': '192.168.1.22', 'port': 8447}
2022-09-02T19:16:26.270 harvester harvester_server        : INFO     Connected with farmer {'host': '192.168.1.22', 'port': 8447}
2022-09-02T19:16:26.284 harvester chia.plotting.cache     : INFO     Loaded 970540 bytes of cached data
2022-09-02T19:16:26.317 daemon chia.daemon.server         : INFO     Register service {'service': 'chia_harvester'}
2022-09-02T19:16:26.318 daemon chia.daemon.server         : INFO     registered for service chia_harvester
2022-09-02T19:16:26.319 daemon chia.daemon.server         : INFO     {'success': True}
2022-09-02T19:16:26.604 harvester chia.plotting.manager   : INFO     Found plot /media/chia/1/…..

benjfranklin100 avatar Sep 04 '22 23:09 benjfranklin100

Several times a day... not sure what to do...

maxcom99 avatar Sep 13 '22 05:09 maxcom99

@benjfranklin100 Can you please provide logs from that harvester with DEBUG log level?

xdustinface avatar Sep 14 '22 21:09 xdustinface

@xdustinface sure thing:

2022-09-14T16:31:56.703 daemon chia.daemon.server : DEBUG About to ping: chia_harvester 2022-09-14T16:32:03.221 harvester harvester_server : DEBUG <- new_signage_point_harvester from peer b84dd1c8a9d8610c799a98f51ccc02692887146e357810438ea96 192.168.1.22 2022-09-14T16:32:03.657 harvester chia.harvester.harvester: INFO 9 plots were eligible for farming e3372edd22... Found 0 proofs. Time: 0.43561 s. Total 6171 plots 2022-09-14T16:32:03.658 harvester harvester_server : DEBUG -> farming_info to peer 192.168.1.22 b84dd1c8a9d8610c799a98f51ccc02692887146e357810438ea96 2022-09-14T16:32:03.659 harvester harvester_server : DEBUG Time taken to process new_signage_point_harvester from b84dd1c8a9d8610c799a98f51ccc02692887146e357810438ea96 is 0.43825721740722656 seconds 2022-09-14T16:32:03.659 daemon chia.daemon.server : DEBUG Received message: WSMessage(type=<WSMsgType.TEXT: 1>, data='{"ack": false, "command": "farming_info", "data": {"challenge_hash": "e3372edd227194e2f9c2108867340a6520b29ca7e86c8dce003425643a15e7e2", "eligible_plots": 9, "found_proofs": 0, "success": true, "time": 0.4356060028076172, "total_plots": 6171}, "destination": "metrics", "origin": "chia_harvester", "request_id": "01296eee040aa0f5f47e2d3703736b20ed22062edfac011e1613d15ad35da218"}', extra='') 2022-09-14T16:32:12.889 harvester harvester_server : DEBUG <- new_signage_point_harvester from peer b84dd1c8a9d8610c799a98f51ccc02692887146e357810438ea96 192.168.1.22 2022-09-14T16:32:13.428 harvester chia.harvester.harvester: INFO 11 plots were eligible for farming e3372edd22... Found 0 proofs. Time: 0.53804 s. Total 6171 plots 2022-09-14T16:32:13.429 harvester harvester_server : DEBUG -> farming_info to peer 192.168.1.22 b84dd1c8a9d8610c799a98f51ccc02692887146e357810438ea96 2022-09-14T16:32:13.429 harvester harvester_server : DEBUG Time taken to process new_signage_point_harvester from b84dd1c8a9d8610c799a98f51ccc02692887146e357810438ea96 is 0.5399014949798584 seconds 2022-09-14T16:32:13.430 daemon chia.daemon.server : DEBUG Received message: WSMessage(type=<WSMsgType.TEXT: 1>, data='{"ack": false, "command": "farming_info", "data": {"challenge_hash": "e3372edd227194e2f9c2108867340a6520b29ca7e86c8dce003425643a15e7e2", "eligible_plots": 11, "found_proofs": 0, "success": true, "time": 0.538038969039917, "total_plots": 6171}, "destination": "metrics", "origin": "chia_harvester", "request_id": "8995adbd24cce86fce9e3b11fae39b4ff9fda27d1645c096a09c480d3e038994"}', extra='') 2022-09-14T16:32:21.624 harvester harvester_server : DEBUG <- new_signage_point_harvester from peer b84dd1c8a9d8610c799a98f51ccc02692887146e357810438ea96 192.168.1.22 2022-09-14T16:32:21.995 harvester chia.harvester.harvester: INFO 16 plots were eligible for farming e3372edd22... Found 0 proofs. Time: 0.37004 s. Total 6171 plots 2022-09-14T16:32:21.995 harvester harvester_server : DEBUG -> farming_info to peer 192.168.1.22 b84dd1c8a9d8610c799a98f51ccc02692887146e357810438ea96 2022-09-14T16:32:21.996 harvester harvester_server : DEBUG Time taken to process new_signage_point_harvester from b84dd1c8a9d8610c799a98f51ccc02692887146e357810438ea96 is 0.37210893630981445 seconds 2022-09-14T16:32:21.996 daemon chia.daemon.server : DEBUG Received message: WSMessage(type=<WSMsgType.TEXT: 1>, data='{"ack": false, "command": "farming_info", "data": {"challenge_hash": "e3372edd227194e2f9c2108867340a6520b29ca7e86c8dce003425643a15e7e2", "eligible_plots": 16, "found_proofs": 0, "success": true, "time": 0.3700413703918457, "total_plots": 6171}, "destination": "metrics", "origin": "chia_harvester", "request_id": "53a952e5f66095ac6a811d919594e24c776402193937b78d14da71e29b633078"}', extra='') 2022-09-14T16:32:26.709 daemon chia.daemon.server : DEBUG About to ping: chia_harvester 2022-09-14T16:32:30.360 harvester harvester_server : DEBUG <- new_signage_point_harvester from peer b84dd1c8a9d8610c799a98f51ccc02692887146e357810438ea96 192.168.1.22 2022-09-14T16:32:30.621 harvester harvester_server : DEBUG Peer closed connection farmer 192.168.1.22:8447/8447 2022-09-14T16:32:30.622 harvester harvester_server : DEBUG Inbound_handler task cancelled 2022-09-14T16:32:30.623 harvester harvester_server : INFO Connection closed: 192.168.1.22, node id: b84dd1c8a9d8610c799a98f51ccc02692887146e357810438ea96 2022-09-14T16:32:30.623 harvester chia.harvester.harvester: INFO peer disconnected {'host': '192.168.1.22', 'port': 8447} 2022-09-14T16:32:56.741 daemon chia.daemon.server : DEBUG About to ping: chia_harvester 2022-09-14T16:33:26.757 daemon chia.daemon.server : DEBUG About to ping: chia_harvester 2022-09-14T16:33:26.757 daemon chia.daemon.server : ERROR Ping error Traceback (most recent call last): File "/home/user/chia-blockchain/chia/daemon/server.py", line 275, in ping_task await socket.ping() File "/home/user/chia-blockchain/venv/lib/python3.10/site-packages/aiohttp/web_ws.py", line 295, in ping await self._writer.ping(message) File "/home/user/chia-blockchain/venv/lib/python3.10/site-packages/aiohttp/http_websocket.py", line 676, in ping await self._send_frame(message, WSMsgType.PING) File "/home/user/chia-blockchain/venv/lib/python3.10/site-packages/aiohttp/http_websocket.py", line 653, in _send_frame self._write(header + message) File "/home/user/chia-blockchain/venv/lib/python3.10/site-packages/aiohttp/http_websocket.py", line 663, in _write raise ConnectionResetError("Cannot write to closing transport") ConnectionResetError: Cannot write to closing transport 2022-09-14T16:33:26.759 daemon chia.daemon.server : ERROR Ping failed, connection closed.

<The above is the last thing that was logged in the harvester log file after I had restarted the farmer. Below is what was logged in the harvester log file after restarting the harvester. The entries go up until it started registering/finding all of the plots in the directories. It proceeds to work properly.>

2022-09-14T16:34:06.255 daemon chia.daemon.server : DEBUG Received message: WSMessage(type=<WSMsgType.TEXT: 1>, data='{"ack": false, "command": "ping", "data": {}, "destination": "daemon", "origin": "client", "request_id": "f1a4dbcaabe119a62ed58138199ebb21601260a6ee7d6dfd6a4d5845947fbdef"}', extra='') 2022-09-14T16:34:06.256 daemon chia.daemon.server : DEBUG Received message: WSMessage(type=<WSMsgType.TEXT: 1>, data='{"ack": false, "command": "exit", "data": {}, "destination": "daemon", "origin": "client", "request_id": "8050ed2fc0f264f53137d9cab9f4255705b3102da97d6b14b1cda3b33cd9fd7c"}', extra='') 2022-09-14T16:34:06.256 daemon chia.daemon.server : INFO sending term signal to chia_harvester 2022-09-14T16:34:21.278 daemon chia.daemon.server : INFO sending kill signal to chia_harvester 2022-09-14T16:34:21.804 daemon chia.daemon.server : INFO process chia_harvester returned -9 2022-09-14T16:34:21.805 daemon chia.daemon.server : INFO Daemon Server stopping, Services stopped: ['chia_harvester'] 2022-09-14T16:34:21.807 daemon chia.daemon.server : INFO chia daemon exiting 2022-09-14T16:34:21.808 daemon chia.daemon.server : INFO Daemon WebSocketServer closed 2022-09-14T16:34:21.808 daemon filelock : DEBUG Attempting to release lock 140268371844864 on /home/user/.chia/mainnet/run/daemon.lock 2022-09-14T16:34:21.808 daemon filelock : DEBUG Lock 140268371844864 released on /home/user/.chia/mainnet/run/daemon.lock 2022-09-14T16:34:24.285 daemon filelock : DEBUG Attempting to acquire lock 139969508795600 on /home/user/.chia/mainnet/run/daemon.lock 2022-09-14T16:34:24.285 daemon filelock : DEBUG Lock 139969508795600 acquired on /home/user/.chia/mainnet/run/daemon.lock 2022-09-14T16:34:24.285 daemon chia.daemon.server : INFO chia-blockchain version: 1.5.1 2022-09-14T16:34:24.286 daemon filelock : DEBUG Attempting to acquire lock 139969508797952 on /home/user/.chia/mainnet/config/config.yaml.lock 2022-09-14T16:34:24.286 daemon filelock : DEBUG Lock 139969508797952 acquired on /home/user/.chia/mainnet/config/config.yaml.lock 2022-09-14T16:34:24.333 daemon filelock : DEBUG Attempting to release lock 139969508797952 on /home/user/.chia/mainnet/config/config.yaml.lock 2022-09-14T16:34:24.334 daemon filelock : DEBUG Lock 139969508797952 released on /home/user/.chia/mainnet/config/config.yaml.lock 2022-09-14T16:34:24.338 daemon chia.daemon.server : INFO Starting Daemon Server 2022-09-14T16:34:26.316 daemon chia.daemon.server : DEBUG Received message: WSMessage(type=<WSMsgType.TEXT: 1>, data='{"ack": false, "command": "ping", "data": {}, "destination": "daemon", "origin": "client", "request_id": "ec87fb01fff1626f1d4e7d8b95c4af1d2ad1901921077f4411a4602ef6e1e588"}', extra='') 2022-09-14T16:34:26.319 daemon chia.daemon.server : DEBUG Received message: WSMessage(type=<WSMsgType.TEXT: 1>, data='{"ack": false, "command": "is_keyring_locked", "data": {}, "destination": "daemon", "origin": "client", "request_id": "cb0f2b062c89c6b192a69fbc6b2c16894d0f01b7390a079b7d4ac70e41b2c656"}', extra='') 2022-09-14T16:34:26.344 daemon chia.daemon.server : DEBUG Received message: WSMessage(type=<WSMsgType.TEXT: 1>, data='{"ack": false, "command": "is_running", "data": {"service": "chia_harvester"}, "destination": "daemon", "origin": "client", "request_id": "241b6315c8a761428f229bd942ad70d8337caa6dc36ee651372e1eff285667d4"}', extra='') 2022-09-14T16:34:26.345 daemon chia.daemon.server : DEBUG Received message: WSMessage(type=<WSMsgType.TEXT: 1>, data='{"ack": false, "command": "start_service", "data": {"service": "chia_harvester"}, "destination": "daemon", "origin": "client", "request_id": "917d048709ac73806f283ad258d6dea04fbb030684cfac9e7b76ef8614b1b38f"}', extra='') 2022-09-14T16:34:26.346 daemon chia.daemon.server : DEBUG Launching service with CHIA_ROOT: /home/user/.chia/mainnet 2022-09-14T16:34:26.348 daemon chia.daemon.server : DEBUG Received message: WSMessage(type=<WSMsgType.CLOSE: 8>, data=1000, extra='') 2022-09-14T16:34:26.348 daemon chia.daemon.server : INFO ConnectionClosed. Closing websocket with Unknown 2022-09-14T16:34:27.046 harvester chia.harvester.harvester: INFO Using plots_refresh_parameter: {'batch_size': 300, 'batch_sleep_milliseconds': 1, 'interval_seconds': 120, 'retry_invalid_seconds': 1200} 2022-09-14T16:34:27.047 harvester harvester : INFO chia-blockchain version: 1.5.1 2022-09-14T16:34:27.047 harvester harvester_server : INFO Service capabilities: [(1, '1'), (2, '1'), (3, '1')] 2022-09-14T16:34:27.049 harvester filelock : DEBUG Attempting to acquire lock 140128955022880 on /home/user/.chia/mainnet/run/harvester.lock 2022-09-14T16:34:27.049 harvester filelock : DEBUG Lock 140128955022880 acquired on /home/user/.chia/mainnet/run/harvester.lock 2022-09-14T16:34:27.050 harvester harvester : INFO Started harvester service on network_id: mainnet 2022-09-14T16:34:27.050 harvester harvester : INFO Reconnecting to peer {'host': '192.168.1.22', 'port': 8447} 2022-09-14T16:34:27.055 harvester harvester_server : DEBUG Connecting: wss://192.168.1.22:8447/ws, Peer info: {'host': '192.168.1.22', 'port': 8447} 2022-09-14T16:34:27.075 harvester harvester_server : DEBUG -> handshake to peer 192.168.1.22 b84dd1c8a9d8610c799a98f51ccc02692887146e357810438ea96 2022-09-14T16:34:27.076 harvester harvester_server : INFO Connected with farmer {'host': '192.168.1.22', 'port': 8447} 2022-09-14T16:34:27.076 harvester harvester_server : DEBUG <- harvester_handshake from peer b84dd1c8a9d8610c799a98f51ccc02692887146e357810438ea96 192.168.1.22 2022-09-14T16:34:27.090 harvester chia.plotting.cache : INFO Loaded 970540 bytes of cached data 2022-09-14T16:34:27.116 harvester filelock : DEBUG Attempting to acquire lock 140128955007040 on /home/user/.chia/mainnet/config/config.yaml.lock 2022-09-14T16:34:27.117 harvester harvester_server : DEBUG Time taken to process harvester_handshake from b84dd1c8a9d8610c799a98f51ccc02692887146e357810438ea96 is 0.04101443290710449 seconds 2022-09-14T16:34:27.118 harvester filelock : DEBUG Lock 140128955007040 acquired on /home/user/.chia/mainnet/config/config.yaml.lock 2022-09-14T16:34:27.118 daemon chia.daemon.server : DEBUG Received message: WSMessage(type=<WSMsgType.TEXT: 1>, data='{"ack": false, "command": "register_service", "data": {"service": "chia_harvester"}, "destination": "daemon", "origin": "chia_harvester", "request_id": "255e9afa08037672fa93cadc37833b028c721ddf48c1edcbe1fe6fcc67c000be"}', extra='') 2022-09-14T16:34:27.119 daemon chia.daemon.server : INFO Register service {'service': 'chia_harvester'} 2022-09-14T16:34:27.119 daemon chia.daemon.server : INFO registered for service chia_harvester 2022-09-14T16:34:27.120 daemon chia.daemon.server : INFO {'success': True} 2022-09-14T16:34:27.145 harvester chia.rpc.rpc_server : DEBUG Rpc call <- register_service 2022-09-14T16:34:27.167 harvester filelock : DEBUG Attempting to release lock 140128955007040 on /home/user/.chia/mainnet/config/config.yaml.lock 2022-09-14T16:34:27.168 harvester filelock : DEBUG Lock 140128955007040 released on /home/user/.chia/mainnet/config/config.yaml.lock 2022-09-14T16:34:27.171 harvester chia.plotting.util : DEBUG get_filenames: 110 files found in /media/chia/1, recursive: False 2022-09-14T16:34:27.175 harvester chia.plotting.util : DEBUG get_filenames: 110 files found in /media/chia/2, recursive: False 2022-09-14T16:34:27.178 harvester chia.plotting.util : DEBUG get_filenames: 110 files found in /media/chia/3, recursive: False 2022-09-14T16:34:27.182 harvester chia.plotting.util : DEBUG get_filenames: 110 files found in /media/chia/4, recursive: False 2022-09-14T16:34:27.193 harvester chia.plotting.util : DEBUG get_filenames: 110 files found in /media/chia/5, recursive: False 2022-09-14T16:34:27.196 harvester chia.plotting.util : DEBUG get_filenames: 110 files found in /media/chia/6, recursive: False 2022-09-14T16:34:27.200 harvester chia.plotting.util : DEBUG get_filenames: 110 files found in /media/chia/7, recursive: False 2022-09-14T16:34:27.203 harvester chia.plotting.util : DEBUG get_filenames: 110 files found in /media/chia/8, recursive: False 2022-09-14T16:34:27.207 harvester chia.plotting.util : DEBUG get_filenames: 110 files found in /media/chia/9, recursive: False 2022-09-14T16:34:27.211 harvester chia.plotting.util : DEBUG get_filenames: 128 files found in /media/chia/10, recursive: False 2022-09-14T16:34:27.215 harvester chia.plotting.util : DEBUG get_filenames: 147 files found in /media/chia/11, recursive: False 2022-09-14T16:34:27.220 harvester chia.plotting.util : DEBUG get_filenames: 147 files found in /media/chia/12, recursive: False 2022-09-14T16:34:27.224 harvester chia.plotting.util : DEBUG get_filenames: 147 files found in /media/chia/13, recursive: False 2022-09-14T16:34:27.229 harvester chia.plotting.util : DEBUG get_filenames: 147 files found in /media/chia/14, recursive: False 2022-09-14T16:34:27.233 harvester chia.plotting.util : DEBUG get_filenames: 128 files found in /media/chia/15, recursive: False 2022-09-14T16:34:27.237 harvester chia.plotting.util : DEBUG get_filenames: 128 files found in /media/chia/16, recursive: False 2022-09-14T16:34:27.241 harvester chia.plotting.util : DEBUG get_filenames: 128 files found in /media/chia/17, recursive: False 2022-09-14T16:34:27.244 harvester chia.plotting.util : DEBUG get_filenames: 128 files found in /media/chia/18, recursive: False 2022-09-14T16:34:27.248 harvester chia.plotting.util : DEBUG get_filenames: 128 files found in /media/chia/19, recursive: False 2022-09-14T16:34:27.252 harvester chia.plotting.util : DEBUG get_filenames: 128 files found in /media/chia/20, recursive: False 2022-09-14T16:34:27.256 harvester chia.plotting.util : DEBUG get_filenames: 128 files found in /media/chia/21, recursive: False 2022-09-14T16:34:27.260 harvester chia.plotting.util : DEBUG get_filenames: 128 files found in /media/chia/22, recursive: False 2022-09-14T16:34:27.264 harvester chia.plotting.util : DEBUG get_filenames: 128 files found in /media/chia/23, recursive: False 2022-09-14T16:34:27.268 harvester chia.plotting.util : DEBUG get_filenames: 128 files found in /media/chia/24, recursive: False 2022-09-14T16:34:27.271 harvester chia.plotting.util : DEBUG get_filenames: 128 files found in /media/chia/25, recursive: False 2022-09-14T16:34:27.275 harvester chia.plotting.util : DEBUG get_filenames: 128 files found in /media/chia/26, recursive: False 2022-09-14T16:34:27.279 harvester chia.plotting.util : DEBUG get_filenames: 128 files found in /media/chia/27, recursive: False 2022-09-14T16:34:27.283 harvester chia.plotting.util : DEBUG get_filenames: 128 files found in /media/chia/28, recursive: False 2022-09-14T16:34:27.287 harvester chia.plotting.util : DEBUG get_filenames: 128 files found in /media/chia/29, recursive: False 2022-09-14T16:34:27.291 harvester chia.plotting.util : DEBUG get_filenames: 128 files found in /media/chia/30, recursive: False 2022-09-14T16:34:27.295 harvester chia.plotting.util : DEBUG get_filenames: 128 files found in /media/chia/31, recursive: False 2022-09-14T16:34:27.298 harvester chia.plotting.util : DEBUG get_filenames: 128 files found in /media/chia/32, recursive: False 2022-09-14T16:34:27.302 harvester chia.plotting.util : DEBUG get_filenames: 128 files found in /media/chia/33, recursive: False 2022-09-14T16:34:27.306 harvester chia.plotting.util : DEBUG get_filenames: 128 files found in /media/chia/34, recursive: False 2022-09-14T16:34:27.310 harvester chia.plotting.util : DEBUG get_filenames: 128 files found in /media/chia/35, recursive: False 2022-09-14T16:34:27.313 harvester chia.plotting.util : DEBUG get_filenames: 128 files found in /media/chia/36, recursive: False 2022-09-14T16:34:27.317 harvester chia.plotting.util : DEBUG get_filenames: 128 files found in /media/chia/37, recursive: False 2022-09-14T16:34:27.318 harvester chia.plotting.util : DEBUG get_filenames: 9 files found in /media/chia/1tb1, recursive: False 2022-09-14T16:34:27.319 harvester chia.plotting.util : DEBUG get_filenames: 9 files found in /media/chia/1tb2, recursive: False 2022-09-14T16:34:27.320 harvester chia.plotting.util : DEBUG get_filenames: 18 files found in /media/chia/2tb1, recursive: False 2022-09-14T16:34:27.321 harvester chia.plotting.util : DEBUG get_filenames: 18 files found in /media/chia/2tb2, recursive: False 2022-09-14T16:34:27.322 harvester chia.plotting.util : DEBUG get_filenames: 18 files found in /media/chia/2tb3, recursive: False 2022-09-14T16:34:27.323 harvester chia.plotting.util : DEBUG get_filenames: 18 files found in /media/chia/2tb4, recursive: False 2022-09-14T16:34:27.324 harvester chia.plotting.util : DEBUG get_filenames: 18 files found in /media/chia/2tb5, recursive: False 2022-09-14T16:34:27.324 harvester chia.plotting.util : DEBUG get_filenames: 18 files found in /media/chia/2tb6, recursive: False 2022-09-14T16:34:27.325 harvester chia.plotting.util : DEBUG get_filenames: 18 files found in /media/chia/2tb7, recursive: False 2022-09-14T16:34:27.326 harvester chia.plotting.util : DEBUG get_filenames: 18 files found in /media/chia/2tb8, recursive: False 2022-09-14T16:34:27.327 harvester chia.plotting.util : DEBUG get_filenames: 18 files found in /media/chia/2tb9, recursive: False 2022-09-14T16:34:27.328 harvester chia.plotting.util : DEBUG get_filenames: 18 files found in /media/chia/2tb10, recursive: False 2022-09-14T16:34:27.330 harvester chia.plotting.util : DEBUG get_filenames: 27 files found in /media/chia/3tb1, recursive: False 2022-09-14T16:34:27.331 harvester chia.plotting.util : DEBUG get_filenames: 27 files found in /media/chia/3tb2, recursive: False 2022-09-14T16:34:27.333 harvester chia.plotting.util : DEBUG get_filenames: 36 files found in /media/chia/4tb1, recursive: False 2022-09-14T16:34:27.334 harvester chia.plotting.util : DEBUG get_filenames: 36 files found in /media/chia/4tb2, recursive: False 2022-09-14T16:34:27.336 harvester chia.plotting.util : DEBUG get_filenames: 36 files found in /media/chia/4tb3, recursive: False 2022-09-14T16:34:27.337 harvester chia.plotting.util : DEBUG get_filenames: 36 files found in /media/chia/4tb4, recursive: False 2022-09-14T16:34:27.339 harvester chia.plotting.util : DEBUG get_filenames: 45 files found in /media/chia/5tb1, recursive: False 2022-09-14T16:34:27.340 harvester chia.plotting.util : DEBUG get_filenames: 45 files found in /media/chia/5tb2, recursive: False 2022-09-14T16:34:27.342 harvester chia.plotting.util : DEBUG get_filenames: 45 files found in /media/chia/5tb3, recursive: False 2022-09-14T16:34:27.344 harvester chia.plotting.util : DEBUG get_filenames: 45 files found in /media/chia/5tb4, recursive: False 2022-09-14T16:34:27.346 harvester chia.plotting.util : DEBUG get_filenames: 45 files found in /media/chia/5tb5, recursive: False 2022-09-14T16:34:27.347 harvester chia.plotting.util : DEBUG get_filenames: 45 files found in /media/chia/5tb6, recursive: False 2022-09-14T16:34:27.349 harvester chia.plotting.util : DEBUG get_filenames: 45 files found in /media/chia/5tb7, recursive: False 2022-09-14T16:34:27.351 harvester chia.plotting.util : DEBUG get_filenames: 45 files found in /media/chia/5tb8, recursive: False 2022-09-14T16:34:27.352 harvester chia.plotting.util : DEBUG get_filenames: 45 files found in /media/chia/5tb9, recursive: False 2022-09-14T16:34:27.354 harvester chia.plotting.util : DEBUG get_filenames: 45 files found in /media/chia/5tb10, recursive: False 2022-09-14T16:34:27.356 harvester chia.plotting.util : DEBUG get_filenames: 45 files found in /media/chia/5tb11, recursive: False 2022-09-14T16:34:27.358 harvester chia.plotting.util : DEBUG get_filenames: 45 files found in /media/chia/5tb12, recursive: False 2022-09-14T16:34:27.359 harvester chia.plotting.util : DEBUG get_filenames: 45 files found in /media/chia/5tb13, recursive: False 2022-09-14T16:34:27.361 harvester chia.plotting.util : DEBUG get_filenames: 45 files found in /media/chia/5tb14, recursive: False 2022-09-14T16:34:27.363 harvester chia.plotting.util : DEBUG get_filenames: 45 files found in /media/chia/5tb15, recursive: False 2022-09-14T16:34:27.365 harvester chia.plotting.util : DEBUG get_filenames: 45 files found in /media/chia/5tb16, recursive: False 2022-09-14T16:34:27.366 harvester chia.plotting.util : DEBUG get_filenames: 45 files found in /media/chia/5tb17, recursive: False 2022-09-14T16:34:27.369 harvester chia.plotting.util : DEBUG get_filenames: 45 files found in /media/chia/5tb18, recursive: False 2022-09-14T16:34:27.371 harvester chia.plotting.util : DEBUG get_filenames: 45 files found in /media/chia/5tb19, recursive: False 2022-09-14T16:34:27.373 harvester chia.plotting.util : DEBUG get_filenames: 45 files found in /media/chia/5tb20, recursive: False 2022-09-14T16:34:27.374 harvester chia.plotting.util : DEBUG get_filenames: 45 files found in /media/chia/5tb21, recursive: False 2022-09-14T16:34:27.376 harvester chia.plotting.util : DEBUG get_filenames: 45 files found in /media/chia/5tb22, recursive: False 2022-09-14T16:34:27.378 harvester chia.plotting.util : DEBUG get_filenames: 45 files found in /media/chia/5tb23, recursive: False 2022-09-14T16:34:27.379 harvester chia.plotting.util : DEBUG get_filenames: 45 files found in /media/chia/5tb24, recursive: False 2022-09-14T16:34:27.381 harvester chia.plotting.util : DEBUG get_filenames: 45 files found in /media/chia/5tb25, recursive: False 2022-09-14T16:34:27.394 harvester chia.harvester.harvester: DEBUG _plot_refresh_callback: event started, loaded 0, removed 0, processed 0, remaining 6171, duration: 0.00 seconds, total plots: 0 2022-09-14T16:34:27.395 harvester chia.plot_sync.sender : DEBUG sync_start sync_id 0, next_message_id 0, messages 0: count 6171, initial True 2022-09-14T16:34:27.395 harvester chia.plot_sync.sender : DEBUG sync_start 1663191267 2022-09-14T16:34:27.423 harvester chia.plot_sync.sender : DEBUG _send_next_message send plot_sync_start: PlotSyncStart: identifier {'message_id': 0, 'sync_id': 1663191267, 'timestamp': 1663191267}, initial True, last_sync_id 0, plot_file_count 6171 2022-09-14T16:34:27.431 harvester chia.plotting.manager : DEBUG refresh_batch: 300 files in directories {PosixPath('/media/chia/5tb15'), PosixPath('/media/chia/2tb1'), PosixPath('/media/chia/20'), PosixPath('/media/chia/5tb17'), PosixPath('/media/chia/37'), PosixPath('/media/chia/2tb8'), PosixPath('/media/chia/8'), PosixPath('/media/chia/6'), PosixPath('/media/chia/5tb24'), PosixPath('/media/chia/18'), PosixPath('/media/chia/28'), PosixPath('/media/chia/13'), PosixPath('/media/chia/5tb21'), PosixPath('/media/chia/5tb23'), PosixPath('/media/chia/17'), PosixPath('/media/chia/26'), PosixPath('/media/chia/5tb2'), PosixPath('/media/chia/5tb22'), PosixPath('/media/chia/24'), PosixPath('/media/chia/22'), PosixPath('/media/chia/2tb6'), PosixPath('/media/chia/30'), PosixPath('/media/chia/16'), PosixPath('/media/chia/25'), PosixPath('/media/chia/4tb2'), PosixPath('/media/chia/10'), PosixPath('/media/chia/2tb2'), PosixPath('/media/chia/5tb6'), PosixPath('/media/chia/5tb10'), PosixPath('/media/chia/9'), PosixPath('/media/chia/5tb7'), PosixPath('/media/chia/19'), PosixPath('/media/chia/2tb3'), PosixPath('/media/chia/4tb1'), PosixPath('/media/chia/5tb1'), PosixPath('/media/chia/3tb1'), PosixPath('/media/chia/2'), PosixPath('/media/chia/27'), PosixPath('/media/chia/14'), PosixPath('/media/chia/1'), PosixPath('/media/chia/36'), PosixPath('/media/chia/5'), PosixPath('/media/chia/32'), PosixPath('/media/chia/2tb7'), PosixPath('/media/chia/3tb2'), PosixPath('/media/chia/2tb9'), PosixPath('/media/chia/4tb4'), PosixPath('/media/chia/1tb1'), PosixPath('/media/chia/5tb18'), PosixPath('/media/chia/5tb5'), PosixPath('/media/chia/5tb4'), PosixPath('/media/chia/31'), PosixPath('/media/chia/5tb11'), PosixPath('/media/chia/2tb5'), PosixPath('/media/chia/5tb13'), PosixPath('/media/chia/2tb10'), PosixPath('/media/chia/21'), PosixPath('/media/chia/15'), PosixPath('/media/chia/5tb20'), PosixPath('/media/chia/5tb12'), PosixPath('/media/chia/34'), PosixPath('/media/chia/5tb19'), PosixPath('/media/chia/12'), PosixPath('/media/chia/3'), PosixPath('/media/chia/2tb4'), PosixPath('/media/chia/5tb16'), PosixPath('/media/chia/4tb3'), PosixPath('/media/chia/23'), PosixPath('/media/chia/33'), PosixPath('/media/chia/5tb14'), PosixPath('/media/chia/5tb8'), PosixPath('/media/chia/5tb9'), PosixPath('/media/chia/5tb25'), PosixPath('/media/chia/29'), PosixPath('/media/chia/35'), PosixPath('/media/chia/4'), PosixPath('/media/chia/5tb3'), PosixPath('/media/chia/7'), PosixPath('/media/chia/11'), PosixPath('/media/chia/1tb2')} 2022-09-14T16:34:27.432 harvester harvester_server : DEBUG -> plot_sync_start to peer 192.168.1.22 b84dd1c8a9d8610c799a98f51ccc02692887146e357810438ea96 2022-09-14T16:34:27.433 harvester chia.plotting.manager : INFO Found plot /media/chia/1/plot-k32-2022-01-22-02-11-dd353742cc9543502acc8d25c33e2dda2fbaff7d5b2c315a77359852c519cdf9.plot of size 32, cache_hit: True

benjfranklin100 avatar Sep 14 '22 22:09 benjfranklin100

I've noticed that the docker containers die permanently but the same exact setup if running on a normal box will continue along after killing the socket. Just in case this helps, this was my workaround.

maxcom99 avatar Sep 14 '22 22:09 maxcom99

Hm, that's interesting. Unfortunately I don't think it applies in this case as it's running on Ubuntu installed onto the bare metal.

benjfranklin100 avatar Sep 14 '22 22:09 benjfranklin100

Hm, that's interesting. Unfortunately I don't think it applies in this case as it's running on Ubuntu installed onto the bare metal.

Strange. Mine has been working once on a VM. I still see socket closed but it continues. I think it might be related to throwing an error vs a warning.

maxcom99 avatar Sep 14 '22 23:09 maxcom99

I checked the windows machine log to see what it's been doing during these farmer restarts, and it just keeps on trying to connect and it eventually does once the farmer restarts.

2022-09-14T18:10:01.235 harvester harvester_server : INFO Connection closed: 192.168.1.22, node id: b84dd1c8a9d8610c799a98f51ccc02692887146e357810438ea96 2022-09-14T18:10:01.238 harvester chia.harvester.harvester: INFO peer disconnected {'host': '192.168.1.22’, 'port': 8447} 2022-09-14T18:10:04.025 harvester harvester : INFO Reconnecting to peer {'host': '192.168.1.22’, 'port': 8447} 2022-09-14T18:10:06.068 harvester harvester_server : INFO Cannot connect to host 192.168.1.22:8447 ssl:<ssl.SSLContext object at 0x000001342C202240> [The remote computer refused the network connection] 2022-09-14T18:10:09.079 harvester harvester : INFO Reconnecting to peer {'host': '192.168.1.22’, 'port': 8447} 2022-09-14T18:10:11.108 harvester harvester_server : INFO Cannot connect to host 192.168.1.22:8447 ssl:<ssl.SSLContext object at 0x000001342C2024C0> [The remote computer refused the network connection] 2022-09-14T18:10:14.122 harvester harvester : INFO Reconnecting to peer {'host': '192.168.1.22’, 'port': 8447} 2022-09-14T18:10:16.171 harvester harvester_server : INFO Cannot connect to host 192.168.1.22:8447 ssl:<ssl.SSLContext object at 0x000001342C2B00C0> [The remote computer refused the network connection] 2022-09-14T18:10:19.185 harvester harvester : INFO Reconnecting to peer {'host': '192.168.1.22’, 'port': 8447} 2022-09-14T18:10:19.278 harvester harvester_server : INFO Connected with farmer {'host': '192.168.1.22’, 'port': 8447} 2022-09-14T18:10:19.296 harvester chia.plotting.manager : INFO Loaded 20526 bytes of cached data

benjfranklin100 avatar Sep 14 '22 23:09 benjfranklin100

FYI, testing this possible solution from 13403 . Looks ok so far

maxcom99 avatar Sep 17 '22 04:09 maxcom99

FYI, testing this possible solution from 13403 . Looks ok so far

Doesn't seem to be an incongruence in python version, all machines are on 3.10.4

I'm not entirely familiar with the process of safely rolling back python. I know completely removing python just absolutely shreds the system, however, I'm getting mixed answers regarding rolling it back and most posts I see recommend doing so in a virtual environment.

benjfranklin100 avatar Sep 19 '22 09:09 benjfranklin100

FYI, testing this possible solution from 13403 . Looks ok so far

Doesn't seem to be an incongruence in python version, all machines are on 3.10.4

I'm not entirely familiar with the process of safely rolling back python. I know completely removing python just absolutely shreds the system, however, I'm getting mixed answers regarding rolling it back and most posts I see recommend doing so in a virtual environment.

From what I saw in the other thread, the UI is built with 3.9 -- I ended up dumping the UI and using CLI for the actual farmer and turned the UI farmer into a wallet. All good 24h now. I still see it failing pings but it just continues on.

maxcom99 avatar Sep 19 '22 09:09 maxcom99

FYI, testing this possible solution from 13403 . Looks ok so far

Doesn't seem to be an incongruence in python version, all machines are on 3.10.4 I'm not entirely familiar with the process of safely rolling back python. I know completely removing python just absolutely shreds the system, however, I'm getting mixed answers regarding rolling it back and most posts I see recommend doing so in a virtual environment.

From what I saw in the other thread, the UI is built with 3.9 -- I ended up dumping the UI and using CLI for the actual farmer and turned the UI farmer into a wallet. All good 24h now. I still see it failing pings but it just continues on.

I managed to roll back python to 3.9.14 but the problem persists. Still fails to reconnect when the farmer is restarted while the windows machine reconnects immediately and the ubuntu machine needs to have its harvester restarted manually.

benjfranklin100 avatar Sep 19 '22 10:09 benjfranklin100

I'm very sorry, I made a mistake. I thought my Windows machine was on 1.5.1 but it was actually on 1.3.4. I thought I had updated it along with the other harvester and my farmer. Updated windows harvester to 1.5.1, it was able to reconnect to the farmer after it restarted no problem. Earlier I had said that they both exhibited the dropped connection, however, I realized it was picking it up just fine so I edited this comment to reflect that.

benjfranklin100 avatar Sep 20 '22 03:09 benjfranklin100

Updated my entire chia ecosystem to 1.6.0, problem continues to persist on Ubuntu harvester, no issues with windows harvester.

benjfranklin100 avatar Sep 21 '22 23:09 benjfranklin100

FYI, testing this possible solution from 13403 . Looks ok so far

Doesn't seem to be an incongruence in python version, all machines are on 3.10.4 I'm not entirely familiar with the process of safely rolling back python. I know completely removing python just absolutely shreds the system, however, I'm getting mixed answers regarding rolling it back and most posts I see recommend doing so in a virtual environment.

From what I saw in the other thread, the UI is built with 3.9 -- I ended up dumping the UI and using CLI for the actual farmer and turned the UI farmer into a wallet. All good 24h now. I still see it failing pings but it just continues on.

How's yours running? Hopefully still well? I noticed another user was having the same traceback after farmer restart/disconnection in #13480

benjfranklin100 avatar Sep 22 '22 02:09 benjfranklin100

Updated my entire chia ecosystem to 1.6.0, problem continues to persist on Ubuntu harvester, no issues with windows harvester.

For your Ubuntu harvester, you are just running via the CLI, correct?

wallentx avatar Sep 22 '22 05:09 wallentx

Anyone here able to send me a full debug.log with this issue and DEBUG log level while https://github.com/Chia-Network/chia-blockchain/pull/13508 is running on the harvester + the log from the farmer? Just send it on keybase http://keybase.io/dustinface

xdustinface avatar Sep 22 '22 16:09 xdustinface

Updated my entire chia ecosystem to 1.6.0, problem continues to persist on Ubuntu harvester, no issues with windows harvester.

For your Ubuntu harvester, you are just running via the CLI, correct?

Yes, just running through the CLI.

benjfranklin100 avatar Oct 01 '22 19:10 benjfranklin100

Anyone here able to send me a full debug.log with this issue and DEBUG log level while #13508 is running on the harvester + the log from the farmer? Just send it on keybase http://keybase.io/dustinface

I can do this within the next new days.

benjfranklin100 avatar Oct 01 '22 19:10 benjfranklin100

Closing issue - feel free to reopen when more info is available

emlowe avatar Jan 05 '23 22:01 emlowe