python-teos
python-teos copied to clipboard
Published
20 hours ago •
talaia-labs
talaia-labs
BTCPay teos Tor
Hi,
It seems that teos integration in BTCPay-dockerized latest stable version is not working either via ipv4:port tor:port nothing works. Any idea where to start and or what to track?
I'm giving a look at it at the moment.
Can you provide a docker log for the teos container so I can get a rough idea of what may be going on?
Should be something on the lines of: docker logs generated_bitcoin_teos_1
Here it is, some error keeps repeating from time to time, and the startup its weird also:
root@btc:~/btcpayserver-docker# docker logs generated_bitcoin_teos_1
27/02/2021 16:01:33 [Daemon] Tower identity found. Loading keys
27/02/2021 16:01:33 [Daemon] ERROR: An error occurred: Cannot connect to bitcoind. Shutting down
27/02/2021 16:01:37 [Daemon] Tower identity found. Loading keys
27/02/2021 16:01:41 [Daemon] Tower identity found. Loading keys
27/02/2021 16:01:41 [Daemon] ERROR: An error occurred: Cannot connect to bitcoind. Shutting down
27/02/2021 16:01:48 [Daemon] Tower identity found. Loading keys
27/02/2021 16:01:48 [Daemon] tower_id = 026247def18440eb3a3e59be829b7c12dd2e86da6989b88644de306132adf72761
27/02/2021 16:01:49 [Daemon] Starting TEOS
27/02/2021 16:01:49 [Daemon] Fresh bootstrap
27/02/2021 16:01:49 [Daemon] Internal API initialized. Serving at localhost:50051
[2021-02-27 16:01:49 +0000] [30] [INFO] Starting gunicorn 20.0.4
[2021-02-27 16:01:49 +0000] [30] [INFO] Listening at: http://0.0.0.0:9814 (30)
[2021-02-27 16:01:49 +0000] [30] [INFO] Using worker: sync
[2021-02-27 16:01:49 +0000] [31] [INFO] Booting worker with pid: 31
27/02/2021 16:01:49 [RPC] Initialized. Serving at localhost:8814
27/02/2021 16:01:49 [API] Initialized. Serving at 0.0.0.0:9814
27/02/2021 16:20:40 [ChainMonitor] New block received via zmq (block_hash=00000000000000000000564444c8b005b5bb6942e6023844842b81a10bc8bf4b)
27/02/2021 16:20:41 [Watcher] New block received (block_hash=00000000000000000000564444c8b005b5bb6942e6023844842b81a10bc8bf4b, prev_block_hash=00000000000000000008e2b41bf3415514e4e1a5d741971f38e75443d88046ee)
27/02/2021 16:20:41 [Responder] New block received (block_hash=00000000000000000000564444c8b005b5bb6942e6023844842b81a10bc8bf4b, prev_block_hash=00000000000000000008e2b41bf3415514e4e1a5d741971f38e75443d88046ee)
27/02/2021 16:35:17 [ChainMonitor] New block received via zmq (block_hash=000000000000000000049f53a64e4f30e5c7ae180ee6ff5f91a65c01340dc79e)
27/02/2021 16:35:17 [Responder] New block received (block_hash=000000000000000000049f53a64e4f30e5c7ae180ee6ff5f91a65c01340dc79e, prev_block_hash=00000000000000000000564444c8b005b5bb6942e6023844842b81a10bc8bf4b)
27/02/2021 16:35:17 [Watcher] New block received (block_hash=000000000000000000049f53a64e4f30e5c7ae180ee6ff5f91a65c01340dc79e, prev_block_hash=00000000000000000000564444c8b005b5bb6942e6023844842b81a10bc8bf4b)
27/02/2021 16:45:23 [ChainMonitor] New block received via zmq (block_hash=00000000000000000000450cba6f5a4fabb3cf5294c63592c145adc2238722cf)
27/02/2021 16:45:23 [Watcher] New block received (block_hash=00000000000000000000450cba6f5a4fabb3cf5294c63592c145adc2238722cf, prev_block_hash=000000000000000000049f53a64e4f30e5c7ae180ee6ff5f91a65c01340dc79e)
27/02/2021 16:45:23 [Responder] New block received (block_hash=00000000000000000000450cba6f5a4fabb3cf5294c63592c145adc2238722cf, prev_block_hash=000000000000000000049f53a64e4f30e5c7ae180ee6ff5f91a65c01340dc79e)
27/02/2021 16:50:40 [ChainMonitor] New block received via zmq (block_hash=0000000000000000000a853826cbab7fd1d283b074449f863b98700bc8580625)
27/02/2021 16:50:40 [Responder] New block received (block_hash=0000000000000000000a853826cbab7fd1d283b074449f863b98700bc8580625, prev_block_hash=00000000000000000000450cba6f5a4fabb3cf5294c63592c145adc2238722cf)
27/02/2021 16:50:40 [Watcher] New block received (block_hash=0000000000000000000a853826cbab7fd1d283b074449f863b98700bc8580625, prev_block_hash=00000000000000000000450cba6f5a4fabb3cf5294c63592c145adc2238722cf)
27/02/2021 17:03:29 [Daemon] Tower identity found. Loading keys
27/02/2021 17:03:29 [Daemon] tower_id = 026247def18440eb3a3e59be829b7c12dd2e86da6989b88644de306132adf72761
27/02/2021 17:03:29 [Daemon] Starting TEOS
27/02/2021 17:03:29 [Daemon] Fresh bootstrap
27/02/2021 17:03:29 [Daemon] Internal API initialized. Serving at localhost:50051
27/02/2021 17:03:30 [RPC] Initialized. Serving at localhost:8814
[2021-02-27 17:03:30 +0000] [30] [INFO] Starting gunicorn 20.0.4
[2021-02-27 17:03:30 +0000] [30] [INFO] Listening at: http://0.0.0.0:9814 (30)
[2021-02-27 17:03:30 +0000] [30] [INFO] Using worker: sync
[2021-02-27 17:03:30 +0000] [36] [INFO] Booting worker with pid: 36
27/02/2021 17:03:30 [API] Initialized. Serving at 0.0.0.0:9814
27/02/2021 17:14:34 [ChainMonitor] New block received via zmq (block_hash=000000000000000000031dd30757501a8cd7374bec9a6edb2824c17aff5b4340)
27/02/2021 17:14:34 [Watcher] New block received (block_hash=000000000000000000031dd30757501a8cd7374bec9a6edb2824c17aff5b4340, prev_block_hash=0000000000000000000a853826cbab7fd1d283b074449f863b98700bc8580625)
27/02/2021 17:14:34 [Responder] New block received (block_hash=000000000000000000031dd30757501a8cd7374bec9a6edb2824c17aff5b4340, prev_block_hash=0000000000000000000a853826cbab7fd1d283b074449f863b98700bc8580625)
27/02/2021 17:14:54 [ChainMonitor] New block received via zmq (block_hash=00000000000000000009b4a455271f63b53624b6f590a86ac6565e80e1c7e9be)
27/02/2021 17:14:54 [Responder] New block received (block_hash=00000000000000000009b4a455271f63b53624b6f590a86ac6565e80e1c7e9be, prev_block_hash=000000000000000000031dd30757501a8cd7374bec9a6edb2824c17aff5b4340)
27/02/2021 17:14:54 [Watcher] New block received (block_hash=00000000000000000009b4a455271f63b53624b6f590a86ac6565e80e1c7e9be, prev_block_hash=000000000000000000031dd30757501a8cd7374bec9a6edb2824c17aff5b4340)
27/02/2021 17:24:15 [ChainMonitor] New block received via zmq (block_hash=00000000000000000008f4f145b65396f92cea6bb500b0d922ec94c1730dad83)
27/02/2021 17:24:15 [Responder] New block received (block_hash=00000000000000000008f4f145b65396f92cea6bb500b0d922ec94c1730dad83, prev_block_hash=00000000000000000009b4a455271f63b53624b6f590a86ac6565e80e1c7e9be)
27/02/2021 17:24:15 [Watcher] New block received (block_hash=00000000000000000008f4f145b65396f92cea6bb500b0d922ec94c1730dad83, prev_block_hash=00000000000000000009b4a455271f63b53624b6f590a86ac6565e80e1c7e9be)
27/02/2021 17:26:44 [ChainMonitor] New block received via zmq (block_hash=00000000000000000008a3c9f745769d9a58ba321dc32c0a5ca54ae652c9f6aa)
27/02/2021 17:26:44 [Responder] New block received (block_hash=00000000000000000008a3c9f745769d9a58ba321dc32c0a5ca54ae652c9f6aa, prev_block_hash=00000000000000000008f4f145b65396f92cea6bb500b0d922ec94c1730dad83)
27/02/2021 17:26:44 [Watcher] New block received (block_hash=00000000000000000008a3c9f745769d9a58ba321dc32c0a5ca54ae652c9f6aa, prev_block_hash=00000000000000000008f4f145b65396f92cea6bb500b0d922ec94c1730dad83)
27/02/2021 17:29:29 [ChainMonitor] New block received via zmq (block_hash=00000000000000000005ee92a6f587ff582dccdf717c7f5a8db1086c0da3d5fa)
27/02/2021 17:29:29 [Watcher] New block received (block_hash=00000000000000000005ee92a6f587ff582dccdf717c7f5a8db1086c0da3d5fa, prev_block_hash=00000000000000000008a3c9f745769d9a58ba321dc32c0a5ca54ae652c9f6aa)
27/02/2021 17:29:29 [Responder] New block received (block_hash=00000000000000000005ee92a6f587ff582dccdf717c7f5a8db1086c0da3d5fa, prev_block_hash=00000000000000000008a3c9f745769d9a58ba321dc32c0a5ca54ae652c9f6aa)
27/02/2021 17:36:29 [ChainMonitor] New block received via zmq (block_hash=00000000000000000001cf0b8d58903d6bba81c93a1def6596f6ca78d81447c1)
27/02/2021 17:36:29 [Watcher] New block received (block_hash=00000000000000000001cf0b8d58903d6bba81c93a1def6596f6ca78d81447c1, prev_block_hash=00000000000000000005ee92a6f587ff582dccdf717c7f5a8db1086c0da3d5fa)
27/02/2021 17:36:29 [Responder] New block received (block_hash=00000000000000000001cf0b8d58903d6bba81c93a1def6596f6ca78d81447c1, prev_block_hash=00000000000000000005ee92a6f587ff582dccdf717c7f5a8db1086c0da3d5fa)
27/02/2021 17:51:25 [ChainMonitor] New block received via zmq (block_hash=000000000000000000072237d3be5dd3f46837ee86542802a0ebb21e6c6968f6)
27/02/2021 17:51:25 [Watcher] New block received (block_hash=000000000000000000072237d3be5dd3f46837ee86542802a0ebb21e6c6968f6, prev_block_hash=00000000000000000001cf0b8d58903d6bba81c93a1def6596f6ca78d81447c1)
27/02/2021 17:51:25 [Responder] New block received (block_hash=000000000000000000072237d3be5dd3f46837ee86542802a0ebb21e6c6968f6, prev_block_hash=00000000000000000001cf0b8d58903d6bba81c93a1def6596f6ca78d81447c1)
27/02/2021 17:52:55 [ChainMonitor] New block received via zmq (block_hash=0000000000000000000a8b4b4ebd9e3ec670fc99e3b86c7951590745d8fc645b)
27/02/2021 17:52:55 [Responder] New block received (block_hash=0000000000000000000a8b4b4ebd9e3ec670fc99e3b86c7951590745d8fc645b, prev_block_hash=000000000000000000072237d3be5dd3f46837ee86542802a0ebb21e6c6968f6)
27/02/2021 17:52:55 [Watcher] New block received (block_hash=0000000000000000000a8b4b4ebd9e3ec670fc99e3b86c7951590745d8fc645b, prev_block_hash=000000000000000000072237d3be5dd3f46837ee86542802a0ebb21e6c6968f6)
27/02/2021 18:01:26 [ChainMonitor] New block received via zmq (block_hash=00000000000000000003563a63eaa804414c182eef2127fe4aae9e61f5a2138d)
27/02/2021 18:01:26 [Watcher] New block received (block_hash=00000000000000000003563a63eaa804414c182eef2127fe4aae9e61f5a2138d, prev_block_hash=0000000000000000000a8b4b4ebd9e3ec670fc99e3b86c7951590745d8fc645b)
27/02/2021 18:01:26 [Responder] New block received (block_hash=00000000000000000003563a63eaa804414c182eef2127fe4aae9e61f5a2138d, prev_block_hash=0000000000000000000a8b4b4ebd9e3ec670fc99e3b86c7951590745d8fc645b)
27/02/2021 18:08:16 [ChainMonitor] New block received via zmq (block_hash=000000000000000000090ef0d38c6a5a8b7bf9348dbcb3dc801a0289fcd9bd29)
27/02/2021 18:08:16 [Responder] New block received (block_hash=000000000000000000090ef0d38c6a5a8b7bf9348dbcb3dc801a0289fcd9bd29, prev_block_hash=00000000000000000003563a63eaa804414c182eef2127fe4aae9e61f5a2138d)
27/02/2021 18:08:16 [Watcher] New block received (block_hash=000000000000000000090ef0d38c6a5a8b7bf9348dbcb3dc801a0289fcd9bd29, prev_block_hash=00000000000000000003563a63eaa804414c182eef2127fe4aae9e61f5a2138d)
27/02/2021 18:22:01 [ChainMonitor] New block received via zmq (block_hash=00000000000000000004c423a33586a0d8dbbd2d4f1122d16cfe11742181b688)
27/02/2021 18:22:01 [Responder] New block received (block_hash=00000000000000000004c423a33586a0d8dbbd2d4f1122d16cfe11742181b688, prev_block_hash=000000000000000000090ef0d38c6a5a8b7bf9348dbcb3dc801a0289fcd9bd29)
27/02/2021 18:22:01 [Watcher] New block received (block_hash=00000000000000000004c423a33586a0d8dbbd2d4f1122d16cfe11742181b688, prev_block_hash=000000000000000000090ef0d38c6a5a8b7bf9348dbcb3dc801a0289fcd9bd29)
27/02/2021 18:54:30 [ChainMonitor] New block received via zmq (block_hash=0000000000000000000081562e321c0528643d92508555707f02e702b3846578)
27/02/2021 18:54:30 [Responder] New block received (block_hash=0000000000000000000081562e321c0528643d92508555707f02e702b3846578, prev_block_hash=00000000000000000004c423a33586a0d8dbbd2d4f1122d16cfe11742181b688)
27/02/2021 18:54:30 [Watcher] New block received (block_hash=0000000000000000000081562e321c0528643d92508555707f02e702b3846578, prev_block_hash=00000000000000000004c423a33586a0d8dbbd2d4f1122d16cfe11742181b688)
27/02/2021 18:55:14 [ChainMonitor] New block received via zmq (block_hash=00000000000000000006644ad08cd575c280aab60e2a8e67cf7bba703565d5aa)
27/02/2021 18:55:14 [Watcher] New block received (block_hash=00000000000000000006644ad08cd575c280aab60e2a8e67cf7bba703565d5aa, prev_block_hash=0000000000000000000081562e321c0528643d92508555707f02e702b3846578)
27/02/2021 18:55:14 [Responder] New block received (block_hash=00000000000000000006644ad08cd575c280aab60e2a8e67cf7bba703565d5aa, prev_block_hash=0000000000000000000081562e321c0528643d92508555707f02e702b3846578)
27/02/2021 18:56:24 [ChainMonitor] New block received via zmq (block_hash=0000000000000000000b691cd63d88edb465e43b477948d9fd82d1e927b15eb6)
27/02/2021 18:56:24 [Responder] New block received (block_hash=0000000000000000000b691cd63d88edb465e43b477948d9fd82d1e927b15eb6, prev_block_hash=00000000000000000006644ad08cd575c280aab60e2a8e67cf7bba703565d5aa)
27/02/2021 18:56:24 [Watcher] New block received (block_hash=0000000000000000000b691cd63d88edb465e43b477948d9fd82d1e927b15eb6, prev_block_hash=00000000000000000006644ad08cd575c280aab60e2a8e67cf7bba703565d5aa)
27/02/2021 19:04:59 [ChainMonitor] New block received via zmq (block_hash=00000000000000000001cadb8fe9824fd0778a551445ca3c82a8edd52ee78a2b)
27/02/2021 19:05:00 [Watcher] New block received (block_hash=00000000000000000001cadb8fe9824fd0778a551445ca3c82a8edd52ee78a2b, prev_block_hash=0000000000000000000b691cd63d88edb465e43b477948d9fd82d1e927b15eb6)
27/02/2021 19:05:00 [Responder] New block received (block_hash=00000000000000000001cadb8fe9824fd0778a551445ca3c82a8edd52ee78a2b, prev_block_hash=0000000000000000000b691cd63d88edb465e43b477948d9fd82d1e927b15eb6)
27/02/2021 19:22:32 [ChainMonitor] New block received via zmq (block_hash=000000000000000000074b9d187cca8c5b69dc636f3640d04423705a10715d70)
27/02/2021 19:22:32 [Watcher] New block received (block_hash=000000000000000000074b9d187cca8c5b69dc636f3640d04423705a10715d70, prev_block_hash=00000000000000000001cadb8fe9824fd0778a551445ca3c82a8edd52ee78a2b)
27/02/2021 19:22:32 [Responder] New block received (block_hash=000000000000000000074b9d187cca8c5b69dc636f3640d04423705a10715d70, prev_block_hash=00000000000000000001cadb8fe9824fd0778a551445ca3c82a8edd52ee78a2b)
27/02/2021 19:34:47 [ChainMonitor] New block received via zmq (block_hash=00000000000000000007d34ecc4de69e3f6659535669733ea349f1872095a3eb)
27/02/2021 19:34:47 [Watcher] New block received (block_hash=00000000000000000007d34ecc4de69e3f6659535669733ea349f1872095a3eb, prev_block_hash=000000000000000000074b9d187cca8c5b69dc636f3640d04423705a10715d70)
27/02/2021 19:34:47 [Responder] New block received (block_hash=00000000000000000007d34ecc4de69e3f6659535669733ea349f1872095a3eb, prev_block_hash=000000000000000000074b9d187cca8c5b69dc636f3640d04423705a10715d70)
27/02/2021 19:36:01 [ChainMonitor] New block received via zmq (block_hash=0000000000000000000c0f2eb49170779b9277cdfaaead94395711ef242ee255)
27/02/2021 19:36:01 [Responder] New block received (block_hash=0000000000000000000c0f2eb49170779b9277cdfaaead94395711ef242ee255, prev_block_hash=00000000000000000007d34ecc4de69e3f6659535669733ea349f1872095a3eb)
27/02/2021 19:36:01 [Watcher] New block received (block_hash=0000000000000000000c0f2eb49170779b9277cdfaaead94395711ef242ee255, prev_block_hash=00000000000000000007d34ecc4de69e3f6659535669733ea349f1872095a3eb)
27/02/2021 19:37:38 [ChainMonitor] New block received via zmq (block_hash=00000000000000000008c3e0d4288ef0d8b2474abd322fd35b5bed69a5cd4158)
27/02/2021 19:37:38 [Watcher] New block received (block_hash=00000000000000000008c3e0d4288ef0d8b2474abd322fd35b5bed69a5cd4158, prev_block_hash=0000000000000000000c0f2eb49170779b9277cdfaaead94395711ef242ee255)
27/02/2021 19:37:38 [Responder] New block received (block_hash=00000000000000000008c3e0d4288ef0d8b2474abd322fd35b5bed69a5cd4158, prev_block_hash=0000000000000000000c0f2eb49170779b9277cdfaaead94395711ef242ee255)
27/02/2021 19:43:18 [ChainMonitor] New block received via zmq (block_hash=0000000000000000000182d8acc52e4cdc4a1b3480850e9904575f3ff8fc5873)
27/02/2021 19:43:18 [Responder] New block received (block_hash=0000000000000000000182d8acc52e4cdc4a1b3480850e9904575f3ff8fc5873, prev_block_hash=00000000000000000008c3e0d4288ef0d8b2474abd322fd35b5bed69a5cd4158)
27/02/2021 19:43:18 [Watcher] New block received (block_hash=0000000000000000000182d8acc52e4cdc4a1b3480850e9904575f3ff8fc5873, prev_block_hash=00000000000000000008c3e0d4288ef0d8b2474abd322fd35b5bed69a5cd4158)
27/02/2021 19:45:05 [ChainMonitor] New block received via zmq (block_hash=0000000000000000000957793c7f9bc44044ddaac907591e2ef1ee36d5e8c448)
27/02/2021 19:45:05 [Responder] New block received (block_hash=0000000000000000000957793c7f9bc44044ddaac907591e2ef1ee36d5e8c448, prev_block_hash=0000000000000000000182d8acc52e4cdc4a1b3480850e9904575f3ff8fc5873)
27/02/2021 19:45:05 [Watcher] New block received (block_hash=0000000000000000000957793c7f9bc44044ddaac907591e2ef1ee36d5e8c448, prev_block_hash=0000000000000000000182d8acc52e4cdc4a1b3480850e9904575f3ff8fc5873)
27/02/2021 19:51:37 [ChainMonitor] New block received via zmq (block_hash=00000000000000000003034338d5e29648e175ad3c85723045ff0524de7a7542)
27/02/2021 19:51:37 [Responder] New block received (block_hash=00000000000000000003034338d5e29648e175ad3c85723045ff0524de7a7542, prev_block_hash=0000000000000000000957793c7f9bc44044ddaac907591e2ef1ee36d5e8c448)
27/02/2021 19:51:37 [Watcher] New block received (block_hash=00000000000000000003034338d5e29648e175ad3c85723045ff0524de7a7542, prev_block_hash=0000000000000000000957793c7f9bc44044ddaac907591e2ef1ee36d5e8c448)
27/02/2021 20:01:51 [ChainMonitor] New block received via zmq (block_hash=00000000000000000000368cd2eaab5ab9598b51f01ebce83ea8419232a66e9d)
27/02/2021 20:01:51 [Watcher] New block received (block_hash=00000000000000000000368cd2eaab5ab9598b51f01ebce83ea8419232a66e9d, prev_block_hash=00000000000000000003034338d5e29648e175ad3c85723045ff0524de7a7542)
27/02/2021 20:01:51 [Responder] New block received (block_hash=00000000000000000000368cd2eaab5ab9598b51f01ebce83ea8419232a66e9d, prev_block_hash=00000000000000000003034338d5e29648e175ad3c85723045ff0524de7a7542)
27/02/2021 20:03:56 [ChainMonitor] New block received via zmq (block_hash=0000000000000000000b6549205de4024e92b9bdc21f87017adf1a01dc81a896)
27/02/2021 20:03:56 [Watcher] New block received (block_hash=0000000000000000000b6549205de4024e92b9bdc21f87017adf1a01dc81a896, prev_block_hash=00000000000000000000368cd2eaab5ab9598b51f01ebce83ea8419232a66e9d)
27/02/2021 20:03:56 [Responder] New block received (block_hash=0000000000000000000b6549205de4024e92b9bdc21f87017adf1a01dc81a896, prev_block_hash=00000000000000000000368cd2eaab5ab9598b51f01ebce83ea8419232a66e9d)
27/02/2021 20:26:05 [ChainMonitor] New block received via zmq (block_hash=0000000000000000000cae534e4d44a2278f575e56a5f967ef18ba916a16806d)
27/02/2021 20:26:06 [Watcher] New block received (block_hash=0000000000000000000cae534e4d44a2278f575e56a5f967ef18ba916a16806d, prev_block_hash=0000000000000000000b6549205de4024e92b9bdc21f87017adf1a01dc81a896)
27/02/2021 20:26:06 [Responder] New block received (block_hash=0000000000000000000cae534e4d44a2278f575e56a5f967ef18ba916a16806d, prev_block_hash=0000000000000000000b6549205de4024e92b9bdc21f87017adf1a01dc81a896)
27/02/2021 20:37:22 [ChainMonitor] New block received via zmq (block_hash=00000000000000000006a2a8f976a5ba4decd4bf724290b9db64fce935c7960d)
27/02/2021 20:37:22 [Watcher] New block received (block_hash=00000000000000000006a2a8f976a5ba4decd4bf724290b9db64fce935c7960d, prev_block_hash=0000000000000000000cae534e4d44a2278f575e56a5f967ef18ba916a16806d)
27/02/2021 20:37:22 [Responder] New block received (block_hash=00000000000000000006a2a8f976a5ba4decd4bf724290b9db64fce935c7960d, prev_block_hash=0000000000000000000cae534e4d44a2278f575e56a5f967ef18ba916a16806d)
27/02/2021 21:06:33 [ChainMonitor] New block received via zmq (block_hash=00000000000000000007d701ed789c6831cec5bdc8ae859b13ecb71eb185a29e)
27/02/2021 21:06:33 [Watcher] New block received (block_hash=00000000000000000007d701ed789c6831cec5bdc8ae859b13ecb71eb185a29e, prev_block_hash=00000000000000000006a2a8f976a5ba4decd4bf724290b9db64fce935c7960d)
27/02/2021 21:06:33 [Responder] New block received (block_hash=00000000000000000007d701ed789c6831cec5bdc8ae859b13ecb71eb185a29e, prev_block_hash=00000000000000000006a2a8f976a5ba4decd4bf724290b9db64fce935c7960d)
27/02/2021 21:29:21 [ChainMonitor] New block received via zmq (block_hash=0000000000000000000532fccdb8329d0ae1ba59c9f43471e6e4a1efe69e9100)
27/02/2021 21:29:21 [Watcher] New block received (block_hash=0000000000000000000532fccdb8329d0ae1ba59c9f43471e6e4a1efe69e9100, prev_block_hash=00000000000000000007d701ed789c6831cec5bdc8ae859b13ecb71eb185a29e)
27/02/2021 21:29:21 [Responder] New block received (block_hash=0000000000000000000532fccdb8329d0ae1ba59c9f43471e6e4a1efe69e9100, prev_block_hash=00000000000000000007d701ed789c6831cec5bdc8ae859b13ecb71eb185a29e)
27/02/2021 21:42:05 [ChainMonitor] New block received via zmq (block_hash=00000000000000000009a5fbe17ed811f49ef03ae883cc88f6baeb51852ab3ac)
27/02/2021 21:42:05 [Watcher] New block received (block_hash=00000000000000000009a5fbe17ed811f49ef03ae883cc88f6baeb51852ab3ac, prev_block_hash=0000000000000000000532fccdb8329d0ae1ba59c9f43471e6e4a1efe69e9100)
27/02/2021 21:42:05 [Responder] New block received (block_hash=00000000000000000009a5fbe17ed811f49ef03ae883cc88f6baeb51852ab3ac, prev_block_hash=0000000000000000000532fccdb8329d0ae1ba59c9f43471e6e4a1efe69e9100)
27/02/2021 22:01:05 [ChainMonitor] New block received via zmq (block_hash=00000000000000000006cae670f70127ec3d6422aad0d61d9984178c0caa2e02)
27/02/2021 22:01:05 [Watcher] New block received (block_hash=00000000000000000006cae670f70127ec3d6422aad0d61d9984178c0caa2e02, prev_block_hash=00000000000000000009a5fbe17ed811f49ef03ae883cc88f6baeb51852ab3ac)
27/02/2021 22:01:05 [Responder] New block received (block_hash=00000000000000000006cae670f70127ec3d6422aad0d61d9984178c0caa2e02, prev_block_hash=00000000000000000009a5fbe17ed811f49ef03ae883cc88f6baeb51852ab3ac)
27/02/2021 22:20:13 [ChainMonitor] New block received via zmq (block_hash=0000000000000000000201084b7f4b42f95c1e01923dda308f944ee9ce936661)
27/02/2021 22:20:13 [Watcher] New block received (block_hash=0000000000000000000201084b7f4b42f95c1e01923dda308f944ee9ce936661, prev_block_hash=00000000000000000006cae670f70127ec3d6422aad0d61d9984178c0caa2e02)
27/02/2021 22:20:13 [Responder] New block received (block_hash=0000000000000000000201084b7f4b42f95c1e01923dda308f944ee9ce936661, prev_block_hash=00000000000000000006cae670f70127ec3d6422aad0d61d9984178c0caa2e02)
27/02/2021 22:35:00 [ChainMonitor] New block received via zmq (block_hash=0000000000000000000c6ccf9473f54126cf1c1bbc2086e123dc26d816276a35)
27/02/2021 22:35:00 [Watcher] New block received (block_hash=0000000000000000000c6ccf9473f54126cf1c1bbc2086e123dc26d816276a35, prev_block_hash=0000000000000000000201084b7f4b42f95c1e01923dda308f944ee9ce936661)
27/02/2021 22:35:00 [Responder] New block received (block_hash=0000000000000000000c6ccf9473f54126cf1c1bbc2086e123dc26d816276a35, prev_block_hash=0000000000000000000201084b7f4b42f95c1e01923dda308f944ee9ce936661)
27/02/2021 22:57:31 [ChainMonitor] New block received via zmq (block_hash=0000000000000000000c315e04e62f4a940ae70757c22b6bdaf6680efb5da013)
27/02/2021 22:57:31 [Watcher] New block received (block_hash=0000000000000000000c315e04e62f4a940ae70757c22b6bdaf6680efb5da013, prev_block_hash=0000000000000000000c6ccf9473f54126cf1c1bbc2086e123dc26d816276a35)
27/02/2021 22:57:31 [Responder] New block received (block_hash=0000000000000000000c315e04e62f4a940ae70757c22b6bdaf6680efb5da013, prev_block_hash=0000000000000000000c6ccf9473f54126cf1c1bbc2086e123dc26d816276a35)
27/02/2021 23:03:06 [ChainMonitor] New block received via zmq (block_hash=00000000000000000009d4c70612485bd3a8f4fb05cc04bb12a47084884c0381)
27/02/2021 23:03:06 [Watcher] New block received (block_hash=00000000000000000009d4c70612485bd3a8f4fb05cc04bb12a47084884c0381, prev_block_hash=0000000000000000000c315e04e62f4a940ae70757c22b6bdaf6680efb5da013)
27/02/2021 23:03:06 [Responder] New block received (block_hash=00000000000000000009d4c70612485bd3a8f4fb05cc04bb12a47084884c0381, prev_block_hash=0000000000000000000c315e04e62f4a940ae70757c22b6bdaf6680efb5da013)
27/02/2021 23:08:02 [ChainMonitor] New block received via zmq (block_hash=00000000000000000002d56ca740d0346c0c9522bd2636d5e260a975280c2588)
27/02/2021 23:08:02 [Watcher] New block received (block_hash=00000000000000000002d56ca740d0346c0c9522bd2636d5e260a975280c2588, prev_block_hash=00000000000000000009d4c70612485bd3a8f4fb05cc04bb12a47084884c0381)
27/02/2021 23:08:02 [Responder] New block received (block_hash=00000000000000000002d56ca740d0346c0c9522bd2636d5e260a975280c2588, prev_block_hash=00000000000000000009d4c70612485bd3a8f4fb05cc04bb12a47084884c0381)
27/02/2021 23:11:10 [ChainMonitor] New block received via zmq (block_hash=000000000000000000077b8a2453eb2544db153dcf9d17923b17e6d194a50934)
27/02/2021 23:11:10 [Watcher] New block received (block_hash=000000000000000000077b8a2453eb2544db153dcf9d17923b17e6d194a50934, prev_block_hash=00000000000000000002d56ca740d0346c0c9522bd2636d5e260a975280c2588)
27/02/2021 23:11:10 [Responder] New block received (block_hash=000000000000000000077b8a2453eb2544db153dcf9d17923b17e6d194a50934, prev_block_hash=00000000000000000002d56ca740d0346c0c9522bd2636d5e260a975280c2588)
27/02/2021 23:25:14 [ChainMonitor] New block received via zmq (block_hash=0000000000000000000745d736aa6e6d548d17006aaa84a9204801e05735377f)
27/02/2021 23:25:14 [Responder] New block received (block_hash=0000000000000000000745d736aa6e6d548d17006aaa84a9204801e05735377f, prev_block_hash=000000000000000000077b8a2453eb2544db153dcf9d17923b17e6d194a50934)
27/02/2021 23:25:14 [Watcher] New block received (block_hash=0000000000000000000745d736aa6e6d548d17006aaa84a9204801e05735377f, prev_block_hash=000000000000000000077b8a2453eb2544db153dcf9d17923b17e6d194a50934)
27/02/2021 23:25:36 [ChainMonitor] New block received via zmq (block_hash=0000000000000000000bddacd380e2d85146dbba5ae2954a31a9ff3e32ae1ad3)
27/02/2021 23:25:36 [Responder] New block received (block_hash=0000000000000000000bddacd380e2d85146dbba5ae2954a31a9ff3e32ae1ad3, prev_block_hash=0000000000000000000745d736aa6e6d548d17006aaa84a9204801e05735377f)
27/02/2021 23:25:36 [Watcher] New block received (block_hash=0000000000000000000bddacd380e2d85146dbba5ae2954a31a9ff3e32ae1ad3, prev_block_hash=0000000000000000000745d736aa6e6d548d17006aaa84a9204801e05735377f)
27/02/2021 23:54:08 [ChainMonitor] New block received via zmq (block_hash=0000000000000000000c03f035c13f6de01236875ff96d121ac177396b3899f5)
27/02/2021 23:54:08 [Responder] New block received (block_hash=0000000000000000000c03f035c13f6de01236875ff96d121ac177396b3899f5, prev_block_hash=0000000000000000000bddacd380e2d85146dbba5ae2954a31a9ff3e32ae1ad3)
27/02/2021 23:54:08 [Watcher] New block received (block_hash=0000000000000000000c03f035c13f6de01236875ff96d121ac177396b3899f5, prev_block_hash=0000000000000000000bddacd380e2d85146dbba5ae2954a31a9ff3e32ae1ad3)
28/02/2021 00:14:58 [ChainMonitor] New block received via zmq (block_hash=0000000000000000000766eaffd467cc9a5c54d740ddf3c11283bc76f5dcd279)
28/02/2021 00:14:58 [Responder] New block received (block_hash=0000000000000000000766eaffd467cc9a5c54d740ddf3c11283bc76f5dcd279, prev_block_hash=0000000000000000000c03f035c13f6de01236875ff96d121ac177396b3899f5)
28/02/2021 00:14:58 [Watcher] New block received (block_hash=0000000000000000000766eaffd467cc9a5c54d740ddf3c11283bc76f5dcd279, prev_block_hash=0000000000000000000c03f035c13f6de01236875ff96d121ac177396b3899f5)
28/02/2021 00:33:00 [ChainMonitor] New block received via zmq (block_hash=00000000000000000007b619218d276894e8768876fd42f81131a98dc000da77)
28/02/2021 00:33:00 [Responder] New block received (block_hash=00000000000000000007b619218d276894e8768876fd42f81131a98dc000da77, prev_block_hash=0000000000000000000766eaffd467cc9a5c54d740ddf3c11283bc76f5dcd279)
28/02/2021 00:33:00 [Watcher] New block received (block_hash=00000000000000000007b619218d276894e8768876fd42f81131a98dc000da77, prev_block_hash=0000000000000000000766eaffd467cc9a5c54d740ddf3c11283bc76f5dcd279)
28/02/2021 00:55:22 [ChainMonitor] New block received via zmq (block_hash=0000000000000000000c428a229a3c23d084cc800002faaff801a5c34edcd837)
28/02/2021 00:55:22 [Responder] New block received (block_hash=0000000000000000000c428a229a3c23d084cc800002faaff801a5c34edcd837, prev_block_hash=00000000000000000007b619218d276894e8768876fd42f81131a98dc000da77)
28/02/2021 00:55:22 [Watcher] New block received (block_hash=0000000000000000000c428a229a3c23d084cc800002faaff801a5c34edcd837, prev_block_hash=00000000000000000007b619218d276894e8768876fd42f81131a98dc000da77)
28/02/2021 01:03:04 [ChainMonitor] New block received via zmq (block_hash=0000000000000000000cdd9dfa6b93e9b7e190e23c656b79013e644fb7bbbab3)
28/02/2021 01:03:04 [Responder] New block received (block_hash=0000000000000000000cdd9dfa6b93e9b7e190e23c656b79013e644fb7bbbab3, prev_block_hash=0000000000000000000c428a229a3c23d084cc800002faaff801a5c34edcd837)
28/02/2021 01:03:04 [Watcher] New block received (block_hash=0000000000000000000cdd9dfa6b93e9b7e190e23c656b79013e644fb7bbbab3, prev_block_hash=0000000000000000000c428a229a3c23d084cc800002faaff801a5c34edcd837)
28/02/2021 01:05:59 [ChainMonitor] New block received via zmq (block_hash=0000000000000000000ce5008ba21432ae8bc5e3ef1ade5c7a9b810391012643)
28/02/2021 01:05:59 [Responder] New block received (block_hash=0000000000000000000ce5008ba21432ae8bc5e3ef1ade5c7a9b810391012643, prev_block_hash=0000000000000000000cdd9dfa6b93e9b7e190e23c656b79013e644fb7bbbab3)
28/02/2021 01:05:59 [Watcher] New block received (block_hash=0000000000000000000ce5008ba21432ae8bc5e3ef1ade5c7a9b810391012643, prev_block_hash=0000000000000000000cdd9dfa6b93e9b7e190e23c656b79013e644fb7bbbab3)
28/02/2021 01:08:40 [ChainMonitor] New block received via zmq (block_hash=0000000000000000000670dd30be6229e99eb13e3b163fa30990f8c3f2a05617)
28/02/2021 01:08:40 [Watcher] New block received (block_hash=0000000000000000000670dd30be6229e99eb13e3b163fa30990f8c3f2a05617, prev_block_hash=0000000000000000000ce5008ba21432ae8bc5e3ef1ade5c7a9b810391012643)
28/02/2021 01:08:40 [Responder] New block received (block_hash=0000000000000000000670dd30be6229e99eb13e3b163fa30990f8c3f2a05617, prev_block_hash=0000000000000000000ce5008ba21432ae8bc5e3ef1ade5c7a9b810391012643)
28/02/2021 01:14:31 [ChainMonitor] New block received via zmq (block_hash=0000000000000000000bdcbb5ffe27549ef9c4ee0320aeaea6aad918fbfa6d16)
28/02/2021 01:14:32 [Responder] New block received (block_hash=0000000000000000000bdcbb5ffe27549ef9c4ee0320aeaea6aad918fbfa6d16, prev_block_hash=0000000000000000000670dd30be6229e99eb13e3b163fa30990f8c3f2a05617)
28/02/2021 01:14:32 [Watcher] New block received (block_hash=0000000000000000000bdcbb5ffe27549ef9c4ee0320aeaea6aad918fbfa6d16, prev_block_hash=0000000000000000000670dd30be6229e99eb13e3b163fa30990f8c3f2a05617)
28/02/2021 01:37:28 [ChainMonitor] New block received via zmq (block_hash=000000000000000000015b3cdd37fd72df65a5e846a35bb7e6a9c60b589bf22c)
28/02/2021 01:37:28 [Responder] New block received (block_hash=000000000000000000015b3cdd37fd72df65a5e846a35bb7e6a9c60b589bf22c, prev_block_hash=0000000000000000000bdcbb5ffe27549ef9c4ee0320aeaea6aad918fbfa6d16)
28/02/2021 01:37:28 [Watcher] New block received (block_hash=000000000000000000015b3cdd37fd72df65a5e846a35bb7e6a9c60b589bf22c, prev_block_hash=0000000000000000000bdcbb5ffe27549ef9c4ee0320aeaea6aad918fbfa6d16)
28/02/2021 01:39:59 [ChainMonitor] New block received via zmq (block_hash=000000000000000000087fe6f18a447440cb554ad13bcab19ceaf182849b7cc3)
28/02/2021 01:40:00 [Watcher] New block received (block_hash=000000000000000000087fe6f18a447440cb554ad13bcab19ceaf182849b7cc3, prev_block_hash=000000000000000000015b3cdd37fd72df65a5e846a35bb7e6a9c60b589bf22c)
28/02/2021 01:40:00 [Responder] New block received (block_hash=000000000000000000087fe6f18a447440cb554ad13bcab19ceaf182849b7cc3, prev_block_hash=000000000000000000015b3cdd37fd72df65a5e846a35bb7e6a9c60b589bf22c)
28/02/2021 02:08:56 [ChainMonitor] New block received via zmq (block_hash=00000000000000000009a997551d19d2d4aabe245f7ffabea5d2e8425e6aef78)
28/02/2021 02:08:56 [Watcher] New block received (block_hash=00000000000000000009a997551d19d2d4aabe245f7ffabea5d2e8425e6aef78, prev_block_hash=000000000000000000087fe6f18a447440cb554ad13bcab19ceaf182849b7cc3)
28/02/2021 02:08:56 [Responder] New block received (block_hash=00000000000000000009a997551d19d2d4aabe245f7ffabea5d2e8425e6aef78, prev_block_hash=000000000000000000087fe6f18a447440cb554ad13bcab19ceaf182849b7cc3)
28/02/2021 02:09:28 [ChainMonitor] New block received via zmq (block_hash=00000000000000000000f5a0080335e00891b96c5db64313e8d116728437fa29)
28/02/2021 02:09:28 [Responder] New block received (block_hash=00000000000000000000f5a0080335e00891b96c5db64313e8d116728437fa29, prev_block_hash=00000000000000000009a997551d19d2d4aabe245f7ffabea5d2e8425e6aef78)
28/02/2021 02:09:28 [Watcher] New block received (block_hash=00000000000000000000f5a0080335e00891b96c5db64313e8d116728437fa29, prev_block_hash=00000000000000000009a997551d19d2d4aabe245f7ffabea5d2e8425e6aef78)
Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/threading.py", line 950, in _bootstrap_inner
self.run()
File "/usr/local/lib/python3.9/threading.py", line 888, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.9/site-packages/teos/chain_monitor.py", line 123, in monitor_chain_polling
current_tip = self.block_processor.get_best_block_hash()
File "/usr/local/lib/python3.9/site-packages/teos/block_processor.py", line 62, in get_best_block_hash
block_hash = bitcoin_cli(self.btc_connect_params).getbestblockhash()
File "/usr/local/lib/python3.9/site-packages/teos/utils/auth_proxy.py", line 151, in __call__
response, status = self._request("POST", self.__url.path, postdata.encode("utf-8"))
File "/usr/local/lib/python3.9/site-packages/teos/utils/auth_proxy.py", line 114, in _request
self.__conn.request(method, path, postdata, headers)
File "/usr/local/lib/python3.9/http/client.py", line 1255, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.9/http/client.py", line 1301, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.9/http/client.py", line 1250, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.9/http/client.py", line 1010, in _send_output
self.send(msg)
File "/usr/local/lib/python3.9/http/client.py", line 950, in send
self.connect()
File "/usr/local/lib/python3.9/http/client.py", line 921, in connect
self.sock = self._create_connection(
File "/usr/local/lib/python3.9/socket.py", line 843, in create_connection
raise err
File "/usr/local/lib/python3.9/socket.py", line 831, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
28/02/2021 02:13:42 [Daemon] Tower identity found. Loading keys
28/02/2021 02:13:42 [Daemon] tower_id = 026247def18440eb3a3e59be829b7c12dd2e86da6989b88644de306132adf72761
28/02/2021 02:13:44 [Daemon] Starting TEOS
28/02/2021 02:13:45 [Daemon] Fresh bootstrap
28/02/2021 02:13:45 [Daemon] Internal API initialized. Serving at localhost:50051
28/02/2021 02:13:45 [RPC] Initialized. Serving at localhost:8814
[2021-02-28 02:13:46 +0000] [30] [INFO] Starting gunicorn 20.0.4
[2021-02-28 02:13:46 +0000] [30] [INFO] Listening at: http://0.0.0.0:9814 (30)
[2021-02-28 02:13:46 +0000] [30] [INFO] Using worker: sync
[2021-02-28 02:13:46 +0000] [36] [INFO] Booting worker with pid: 36
28/02/2021 02:13:46 [API] Initialized. Serving at 0.0.0.0:9814
28/02/2021 02:18:01 [ChainMonitor] New block received via zmq (block_hash=0000000000000000000ba5232523408775fd70b9d731200267e6d36073b9186f)
28/02/2021 02:18:01 [Responder] New block received (block_hash=0000000000000000000ba5232523408775fd70b9d731200267e6d36073b9186f, prev_block_hash=00000000000000000000f5a0080335e00891b96c5db64313e8d116728437fa29)
28/02/2021 02:18:01 [Watcher] New block received (block_hash=0000000000000000000ba5232523408775fd70b9d731200267e6d36073b9186f, prev_block_hash=00000000000000000000f5a0080335e00891b96c5db64313e8d116728437fa29)
28/02/2021 02:19:10 [ChainMonitor] New block received via zmq (block_hash=0000000000000000000434a0210f460be762de3a65b79517b9d537a8b10c4c71)
28/02/2021 02:19:10 [Responder] New block received (block_hash=0000000000000000000434a0210f460be762de3a65b79517b9d537a8b10c4c71, prev_block_hash=0000000000000000000ba5232523408775fd70b9d731200267e6d36073b9186f)
28/02/2021 02:19:10 [Watcher] New block received (block_hash=0000000000000000000434a0210f460be762de3a65b79517b9d537a8b10c4c71, prev_block_hash=0000000000000000000ba5232523408775fd70b9d731200267e6d36073b9186f)
28/02/2021 02:43:49 [ChainMonitor] New block received via zmq (block_hash=0000000000000000000136c6998cad74e2252c6295a0969fc250168e1fe78650)
28/02/2021 02:43:49 [Watcher] New block received (block_hash=0000000000000000000136c6998cad74e2252c6295a0969fc250168e1fe78650, prev_block_hash=0000000000000000000434a0210f460be762de3a65b79517b9d537a8b10c4c71)
28/02/2021 02:43:49 [Responder] New block received (block_hash=0000000000000000000136c6998cad74e2252c6295a0969fc250168e1fe78650, prev_block_hash=0000000000000000000434a0210f460be762de3a65b79517b9d537a8b10c4c71)
28/02/2021 02:49:31 [ChainMonitor] New block received via zmq (block_hash=00000000000000000001a6277192705cdf28da247c64bd942d261f944a923d85)
28/02/2021 02:49:31 [Watcher] New block received (block_hash=00000000000000000001a6277192705cdf28da247c64bd942d261f944a923d85, prev_block_hash=0000000000000000000136c6998cad74e2252c6295a0969fc250168e1fe78650)
28/02/2021 02:49:31 [Responder] New block received (block_hash=00000000000000000001a6277192705cdf28da247c64bd942d261f944a923d85, prev_block_hash=0000000000000000000136c6998cad74e2252c6295a0969fc250168e1fe78650)
28/02/2021 02:50:41 [ChainMonitor] New block received via zmq (block_hash=0000000000000000000012dfc019be41ebffb805d43561327d667764e677c06e)
28/02/2021 02:50:41 [Watcher] New block received (block_hash=0000000000000000000012dfc019be41ebffb805d43561327d667764e677c06e, prev_block_hash=00000000000000000001a6277192705cdf28da247c64bd942d261f944a923d85)
28/02/2021 02:50:41 [Responder] New block received (block_hash=0000000000000000000012dfc019be41ebffb805d43561327d667764e677c06e, prev_block_hash=00000000000000000001a6277192705cdf28da247c64bd942d261f944a923d85)
28/02/2021 02:52:38 [ChainMonitor] New block received via zmq (block_hash=00000000000000000005f1b6b247dfaa6f6856686ae527b4ed0781ba18024daf)
28/02/2021 02:52:38 [Watcher] New block received (block_hash=00000000000000000005f1b6b247dfaa6f6856686ae527b4ed0781ba18024daf, prev_block_hash=0000000000000000000012dfc019be41ebffb805d43561327d667764e677c06e)
28/02/2021 02:52:38 [Responder] New block received (block_hash=00000000000000000005f1b6b247dfaa6f6856686ae527b4ed0781ba18024daf, prev_block_hash=0000000000000000000012dfc019be41ebffb805d43561327d667764e677c06e)
28/02/2021 03:01:05 [ChainMonitor] New block received via zmq (block_hash=000000000000000000004af116f2d8b8ae5d0e4e903bc66b5e69cd8ff58ff7b0)
28/02/2021 03:01:05 [Responder] New block received (block_hash=000000000000000000004af116f2d8b8ae5d0e4e903bc66b5e69cd8ff58ff7b0, prev_block_hash=00000000000000000005f1b6b247dfaa6f6856686ae527b4ed0781ba18024daf)
28/02/2021 03:01:05 [Watcher] New block received (block_hash=000000000000000000004af116f2d8b8ae5d0e4e903bc66b5e69cd8ff58ff7b0, prev_block_hash=00000000000000000005f1b6b247dfaa6f6856686ae527b4ed0781ba18024daf)
28/02/2021 03:19:30 [ChainMonitor] New block received via zmq (block_hash=00000000000000000001c526e814d73a729d98ada0874ba1f2c8da8a6e3b5cf2)
28/02/2021 03:19:30 [Watcher] New block received (block_hash=00000000000000000001c526e814d73a729d98ada0874ba1f2c8da8a6e3b5cf2, prev_block_hash=000000000000000000004af116f2d8b8ae5d0e4e903bc66b5e69cd8ff58ff7b0)
28/02/2021 03:19:30 [Responder] New block received (block_hash=00000000000000000001c526e814d73a729d98ada0874ba1f2c8da8a6e3b5cf2, prev_block_hash=000000000000000000004af116f2d8b8ae5d0e4e903bc66b5e69cd8ff58ff7b0)
28/02/2021 03:20:01 [ChainMonitor] New block received via zmq (block_hash=000000000000000000095ee2df60c299445cb226536a993a162781ecf5a70fcc)
28/02/2021 03:20:01 [Responder] New block received (block_hash=000000000000000000095ee2df60c299445cb226536a993a162781ecf5a70fcc, prev_block_hash=00000000000000000001c526e814d73a729d98ada0874ba1f2c8da8a6e3b5cf2)
28/02/2021 03:20:01 [Watcher] New block received (block_hash=000000000000000000095ee2df60c299445cb226536a993a162781ecf5a70fcc, prev_block_hash=00000000000000000001c526e814d73a729d98ada0874ba1f2c8da8a6e3b5cf2)
28/02/2021 03:21:08 [ChainMonitor] New block received via zmq (block_hash=000000000000000000046547ec9b1e8fe05446add5b0e54cab4222e85b02f087)
28/02/2021 03:21:08 [Responder] New block received (block_hash=000000000000000000046547ec9b1e8fe05446add5b0e54cab4222e85b02f087, prev_block_hash=000000000000000000095ee2df60c299445cb226536a993a162781ecf5a70fcc)
28/02/2021 03:21:08 [Watcher] New block received (block_hash=000000000000000000046547ec9b1e8fe05446add5b0e54cab4222e85b02f087, prev_block_hash=000000000000000000095ee2df60c299445cb226536a993a162781ecf5a70fcc)
28/02/2021 03:40:32 [ChainMonitor] New block received via zmq (block_hash=0000000000000000000abcd2c9949ffe2363d83f56a041cf31a8dd74af7d9882)
28/02/2021 03:40:32 [Responder] New block received (block_hash=0000000000000000000abcd2c9949ffe2363d83f56a041cf31a8dd74af7d9882, prev_block_hash=000000000000000000046547ec9b1e8fe05446add5b0e54cab4222e85b02f087)
28/02/2021 03:40:32 [Watcher] New block received (block_hash=0000000000000000000abcd2c9949ffe2363d83f56a041cf31a8dd74af7d9882, prev_block_hash=000000000000000000046547ec9b1e8fe05446add5b0e54cab4222e85b02f087)
28/02/2021 03:56:06 [ChainMonitor] New block received via zmq (block_hash=000000000000000000043d68823c5a8e6f5a7f62e4e04456ed4ad87a8b81bce4)
28/02/2021 03:56:07 [Responder] New block received (block_hash=000000000000000000043d68823c5a8e6f5a7f62e4e04456ed4ad87a8b81bce4, prev_block_hash=0000000000000000000abcd2c9949ffe2363d83f56a041cf31a8dd74af7d9882)
28/02/2021 03:56:07 [Watcher] New block received (block_hash=000000000000000000043d68823c5a8e6f5a7f62e4e04456ed4ad87a8b81bce4, prev_block_hash=0000000000000000000abcd2c9949ffe2363d83f56a041cf31a8dd74af7d9882)
28/02/2021 03:57:59 [ChainMonitor] New block received via zmq (block_hash=000000000000000000048bdea371aed1061bf0aea904a3a7f4e5c09e354e6951)
28/02/2021 03:57:59 [Watcher] New block received (block_hash=000000000000000000048bdea371aed1061bf0aea904a3a7f4e5c09e354e6951, prev_block_hash=000000000000000000043d68823c5a8e6f5a7f62e4e04456ed4ad87a8b81bce4)
28/02/2021 03:57:59 [Responder] New block received (block_hash=000000000000000000048bdea371aed1061bf0aea904a3a7f4e5c09e354e6951, prev_block_hash=000000000000000000043d68823c5a8e6f5a7f62e4e04456ed4ad87a8b81bce4)
28/02/2021 04:00:12 [ChainMonitor] New block received via zmq (block_hash=0000000000000000000c1154f8cb93095428a2e9f7c07022539ddc826d239134)
28/02/2021 04:00:12 [Watcher] New block received (block_hash=0000000000000000000c1154f8cb93095428a2e9f7c07022539ddc826d239134, prev_block_hash=000000000000000000048bdea371aed1061bf0aea904a3a7f4e5c09e354e6951)
28/02/2021 04:00:12 [Responder] New block received (block_hash=0000000000000000000c1154f8cb93095428a2e9f7c07022539ddc826d239134, prev_block_hash=000000000000000000048bdea371aed1061bf0aea904a3a7f4e5c09e354e6951)
28/02/2021 04:34:39 [ChainMonitor] New block received via zmq (block_hash=0000000000000000000cd9173e22e0eaaf82621c68d5fd758aa938e23bbf6e65)
28/02/2021 04:34:39 [Responder] New block received (block_hash=0000000000000000000cd9173e22e0eaaf82621c68d5fd758aa938e23bbf6e65, prev_block_hash=0000000000000000000c1154f8cb93095428a2e9f7c07022539ddc826d239134)
28/02/2021 04:34:39 [Watcher] New block received (block_hash=0000000000000000000cd9173e22e0eaaf82621c68d5fd758aa938e23bbf6e65, prev_block_hash=0000000000000000000c1154f8cb93095428a2e9f7c07022539ddc826d239134)
28/02/2021 04:49:57 [ChainMonitor] New block received via zmq (block_hash=000000000000000000033a358a3fa1614e9cb1047638fc5c5acd5ffd047dede2)
28/02/2021 04:49:57 [Watcher] New block received (block_hash=000000000000000000033a358a3fa1614e9cb1047638fc5c5acd5ffd047dede2, prev_block_hash=0000000000000000000cd9173e22e0eaaf82621c68d5fd758aa938e23bbf6e65)
28/02/2021 04:49:57 [Responder] New block received (block_hash=000000000000000000033a358a3fa1614e9cb1047638fc5c5acd5ffd047dede2, prev_block_hash=0000000000000000000cd9173e22e0eaaf82621c68d5fd758aa938e23bbf6e65)
28/02/2021 04:53:04 [ChainMonitor] New block received via zmq (block_hash=0000000000000000000b20cf49764981714c5251c3c58fcd567af6e6b8262596)
28/02/2021 04:53:04 [Watcher] New block received (block_hash=0000000000000000000b20cf49764981714c5251c3c58fcd567af6e6b8262596, prev_block_hash=000000000000000000033a358a3fa1614e9cb1047638fc5c5acd5ffd047dede2)
28/02/2021 04:53:05 [Responder] New block received (block_hash=0000000000000000000b20cf49764981714c5251c3c58fcd567af6e6b8262596, prev_block_hash=000000000000000000033a358a3fa1614e9cb1047638fc5c5acd5ffd047dede2)
28/02/2021 05:40:08 [ChainMonitor] New block received via zmq (block_hash=0000000000000000000003e82cd3154f5e7f9e47431c28e6f2af4f0c93eab28b)
28/02/2021 05:40:08 [Watcher] New block received (block_hash=0000000000000000000003e82cd3154f5e7f9e47431c28e6f2af4f0c93eab28b, prev_block_hash=0000000000000000000b20cf49764981714c5251c3c58fcd567af6e6b8262596)
28/02/2021 05:40:08 [Responder] New block received (block_hash=0000000000000000000003e82cd3154f5e7f9e47431c28e6f2af4f0c93eab28b, prev_block_hash=0000000000000000000b20cf49764981714c5251c3c58fcd567af6e6b8262596)
28/02/2021 05:52:43 [ChainMonitor] New block received via zmq (block_hash=000000000000000000057edf6958bc7afad1e8cc5b432a932a29151d75b82744)
28/02/2021 05:52:43 [Responder] New block received (block_hash=000000000000000000057edf6958bc7afad1e8cc5b432a932a29151d75b82744, prev_block_hash=0000000000000000000003e82cd3154f5e7f9e47431c28e6f2af4f0c93eab28b)
28/02/2021 05:52:43 [Watcher] New block received (block_hash=000000000000000000057edf6958bc7afad1e8cc5b432a932a29151d75b82744, prev_block_hash=0000000000000000000003e82cd3154f5e7f9e47431c28e6f2af4f0c93eab28b)
28/02/2021 06:30:25 [ChainMonitor] New block received via zmq (block_hash=000000000000000000091c2e41bc41f49c2a4fc30dc42ff43d07021c62844af4)
28/02/2021 06:30:25 [Watcher] New block received (block_hash=000000000000000000091c2e41bc41f49c2a4fc30dc42ff43d07021c62844af4, prev_block_hash=000000000000000000057edf6958bc7afad1e8cc5b432a932a29151d75b82744)
28/02/2021 06:30:25 [Responder] New block received (block_hash=000000000000000000091c2e41bc41f49c2a4fc30dc42ff43d07021c62844af4, prev_block_hash=000000000000000000057edf6958bc7afad1e8cc5b432a932a29151d75b82744)
28/02/2021 06:40:43 [ChainMonitor] New block received via zmq (block_hash=00000000000000000009a9562775b694776055fa283489e5ee374205069c3bea)
28/02/2021 06:40:43 [Watcher] New block received (block_hash=00000000000000000009a9562775b694776055fa283489e5ee374205069c3bea, prev_block_hash=000000000000000000091c2e41bc41f49c2a4fc30dc42ff43d07021c62844af4)
28/02/2021 06:40:43 [Responder] New block received (block_hash=00000000000000000009a9562775b694776055fa283489e5ee374205069c3bea, prev_block_hash=000000000000000000091c2e41bc41f49c2a4fc30dc42ff43d07021c62844af4)
28/02/2021 06:42:38 [ChainMonitor] New block received via zmq (block_hash=000000000000000000098a2408160d6d96304869470e79668c91aedbd7d5b292)
28/02/2021 06:42:39 [Responder] New block received (block_hash=000000000000000000098a2408160d6d96304869470e79668c91aedbd7d5b292, prev_block_hash=00000000000000000009a9562775b694776055fa283489e5ee374205069c3bea)
28/02/2021 06:42:39 [Watcher] New block received (block_hash=000000000000000000098a2408160d6d96304869470e79668c91aedbd7d5b292, prev_block_hash=00000000000000000009a9562775b694776055fa283489e5ee374205069c3bea)
28/02/2021 06:44:51 [ChainMonitor] New block received via zmq (block_hash=00000000000000000000446eaefee1ae3c7a02f1a97570253123580a02ce15a1)
28/02/2021 06:44:51 [Responder] New block received (block_hash=00000000000000000000446eaefee1ae3c7a02f1a97570253123580a02ce15a1, prev_block_hash=000000000000000000098a2408160d6d96304869470e79668c91aedbd7d5b292)
28/02/2021 06:44:51 [Watcher] New block received (block_hash=00000000000000000000446eaefee1ae3c7a02f1a97570253123580a02ce15a1, prev_block_hash=000000000000000000098a2408160d6d96304869470e79668c91aedbd7d5b292)
28/02/2021 07:05:54 [ChainMonitor] New block received via zmq (block_hash=0000000000000000000400a19ea3becf6f9e665e195980155f486ee557397e4e)
28/02/2021 07:05:54 [Responder] New block received (block_hash=0000000000000000000400a19ea3becf6f9e665e195980155f486ee557397e4e, prev_block_hash=00000000000000000000446eaefee1ae3c7a02f1a97570253123580a02ce15a1)
28/02/2021 07:05:54 [Watcher] New block received (block_hash=0000000000000000000400a19ea3becf6f9e665e195980155f486ee557397e4e, prev_block_hash=00000000000000000000446eaefee1ae3c7a02f1a97570253123580a02ce15a1)
28/02/2021 07:08:24 [ChainMonitor] New block received via zmq (block_hash=00000000000000000006f1ffe4aa24dce7f239e35a078692b65bf592b4b51949)
28/02/2021 07:08:24 [Responder] New block received (block_hash=00000000000000000006f1ffe4aa24dce7f239e35a078692b65bf592b4b51949, prev_block_hash=0000000000000000000400a19ea3becf6f9e665e195980155f486ee557397e4e)
28/02/2021 07:08:24 [Watcher] New block received (block_hash=0
Looks like there's some kind of connection problem between the tower and bitcoind. Let me look into it and see if I can figure out what's going on.
Would you mind also providing the deployment setup you used for btcpay in order to reproduce?
27/02/2021 16:01:33 [Daemon] Tower identity found. Loading keys 27/02/2021 16:01:33 [Daemon] ERROR: An error occurred: Cannot connect to bitcoind. Shutting down 27/02/2021 16:01:37 [Daemon] Tower identity found. Loading keys 27/02/2021 16:01:41 [Daemon] Tower identity found. Loading keys 27/02/2021 16:01:33 [Daemon] ERROR: An error occurred: Cannot connect to bitcoind. Shutting down
This first bit (I'm assuming the initial bootstrap) happens because bitcoind is not still launched (or at least accesible via RPC), so the teos container is keeping restarting until it gets to connect to bitcoind.
Exception in thread Thread-2: Traceback (most recent call last): File "/usr/local/lib/python3.9/threading.py", line 950, in _bootstrap_inner self.run() File "/usr/local/lib/python3.9/threading.py", line 888, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.9/site-packages/teos/chain_monitor.py", line 123, in monitor_chain_polling current_tip = self.block_processor.get_best_block_hash() File "/usr/local/lib/python3.9/site-packages/teos/block_processor.py", line 62, in get_best_block_hash block_hash = bitcoin_cli(self.btc_connect_params).getbestblockhash() File "/usr/local/lib/python3.9/site-packages/teos/utils/auth_proxy.py", line 151, in call response, status = self._request("POST", self.__url.path, postdata.encode("utf-8")) File "/usr/local/lib/python3.9/site-packages/teos/utils/auth_proxy.py", line 114, in _request self.__conn.request(method, path, postdata, headers) File "/usr/local/lib/python3.9/http/client.py", line 1255, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/local/lib/python3.9/http/client.py", line 1301, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/local/lib/python3.9/http/client.py", line 1250, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/local/lib/python3.9/http/client.py", line 1010, in _send_output self.send(msg) File "/usr/local/lib/python3.9/http/client.py", line 950, in send self.connect() File "/usr/local/lib/python3.9/http/client.py", line 921, in connect self.sock = self._create_connection( File "/usr/local/lib/python3.9/socket.py", line 843, in create_connection raise err File "/usr/local/lib/python3.9/socket.py", line 831, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused
This second bit happens because of the same reason, but this time while teos is running. Can you check if the bitcoind container is restarting from time to time?
This is related to #19. teos should halt and keep polling bitcoind if the later goes down. bitcoind is really stable though, so it crashing is something quite strange.
so I am using standard dockerized deployent, all services look normal, none have or report issues. Anything in specific should I provide? I could get you access to the server if you wish, its a testing pre production server, if you think could help debug faster. hence like mentioned this is a clean dockerized version nothing changed.
@uhlhosting I've been working on a patch for this type of crash. Will test it out on your environment.
https://github.com/talaia-labs/python-teos/pull/271
@uhlhosting
I've been taking a look at the logs and, as I expected, your bitcoind container has been resetting a lot. I say I expected it because that's the first reason that comes to mind for the tower to be unable to connect to bitcoind once it has been bootstrap, given the connection is tested during the bootstrap process and assumed valid from that point onwards. However, this is a rather weird thing, since bitcoind is a really stable software (I'm not that sure to bitcoind docker deployments though, since I'm not that used to them, but I'd expect similar performance).
Every time bitcoind was non-reachable the tower code crashed and therefore the container reset. This issue (the tower crashing if bitcoind was offline) has been known for a while, but it was a rather low priority issue given bitcoind does not tend to crash :S
Here's an example of a crash match:
bitcoind crashes
2021-03-05T14:29:39Z torcontrol thread exit
2021-03-05T14:29:39Z Shutdown: In progress...
2021-03-05T14:29:39Z addcon thread exit
2021-03-05T14:29:39Z net thread exit
2021-03-05T14:29:39Z msghand thread exit
2021-03-05T14:29:43Z opencon thread exit
...2021-03-05T14:29:48Z [default wallet] Releasing wallet
2021-03-05T14:29:49Z Shutdown: done
tower crashes
05/03/2021 14:30:20 [BlockProcessor] ERROR: Couldn't get block hash (error={'code': -28, 'message': 'Verifying blocks...'})
05/03/2021 14:32:50 [Daemon] Tower identity found. Loading keys
05/03/2021 14:32:52 [Daemon] tower_id = 026247def18440eb3a3e59be829b7c12dd2e86da6989b88644de306132adf72761
05/03/2021 14:32:53 [Daemon] Starting TEOS
05/03/2021 14:32:53 [Daemon] Fresh bootstrap
AFAICT I haven't seen any error on the bitcoind logs that was triggering a reset, so it's either something I'm unaware of, or it was being triggered from outside (docker?). Also, were you doing IBD on that node?
Looks like the tower has now been up for around 3 days with no issues.
We could try to patch #271 in and monitor the logs. That won't solve the underlying issue AFAICT (why bitcoind was reseting), but at least would prevent the tower from crashing if it does.
The patch has not been extensively tested, but I think we could go for it given you were running a testing setup.
I'll leave it up to you to decide though.
Hi @sr-gi we have a downtime due to the OVH DC Fire. No Data loss involved, yet this downtime...