python-teos icon indicating copy to clipboard operation
python-teos copied to clipboard

Published 20 hours ago • verified publisher icon talaia-labs

lightning-cli plugin stop /path/to/watchtower.py kills lightningd

Open svewa opened this issue 3 years ago • 3 comments

It seems when plugin stop is called on the watchtower right after Sending appointment, before Appointment accepted and signed, it crashes lightningd

022-02-09T19:15:37.554Z INFO    plugin-watchtower.py: Remaining slots: 159981005
2022-02-09T19:15:37.554Z INFO    plugin-watchtower.py: Start block: 722518
2022-02-09T19:15:37.607Z INFO    plugin-watchtower.py: Adding tower to Tower's db (id=$id)
2022-02-09T19:15:37.616Z INFO    plugin-watchtower.py: Sending appointment 531c53e4279195a08711033ed88aef73 to $id
2022-02-09T19:15:37.998Z INFO    plugin-watchtower.py: Killing plugin: stopped by lightningd via RPC
2022-02-09T19:15:37.999Z **BROKEN** lightningd: Already in transaction from lightningd/jsonrpc.c:948
2022-02-09T19:15:38.049Z **BROKEN** lightningd: FATAL SIGNAL 6 (version v0.10.2)
2022-02-09T19:15:38.049Z **BROKEN** lightningd: backtrace: common/daemon.c:38 (send_backtrace) 0x556f12436da1
2022-02-09T19:15:38.049Z **BROKEN** lightningd: backtrace: common/daemon.c:46 (crashdump) 0x556f12436df0
--
2022-02-09T21:24:59.406Z INFO    plugin-watchtower.py: Appointment accepted and signed by $id
2022-02-09T21:24:59.407Z INFO    plugin-watchtower.py: Remaining slots: 159979073
2022-02-09T21:24:59.407Z INFO    plugin-watchtower.py: Start block: 722533
2022-02-09T21:24:59.438Z INFO    plugin-watchtower.py: Adding tower to Tower's db (id=$id)
2022-02-09T21:24:59.584Z INFO    plugin-watchtower.py: Killing plugin: stopped by lightningd via RPC
2022-02-09T21:25:05.432Z INFO    plugin-watchtower.py: A key file for the watchtower client already exists. Loading it
2022-02-09T21:25:05.433Z INFO    plugin-watchtower.py: Plugin watchtower client initialized. User id = $uid
2022-02-09T21:25:08.916Z INFO    plugin-watchtower.py: Sending appointment 07dffc4311d765151942e619bee13e80 to $id
2022-02-09T21:25:09.689Z INFO    plugin-watchtower.py: Appointment accepted and signed by $id
--
2022-02-09T21:25:12.496Z INFO    plugin-watchtower.py: Appointment accepted and signed by $id
2022-02-09T21:25:12.496Z INFO    plugin-watchtower.py: Remaining slots: 159979069
2022-02-09T21:25:12.496Z INFO    plugin-watchtower.py: Start block: 722533
2022-02-09T21:25:12.522Z INFO    plugin-watchtower.py: Adding tower to Tower's db (id=$id)
2022-02-09T21:25:58.038Z INFO    plugin-watchtower.py: Killing plugin: stopped by lightningd via RPC
2022-02-09T21:26:02.717Z INFO    plugin-watchtower.py: A key file for the watchtower client already exists. Loading it
2022-02-09T21:26:02.718Z INFO    plugin-watchtower.py: Plugin watchtower client initialized. User id = $uid
2022-02-09T21:26:07.638Z INFO    plugin-watchtower.py: Killing plugin: stopped by lightningd via RPC
2022-02-09T21:26:10.497Z INFO    plugin-watchtower.py: A key file for the watchtower client already exists. Loading it
2022-02-09T21:26:10.498Z INFO    plugin-watchtower.py: Plugin watchtower client initialized. User id = $uid
2022-02-09T21:26:12.463Z INFO    plugin-watchtower.py: Killing plugin: stopped by lightningd via RPC
2022-02-09T21:26:14.627Z INFO    plugin-watchtower.py: A key file for the watchtower client already exists. Loading it
2022-02-09T21:26:14.627Z INFO    plugin-watchtower.py: Plugin watchtower client initialized. User id = $uid

svewa avatar Feb 09 '22 21:02 svewa

I'm kinda lost with this one. Looks like the plugin is indeed killing the daemon, but there is no backtrace from the plugin.

According to c-lightning docs on plugins, once stop is been called the plugin has up to 30 secs to exit or it'll be killed. But I'm unsure if the timeout is the reason of the issue or what.

022-02-09T19:15:37.554Z INFO plugin-watchtower.py: Remaining slots: 159981005 2022-02-09T19:15:37.554Z INFO plugin-watchtower.py: Start block: 722518 2022-02-09T19:15:37.607Z INFO plugin-watchtower.py: Adding tower to Tower's db (id=$id) 2022-02-09T19:15:37.616Z INFO plugin-watchtower.py: Sending appointment 531c53e4279195a08711033ed88aef73 to $id 2022-02-09T19:15:37.998Z INFO plugin-watchtower.py: Killing plugin: stopped by lightningd via RPC 2022-02-09T19:15:37.999Z BROKEN lightningd: Already in transaction from lightningd/jsonrpc.c:948 2022-02-09T19:15:38.049Z BROKEN lightningd: FATAL SIGNAL 6 (version v0.10.2) 2022-02-09T19:15:38.049Z BROKEN lightningd: backtrace: common/daemon.c:38 (send_backtrace) 0x556f12436da1 2022-02-09T19:15:38.049Z BROKEN lightningd: backtrace: common/daemon.c:46 (crashdump) 0x556f12436df0 --

Is this the end of the execution (i.e. the plugin kills the daemon right after it is called?)

2022-02-09T21:24:59.406Z INFO plugin-watchtower.py: Appointment accepted and signed by $id 2022-02-09T21:24:59.407Z INFO plugin-watchtower.py: Remaining slots: 159979073 2022-02-09T21:24:59.407Z INFO plugin-watchtower.py: Start block: 722533 2022-02-09T21:24:59.438Z INFO plugin-watchtower.py: Adding tower to Tower's db (id=$id) 2022-02-09T21:24:59.584Z INFO plugin-watchtower.py: Killing plugin: stopped by lightningd via RPC 2022-02-09T21:25:05.432Z INFO plugin-watchtower.py: A key file for the watchtower client already exists. Loading it 2022-02-09T21:25:05.433Z INFO plugin-watchtower.py: Plugin watchtower client initialized. User id = $uid 2022-02-09T21:25:08.916Z INFO plugin-watchtower.py: Sending appointment 07dffc4311d765151942e619bee13e80 to $id 2022-02-09T21:25:09.689Z INFO plugin-watchtower.py: Appointment accepted and signed by $id

Is this happening right after the daemon is restarted, are there any longs in the middle? Just trying to make sense of this.

I'll chat with the c-lightning folks anyway to see if there's something expected to be done when handling a stop signal.

sr-gi avatar Feb 14 '22 10:02 sr-gi

the log is complete, so nothing cut out but only on level INFO. Here is the log from the starting of lightningd on, first line is interesting, so I guess I started watchtower later manually.

2022-02-09T18:52:13.031Z INFO    plugin-watchtower.py: Killing plugin: exited before replying to getmanifest
2022-02-09T18:52:13.063Z INFO    plugin-backup.py: root        : Comparing backup version 1692532 versus first write version 1692533
2022-02-09T18:52:13.063Z INFO    plugin-backup.py: Comparing backup version 1692532 versus first write version 1692533
2022-02-09T18:52:13.063Z INFO    plugin-backup.py: root        : Versions match up
2022-02-09T18:52:13.063Z INFO    plugin-backup.py: Versions match up
2022-02-09T18:52:22.759Z INFO    plugin-bcli: bitcoin-cli initialized and connected to bitcoind.
2022-02-09T18:52:27.353Z INFO    plugin-autoclean: autocleaning every 600 seconds
2022-02-09T18:52:27.354Z INFO    lightningd: Restarting onchaind for channel 21995
2022-02-09T18:52:27.356Z UNUSUAL xxx-chan#xxx: Peer permanent failure in ONCHAIN: Funding transaction spent
2022-02-09T18:52:27.356Z INFO    xxx-chan#xxx: State changed from ONCHAIN to FUNDING_SPEND_SEEN
2022-02-09T18:52:27.379Z INFO    lightningd: --------------------------------------------------
2022-02-09T18:52:27.379Z INFO    lightningd: Server started with public key xxx, alias xxx (color #xxx) and lightningd v0.10.2
2022-02-09T18:52:27.385Z INFO    xxx-chan#xxx: State changed from FUNDING_SPEND_SEEN to ONCHAIN
[some channeld and feeadjuster stuff...]
2022-02-09T19:14:37.724Z INFO    plugin-watchtower.py: A key file for the watchtower client already exists. Loading it
2022-02-09T19:14:37.725Z INFO    plugin-watchtower.py: Plugin watchtower client initialized. User id = UID
2022-02-09T19:14:37.901Z INFO    plugin-watchtower.py: Sending appointment e71eaa9ed8aafbffb460d9e17a4ae493 to ID
2022-02-09T19:14:38.308Z INFO    plugin-watchtower.py: Appointment accepted and signed by ID
2022-02-09T19:14:38.308Z INFO    plugin-watchtower.py: Remaining slots: 159981026
2022-02-09T19:14:38.308Z INFO    plugin-watchtower.py: Start block: 722518
2022-02-09T19:14:38.336Z INFO    plugin-watchtower.py: Adding tower to Tower's db (id=ID)
2022-02-09T19:14:38.372Z INFO    plugin-watchtower.py: Sending appointment ca62a33c46ba57d25c5f8a18a080fd7f to ID
2022-02-09T19:14:38.763Z INFO    plugin-watchtower.py: Appointment accepted and signed by ID
2022-02-09T19:14:38.763Z INFO    plugin-watchtower.py: Remaining slots: 159981025
2022-02-09T19:14:38.763Z INFO    plugin-watchtower.py: Start block: 722518
2022-02-09T19:14:38.794Z INFO    plugin-watchtower.py: Adding tower to Tower's db (id=ID)
2022-02-09T19:14:38.804Z INFO    plugin-watchtower.py: Sending appointment 9fe3a076ce28ddcad31b595f90d1cd21 to ID
2022-02-09T19:14:39.174Z INFO    plugin-watchtower.py: Appointment accepted and signed by ID
2022-02-09T19:14:39.174Z INFO    plugin-watchtower.py: Remaining slots: 159981024
2022-02-09T19:14:39.174Z INFO    plugin-watchtower.py: Start block: 722518
2022-02-09T19:14:39.199Z INFO    plugin-watchtower.py: Adding tower to Tower's db (id=ID)
2022-02-09T19:14:45.144Z INFO    plugin-watchtower.py: Sending appointment ba890ec58de109b01c93fb3dc9fc8984 to ID
2022-02-09T19:14:45.623Z INFO    plugin-watchtower.py: Appointment accepted and signed by ID
2022-02-09T19:14:45.623Z INFO    plugin-watchtower.py: Remaining slots: 159981023
2022-02-09T19:14:45.623Z INFO    plugin-watchtower.py: Start block: 722518
2022-02-09T19:14:45.648Z INFO    plugin-watchtower.py: Adding tower to Tower's db (id=ID)
2022-02-09T19:14:45.656Z INFO    plugin-watchtower.py: Sending appointment e33f44c156c1a02dfb851a37826e08c1 to ID
2022-02-09T19:14:46.210Z INFO    plugin-watchtower.py: Appointment accepted and signed by ID
2022-02-09T19:14:46.211Z INFO    plugin-watchtower.py: Remaining slots: 159981022
2022-02-09T19:14:46.211Z INFO    plugin-watchtower.py: Start block: 722518
2022-02-09T19:14:46.236Z INFO    plugin-watchtower.py: Adding tower to Tower's db (id=ID)
2022-02-09T19:14:46.248Z INFO    plugin-watchtower.py: Sending appointment 525f50ec5ea519284c0a6b8ecf7b9376 to ID
2022-02-09T19:14:46.906Z INFO    plugin-watchtower.py: Appointment accepted and signed by ID
2022-02-09T19:14:46.906Z INFO    plugin-watchtower.py: Remaining slots: 159981021
2022-02-09T19:14:46.906Z INFO    plugin-watchtower.py: Start block: 722518
2022-02-09T19:14:46.930Z INFO    plugin-watchtower.py: Adding tower to Tower's db (id=ID)
2022-02-09T19:14:46.944Z INFO    plugin-watchtower.py: Sending appointment e9a8e41446e8a630f26dae483190d9c0 to ID
2022-02-09T19:14:47.445Z INFO    plugin-watchtower.py: Appointment accepted and signed by ID
2022-02-09T19:14:47.446Z INFO    plugin-watchtower.py: Remaining slots: 159981020
2022-02-09T19:14:47.446Z INFO    plugin-watchtower.py: Start block: 722518
2022-02-09T19:14:47.471Z INFO    plugin-watchtower.py: Adding tower to Tower's db (id=ID)
2022-02-09T19:14:47.501Z INFO    plugin-watchtower.py: Sending appointment 0fa44ef68702e489bec0a48ab8a5d09e to ID
2022-02-09T19:14:47.940Z INFO    plugin-watchtower.py: Appointment accepted and signed by ID
2022-02-09T19:14:47.941Z INFO    plugin-watchtower.py: Remaining slots: 159981019
2022-02-09T19:14:47.941Z INFO    plugin-watchtower.py: Start block: 722518
2022-02-09T19:14:47.968Z INFO    plugin-watchtower.py: Adding tower to Tower's db (id=ID)
2022-02-09T19:14:47.980Z INFO    plugin-watchtower.py: Sending appointment f1e71bbb8f941e81422f68b78d21b8f2 to ID
2022-02-09T19:14:48.390Z INFO    plugin-watchtower.py: Appointment accepted and signed by ID
2022-02-09T19:14:48.390Z INFO    plugin-watchtower.py: Remaining slots: 159981018
2022-02-09T19:14:48.391Z INFO    plugin-watchtower.py: Start block: 722518
2022-02-09T19:14:48.414Z INFO    plugin-watchtower.py: Adding tower to Tower's db (id=ID)
2022-02-09T19:14:51.856Z INFO    plugin-watchtower.py: Sending appointment 907473c96b961514edc21efecbe05084 to ID
2022-02-09T19:14:52.286Z INFO    plugin-watchtower.py: Appointment accepted and signed by ID
2022-02-09T19:14:52.286Z INFO    plugin-watchtower.py: Remaining slots: 159981017
2022-02-09T19:14:52.286Z INFO    plugin-watchtower.py: Start block: 722518
2022-02-09T19:14:52.309Z INFO    plugin-watchtower.py: Adding tower to Tower's db (id=ID)
2022-02-09T19:14:52.634Z INFO    plugin-watchtower.py: Sending appointment 5d7a19e1f875c6e312b356ea1691ce0f to ID
2022-02-09T19:14:53.311Z INFO    plugin-watchtower.py: Appointment accepted and signed by ID
2022-02-09T19:14:53.311Z INFO    plugin-watchtower.py: Remaining slots: 159981016
2022-02-09T19:14:53.311Z INFO    plugin-watchtower.py: Start block: 722518
2022-02-09T19:14:53.335Z INFO    plugin-watchtower.py: Adding tower to Tower's db (id=ID)
2022-02-09T19:15:05.888Z INFO    plugin-watchtower.py: Sending appointment c49d05b68c5ffb05eb6fc7c47b956b34 to ID
2022-02-09T19:15:06.430Z INFO    plugin-watchtower.py: Appointment accepted and signed by ID
2022-02-09T19:15:06.431Z INFO    plugin-watchtower.py: Remaining slots: 159981015
2022-02-09T19:15:06.431Z INFO    plugin-watchtower.py: Start block: 722518
2022-02-09T19:15:06.453Z INFO    plugin-watchtower.py: Adding tower to Tower's db (id=ID)
2022-02-09T19:15:06.696Z INFO    plugin-watchtower.py: Sending appointment 917a5a1c7446da382a67ed47213c23f2 to ID
2022-02-09T19:15:07.233Z INFO    plugin-watchtower.py: Appointment accepted and signed by ID
2022-02-09T19:15:07.233Z INFO    plugin-watchtower.py: Remaining slots: 159981014
2022-02-09T19:15:07.233Z INFO    plugin-watchtower.py: Start block: 722518
2022-02-09T19:15:07.263Z INFO    plugin-watchtower.py: Adding tower to Tower's db (id=ID)
2022-02-09T19:15:16.156Z INFO    plugin-watchtower.py: Sending appointment 14abc1ee66f20ce20cbb818d6b96a186 to ID
2022-02-09T19:15:16.611Z INFO    plugin-watchtower.py: Appointment accepted and signed by ID
2022-02-09T19:15:16.611Z INFO    plugin-watchtower.py: Remaining slots: 159981013
2022-02-09T19:15:16.611Z INFO    plugin-watchtower.py: Start block: 722518
2022-02-09T19:15:16.634Z INFO    plugin-watchtower.py: Adding tower to Tower's db (id=ID)
2022-02-09T19:15:16.952Z INFO    plugin-watchtower.py: Sending appointment 1e039ad43e8863a99d8608bb7146f604 to ID
2022-02-09T19:15:17.280Z INFO    plugin-watchtower.py: Appointment accepted and signed by ID
2022-02-09T19:15:17.281Z INFO    plugin-watchtower.py: Remaining slots: 159981012
2022-02-09T19:15:17.281Z INFO    plugin-watchtower.py: Start block: 722518
2022-02-09T19:15:17.304Z INFO    plugin-watchtower.py: Adding tower to Tower's db (id=ID)
2022-02-09T19:15:20.442Z INFO    plugin-watchtower.py: Sending appointment fb89bbb0e574f89609e94901ade1eb20 to ID
2022-02-09T19:15:21.046Z INFO    plugin-watchtower.py: Appointment accepted and signed by ID
2022-02-09T19:15:21.046Z INFO    plugin-watchtower.py: Remaining slots: 159981011
2022-02-09T19:15:21.046Z INFO    plugin-watchtower.py: Start block: 722518
2022-02-09T19:15:21.070Z INFO    plugin-watchtower.py: Adding tower to Tower's db (id=ID)
2022-02-09T19:15:21.223Z INFO    plugin-watchtower.py: Sending appointment 51759524235c71ae3d71adc34daf4c68 to ID
2022-02-09T19:15:21.921Z INFO    plugin-watchtower.py: Appointment accepted and signed by ID
2022-02-09T19:15:21.922Z INFO    plugin-watchtower.py: Remaining slots: 159981010
2022-02-09T19:15:21.922Z INFO    plugin-watchtower.py: Start block: 722518
2022-02-09T19:15:21.948Z INFO    plugin-watchtower.py: Adding tower to Tower's db (id=ID)
2022-02-09T19:15:33.586Z INFO    plugin-watchtower.py: Sending appointment 66b439d24cd2647a1eff547cbff97ca0 to ID
2022-02-09T19:15:34.192Z INFO    plugin-watchtower.py: Appointment accepted and signed by ID
2022-02-09T19:15:34.192Z INFO    plugin-watchtower.py: Remaining slots: 159981009
2022-02-09T19:15:34.192Z INFO    plugin-watchtower.py: Start block: 722518
2022-02-09T19:15:34.215Z INFO    plugin-watchtower.py: Adding tower to Tower's db (id=ID)
2022-02-09T19:15:34.228Z INFO    plugin-watchtower.py: Sending appointment 67ecc7b5208d47b0622c418663a2013f to ID
2022-02-09T19:15:34.824Z INFO    plugin-watchtower.py: Appointment accepted and signed by ID
2022-02-09T19:15:34.824Z INFO    plugin-watchtower.py: Remaining slots: 159981008
2022-02-09T19:15:34.824Z INFO    plugin-watchtower.py: Start block: 722518
2022-02-09T19:15:34.854Z INFO    plugin-watchtower.py: Adding tower to Tower's db (id=ID)
2022-02-09T19:15:35.433Z INFO    plugin-watchtower.py: Sending appointment 5cfecaf7f16acf080fa5790eaa04dc13 to ID
2022-02-09T19:15:35.839Z INFO    plugin-watchtower.py: Appointment accepted and signed by ID
2022-02-09T19:15:35.839Z INFO    plugin-watchtower.py: Remaining slots: 159981007
2022-02-09T19:15:35.839Z INFO    plugin-watchtower.py: Start block: 722518
2022-02-09T19:15:35.862Z INFO    plugin-watchtower.py: Adding tower to Tower's db (id=ID)
2022-02-09T19:15:35.946Z INFO    plugin-watchtower.py: Sending appointment 25c19243e57fcb853ef50e5353549cbb to ID
2022-02-09T19:15:36.300Z INFO    plugin-watchtower.py: Appointment accepted and signed by ID
2022-02-09T19:15:36.300Z INFO    plugin-watchtower.py: Remaining slots: 159981006
2022-02-09T19:15:36.300Z INFO    plugin-watchtower.py: Start block: 722518
2022-02-09T19:15:36.324Z INFO    plugin-watchtower.py: Adding tower to Tower's db (id=ID)
2022-02-09T19:15:37.035Z INFO    plugin-watchtower.py: Sending appointment 93bea258b8a65922f2b2a67fe32f6fd2 to ID
2022-02-09T19:15:37.553Z INFO    plugin-watchtower.py: Appointment accepted and signed by ID
2022-02-09T19:15:37.554Z INFO    plugin-watchtower.py: Remaining slots: 159981005
2022-02-09T19:15:37.554Z INFO    plugin-watchtower.py: Start block: 722518
2022-02-09T19:15:37.607Z INFO    plugin-watchtower.py: Adding tower to Tower's db (id=ID)
2022-02-09T19:15:37.616Z INFO    plugin-watchtower.py: Sending appointment 531c53e4279195a08711033ed88aef73 to ID
2022-02-09T19:15:37.998Z INFO    plugin-watchtower.py: Killing plugin: stopped by lightningd via RPC
2022-02-09T19:15:37.999Z **BROKEN** lightningd: Already in transaction from lightningd/jsonrpc.c:948
2022-02-09T19:15:38.049Z **BROKEN** lightningd: FATAL SIGNAL 6 (version v0.10.2)
2022-02-09T19:15:38.049Z **BROKEN** lightningd: backtrace: common/daemon.c:38 (send_backtrace) 0x556f12436da1
2022-02-09T19:15:38.049Z **BROKEN** lightningd: backtrace: common/daemon.c:46 (crashdump) 0x556f12436df0
2022-02-09T19:15:38.049Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7f2497a1dd5f
2022-02-09T19:15:38.049Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7f2497a1dce1
2022-02-09T19:15:38.049Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7f2497a07536
2022-02-09T19:15:38.049Z **BROKEN** lightningd: backtrace: lightningd/log.c:804 (fatal) 0x556f123da547
2022-02-09T19:15:38.049Z **BROKEN** lightningd: backtrace: wallet/db.c:1060 (db_begin_transaction_) 0x556f1240f85a
2022-02-09T19:15:38.049Z **BROKEN** lightningd: backtrace: lightningd/plugin_hook.c:205 (plugin_hook_callback) 0x556f124058c7
2022-02-09T19:15:38.049Z **BROKEN** lightningd: backtrace: lightningd/plugin_hook.c:131 (plugin_hook_killed) 0x556f124055ec
2022-02-09T19:15:38.049Z **BROKEN** lightningd: backtrace: ccan/ccan/tal/tal.c:240 (notify) 0x556f124ac61d
2022-02-09T19:15:38.049Z **BROKEN** lightningd: backtrace: ccan/ccan/tal/tal.c:402 (del_tree) 0x556f124acb0c
2022-02-09T19:15:38.049Z **BROKEN** lightningd: backtrace: ccan/ccan/tal/tal.c:412 (del_tree) 0x556f124acb5e
2022-02-09T19:15:38.049Z **BROKEN** lightningd: backtrace: ccan/ccan/tal/tal.c:486 (tal_free) 0x556f124ace98
2022-02-09T19:15:38.049Z **BROKEN** lightningd: backtrace: lightningd/plugin.c:345 (plugin_kill) 0x556f123ff302
2022-02-09T19:15:38.049Z **BROKEN** lightningd: backtrace: lightningd/plugin_control.c:116 (plugin_stop) 0x556f12404830
2022-02-09T19:15:38.049Z **BROKEN** lightningd: backtrace: lightningd/plugin_control.c:171 (plugin_dynamic_stop) 0x556f12404aaa
2022-02-09T19:15:38.049Z **BROKEN** lightningd: backtrace: lightningd/plugin_control.c:229 (json_plugin_control) 0x556f12404ce3
2022-02-09T19:15:38.049Z **BROKEN** lightningd: backtrace: lightningd/jsonrpc.c:627 (command_exec) 0x556f123d3792
2022-02-09T19:15:38.050Z **BROKEN** lightningd: backtrace: lightningd/jsonrpc.c:762 (rpc_command_hook_final) 0x556f123d3d97
2022-02-09T19:15:38.050Z **BROKEN** lightningd: backtrace: lightningd/plugin_hook.c:275 (plugin_hook_call_) 0x556f12405c63
2022-02-09T19:15:38.050Z **BROKEN** lightningd: backtrace: lightningd/jsonrpc.c:850 (plugin_hook_call_rpc_command) 0x556f123d4188
2022-02-09T19:15:38.050Z **BROKEN** lightningd: backtrace: lightningd/jsonrpc.c:949 (parse_request) 0x556f123d4656
2022-02-09T19:15:38.050Z **BROKEN** lightningd: backtrace: lightningd/jsonrpc.c:1040 (read_json) 0x556f123d49db
2022-02-09T19:15:38.050Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:59 (next_plan) 0x556f1249b90d
2022-02-09T19:15:38.050Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:407 (do_plan) 0x556f1249c4b7
2022-02-09T19:15:38.050Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:417 (io_ready) 0x556f1249c4f5
2022-02-09T19:15:38.050Z **BROKEN** lightningd: backtrace: ccan/ccan/io/poll.c:453 (io_loop) 0x556f1249e6cb
2022-02-09T19:15:38.050Z **BROKEN** lightningd: backtrace: lightningd/io_loop_with_timers.c:21 (io_loop_with_timers) 0x556f123d19df
2022-02-09T19:15:38.050Z **BROKEN** lightningd: backtrace: lightningd/lightningd.c:1163 (main) 0x556f123d78d7
2022-02-09T19:15:38.050Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7f2497a08d09
2022-02-09T19:15:38.050Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x556f123ad399
2022-02-09T19:15:38.050Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0xffffffffffffffff

svewa avatar Feb 14 '22 12:02 svewa

@svewa I've opened an issue on the c-lightning's repo after chatting with @cdecker since it seems it seems the root of the issue may come from how blocking hooks are dealt with. Feel free to subscribe to that if you're willing to give us a hand testing this.

sr-gi avatar Feb 14 '22 14:02 sr-gi