lightning
lightning copied to clipboard
ElementsProject
v0.12.0 does not start after updating from v0.11.1
Issue and Steps to Reproduce
After updating like this
cd clightning-v0.12.0/usr
sudo systemctl stop lightningd.service
sudo cp -v -r * /usr/
sudo systemctl start lightningd.service
from v0.11.1 to v0.12.0 (with clightning-v0.12.0-Ubuntu-20.04.tar.xz), when I restart the daemon the following is the status:
● lightningd.service - C-Lightning daemon
Loaded: loaded (/etc/systemd/system/lightningd.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2022-08-25 09:55:03 EDT; 32s ago
Process: 2410921 ExecStart=/usr/bin/lightningd --lightning-dir=/mnt/md0/C-Lightning/ --daemon --lo>
Aug 25 09:55:03 BTCPayServer systemd[1]: lightningd.service: Scheduled restart job, restart counter is>
Aug 25 09:55:03 BTCPayServer systemd[1]: Stopped C-Lightning daemon.
Aug 25 09:55:03 BTCPayServer systemd[1]: lightningd.service: Start request repeated too quickly.
Aug 25 09:55:03 BTCPayServer systemd[1]: lightningd.service: Failed with result 'exit-code'.
Aug 25 09:55:03 BTCPayServer systemd[1]: Failed to start C-Lightning daemon.
Rolled back to the v0.11.1 and it works again:
● lightningd.service - C-Lightning daemon
Loaded: loaded (/etc/systemd/system/lightningd.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2022-08-25 09:56:51 EDT; 15min ago
Process: 2411189 ExecStart=/usr/bin/lightningd --lightning-dir=/mnt/md0/C-Lightning/ --daemon --log-file=/mnt/md0/C-Lightning/log --pid-file=/run/lightningd/lightningd.pid (code=exited, status=0/SUCCESS)
Main PID: 2411190 (lightningd)
Tasks: 33 (limit: 14151)
Memory: 387.7M
getinfo output
What did you see in the logs? and also if you have no logs what happens if you start lightningd from the command line?
After trying to start it from the command line:
sudo lightningd --lightning-dir=/mnt/md0/C-Lightning/ --daemon --log-file=/mnt/md0/C-Lightning/log
Traceback (most recent call last):
File "/mnt/md0/C-Lightning/plugins/drain/drain.py", line 2, in <module>
from pyln.client import Plugin, Millisatoshi, RpcError
ModuleNotFoundError: No module named 'pyln'
Traceback (most recent call last):
File "/mnt/md0/C-Lightning/plugins/rebalance/rebalance.py", line 2, in <module>
from pyln.client import Plugin, Millisatoshi, RpcError
ModuleNotFoundError: No module named 'pyln'
Could not connect to bitcoind using bitcoin-cli. Is bitcoind running?
Make sure you have bitcoind running and that bitcoin-cli is able to connect to bitcoind.
You can verify that your Bitcoin Core installation is ready for use by running:
$ bitcoin-cli echo 'hello world'
The Bitcoin backend died.
Bitcoind is running:
bitcoin-cli echo 'hello world'
[
"hello world"
]
bitcoin-cli -getinfo
Chain: main
Blocks: 751077
Headers: 751077
Verification progress: 99.9997%
Difficulty: 28351606743493.77
Network: in 5, out 11, total 16
Version: 230000
Time offset (s): -3
Proxies: 127.0.0.1:9050 (ipv4, ipv6, onion, cjdns)
Min tx relay fee rate (BTC/kvB): 0.00001000
Wallet: ""
Keypool size: 1000
Unlocked until: 0
Transaction fee rate (-paytxfee) (BTC/kvB): 0.00000000
Balance: 0.00000000
Warnings:
LOG
2022-08-25T15:21:24.885Z INFO plugin-drain.py: Killing plugin: exited before replying to getmanifest
2022-08-25T15:21:24.909Z INFO plugin-rebalance.py: Killing plugin: exited before replying to getmanifest
2022-08-25T15:22:47.592Z INFO plugin-drain.py: Killing plugin: exited before replying to getmanifest
2022-08-25T15:22:47.613Z INFO plugin-rebalance.py: Killing plugin: exited before replying to getmanifest
2022-08-25T15:22:50.987Z **BROKEN** plugin-bcli: \nCould not connect to bitcoind using bitcoin-cli. Is bitcoind running?\n\nMake sure >
2022-08-25T15:22:50.988Z INFO plugin-bcli: Killing plugin: exited before we sent init
looks like that cln is not able to contact the bitcoin backend, from the log plugin-bcli: \nCould not connect to bitcoind using bitcoin-cli. Is bitcoind running
But also it looks like you have some python plugin? from the following exception
Traceback (most recent call last):
File "/mnt/md0/C-Lightning/plugins/drain/drain.py", line 2, in <module>
from pyln.client import Plugin, Millisatoshi, RpcError
ModuleNotFoundError: No module named 'pyln'
Traceback (most recent call last):
File "/mnt/md0/C-Lightning/plugins/rebalance/rebalance.py", line 2, in <module>
from pyln.client import Plugin, Millisatoshi, RpcError
ModuleNotFoundError: No module named 'pyln'
BTW, your bitcoin core is running :/ and can you hare your cln conf too?
Remember, the version v0.11.1 has been running for awhile and I was able to easily roll back to it from the v0.12.0. The v.0.11.1 is running now with no problem.
The two plugins are from here https://github.com/lightningd/plugins/tree/master/drain and here https://github.com/lightningd/plugins/tree/master/rebalance and they seems to not affect v0.11.1.
Bitcoin is running and this is the cln config:
#Lightning node customization options
alias=BitcoinBellyButton
fee-base=0
fee-per-satoshi=0
min-capacity-sat=2000000
network=bitcoin
large-channels
#lightning-dir=/mnt/md0/C-Lightning/
#Lightning channel and HTLC options
max-concurrent-htlcs=200
autocleaninvoice-cycle=1800
autocleaninvoice-expired-by=3600
#Networking Options
bind-addr=127.0.0.1:9735
proxy=127.0.0.1:9050
announce-addr=szq5owdupc5z4v3keh3xppkcerlamgffr2nfeexmimfm65u37e6nruqd.onion:9735
#Plugins
plugin=/home/go/c-lightning-REST/plugin.js
rest-port=3001
rest-docport=4001
rest-protocol=https
Default. Almost... Using a symbolic link in the bitcoin default directory to the actual location on the RAID 1 disks.
~/.bitcoin$ ll
total 12
drwxrwxr-x 3 go go 4096 Aug 25 20:35 ./
drwxr-xr-x 43 go go 4096 Aug 25 21:13 ../
lrwxrwxrwx 1 go go 33 Dec 7 2021 bitcoin.conf -> /mnt/md0/BitcoinData/bitcoin.conf*
drwxrwxr-x 2 go go 4096 May 19 2020 wallets/
Easiest way to verify that lightningd should be able to contact bitcoind is by just tring bitcoin-cli as the node would itself:
- If you are running the node as a different user, switch to that user first
- Call
bitcoin-cli echo Hi, and see if you getHiback. If you specify any--bitcoin-rpcuseror--bitcoin-rpcpasswordoptions, then specify them (without the--bitcoin-prefix` on that command line too
If you don't get a Hi back then lightningd will not be able to use bitcoin-cli to talk to bitcoind either.
Guys, sorry but the problem is ONLY with the new release.
I'v been running this ln node for more than a year. The bitcoin node is even older...BTCPayServer is installed on this machine as well working like a charm...
CLN 0.11.1 works fine BUT when I updated it to 0.12.0 it stopped working so I had to roll back.
Running now:
"version": "0.11.1", "blockheight": 751871, "network": "bitcoin", "msatoshi_fees_collected": 12967902, "fees_collected_msat": "12967902msat", "lightning-dir": "/mnt/md0/C-Lightning/bitcoin",
And using the cookie:
#disk locations datadir=/mnt/md0/BitcoinData rpccookiefile=/mnt/md0/BitcoinData/.cookie
Rollback? That should not be possible since there were DB updates inbetween if I'm not mistaken.
Are you sure you were running the bitcoin-cli with the user that is specified in the lightningd.service file?
Because 0.12.0 was never able to even start:
Issue and Steps to Reproduce
After updating like this
cd clightning-v0.12.0/usr sudo systemctl stop lightningd.service sudo cp -v -r * /usr/ sudo systemctl start lightningd.servicefrom v0.11.1 to v0.12.0 (with clightning-v0.12.0-Ubuntu-20.04.tar.xz), when I restart the daemon the following is the status:
● lightningd.service - C-Lightning daemon Loaded: loaded (/etc/systemd/system/lightningd.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Thu 2022-08-25 09:55:03 EDT; 32s ago Process: 2410921 ExecStart=/usr/bin/lightningd --lightning-dir=/mnt/md0/C-Lightning/ --daemon --lo> Aug 25 09:55:03 BTCPayServer systemd[1]: lightningd.service: Scheduled restart job, restart counter is> Aug 25 09:55:03 BTCPayServer systemd[1]: Stopped C-Lightning daemon. Aug 25 09:55:03 BTCPayServer systemd[1]: lightningd.service: Start request repeated too quickly. Aug 25 09:55:03 BTCPayServer systemd[1]: lightningd.service: Failed with result 'exit-code'. Aug 25 09:55:03 BTCPayServer systemd[1]: Failed to start C-Lightning daemon.Rolled back to the v0.11.1 and it works again:
● lightningd.service - C-Lightning daemon Loaded: loaded (/etc/systemd/system/lightningd.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2022-08-25 09:56:51 EDT; 15min ago Process: 2411189 ExecStart=/usr/bin/lightningd --lightning-dir=/mnt/md0/C-Lightning/ --daemon --log-file=/mnt/md0/C-Lightning/log --pid-file=/run/lightningd/lightningd.pid (code=exited, status=0/SUCCESS) Main PID: 2411190 (lightningd) Tasks: 33 (limit: 14151) Memory: 387.7M
getinfooutput
Are you sure you were running the
bitcoin-cliwith the user that is specified in thelightningd.servicefile?
Yes
The question is: Why when I replace the 0.11.1 files in /usr with the 0.12.0, and without touching anything else, CLN can't start? Nothing else changed.
After that, why when I replace back the 0.12.0 files in /usr with the 0.11.1, and without touching anything else, does CLN start and works normally?
Nothing else changed.
Well I manage to isolate the bug.
Until v0.11.1, CLN was ABLE to start loading c-lightning-REST as specified in the config file:
#Plugins
plugin=/home/go/c-lightning-REST/plugin.js
rest-port=3001
rest-docport=4001
rest-protocol=https
AND after that, it loaded the plugins from the .lightning/plugins sub-directories with no problems.
With the latest v0.12.0 apparently, loading the plugins from the .lightning/plugins sub-directories does not work anymore:
plugin-plugin.js: Killing plugin: exited before replying to getmanifest
I renamed the .lightning/plugins directory as lightning/plugins.bak and CLN 0.12.0 finally starts!
2022-09-20T17:40:19.764Z INFO lightningd: Server started with public key 0228cc784d015731cc0c5dc2163c8bb856f59feb3234ce813edeccf2e26733c32d, alias BitcoinBellyButton (color #0228cc) and lightningd 0.12.0
2022-09-20T17:40:21.831Z INFO plugin-chanbackup: Creating Emergency Recovery
2022-09-20T17:40:21.831Z INFO plugin-bookkeeper: Creating database
2022-09-20T17:40:21.831Z UNUSUAL plugin-plugin.js: --- Starting the cl-rest server ---
2022-09-20T17:40:21.831Z UNUSUAL plugin-plugin.js: --- cl-rest api server is ready and listening on port: 3001 ---
2022-09-20T17:40:21.831Z UNUSUAL plugin-plugin.js: --- cl-rest doc server is ready and listening on port: 4001 ---
So something changed in the plugins management generating the problem I had.
Question now is, how am I supposed to load the rebalance.py plugin (the one I really need...)? Well I found the answer, I just added it to the config file and it loads.
2022-09-20T18:02:00.059Z INFO lightningd: Server started with public key 0228cc784d015731cc0c5dc2163c8bb856f59feb3234ce813edeccf2e26733c32d, alias BitcoinBellyButton (color #0228cc) and lightningd 0.12.0
2022-09-20T18:02:00.077Z INFO plugin-rebalance.py: Plugin rebalance initialized with 0msat base / 0 ppm fee cltv_final:18 maxhops:5 msatfactor:4.0 erringnodes:5 getroute:getroute_iterative
2022-09-20T18:02:03.293Z UNUSUAL plugin-plugin.js: --- Starting the cl-rest server ---
2022-09-20T18:02:03.293Z UNUSUAL plugin-plugin.js: --- cl-rest api server is ready and listening on port: 3001 ---
2022-09-20T18:02:03.293Z UNUSUAL plugin-plugin.js: --- cl-rest doc server is ready and listening on port: 4001 ---
Apparently the plugins directory method doesn't work anymore starting with this release?
Ok so it sounds like one of your plugins is failing to start, and taking the node down with it. We changed the msats APIs recently, which caused problems for a number of plugins.
You can try adding plugins back in one by one, until you isolate the one that's causing the node to fail to start. That'd be useful for figuring out exactly where the problem is.
@niftynei , the plugin WORKS if started from within the config file:
#Plugins
plugin=/home/go/c-lightning-REST/plugin.js
rest-port=3001
rest-docport=4001
rest-protocol=https
plugin=/home/go/.lightning/plugins.bak/rebalance/rebalance.py
systemctl status output:
CGroup: /system.slice/lightningd.service
├─3143207 /usr/bin/lightningd --lightning-dir=/mnt/md0/C-Lightning/ --daemon --log-file=/mnt/md0/C-Lightning/log --pid-file=/run/lightningd/lightningd.pid
├─3143208 /usr/libexec/c-lightning/plugins/autoclean
├─3143209 /usr/libexec/c-lightning/plugins/chanbackup
├─3143210 /usr/libexec/c-lightning/plugins/bcli
├─3143211 /usr/libexec/c-lightning/plugins/commando
├─3143213 /usr/libexec/c-lightning/plugins/funder
├─3143214 /usr/libexec/c-lightning/plugins/topology
├─3143215 /usr/libexec/c-lightning/plugins/keysend
├─3143216 /usr/libexec/c-lightning/plugins/offers
├─3143217 /usr/libexec/c-lightning/plugins/pay
├─3143218 /usr/libexec/c-lightning/plugins/txprepare
├─3143219 /usr/libexec/c-lightning/plugins/spenderp
├─3143221 /usr/libexec/c-lightning/plugins/bookkeeper
├─3143223 node /home/go/c-lightning-REST/plugin.js
├─3143224 python3 /mnt/md0/C-Lightning/plugins.bak/rebalance/rebalance.py
├─3143239 /usr/libexec/c-lightning/lightning_hsmd
├─3143241 /usr/libexec/c-lightning/lightning_connectd
├─3143255 /usr/libexec/c-lightning/lightning_gossipd
├─3835985 /usr/libexec/c-lightning/lightning_channeld
├─3836004 /usr/libexec/c-lightning/lightning_channeld
├─3836010 /usr/libexec/c-lightning/lightning_channeld
├─3836216 /usr/libexec/c-lightning/lightning_channeld
├─3836242 /usr/libexec/c-lightning/lightning_channeld
├─3836249 /usr/libexec/c-lightning/lightning_channeld
├─3836402 /usr/libexec/c-lightning/lightning_channeld
├─3836421 /usr/libexec/c-lightning/lightning_channeld
├─3837020 /usr/libexec/c-lightning/lightning_channeld
└─3969241 /usr/libexec/c-lightning/lightning_channeld
It doesn't if I use the "directory method" like I used before the v0.12.0...
Well after managing to start the version 0.12.0 none of the plugin I used to use is working anymore...
For example:
lightning-cli rebalance -k outgoing_scid=123456x906x0 incoming_scid=123457x1464x1
{
"code": -32600,
"message": "Error while processing rebalance: 'Plugin' object has no attribute 'getroute'",
"traceback": "Traceback (most recent call last):\n File \"/home/go/.local/lib/python3.8/site-packages/pyln/client/plugin.py\", line 639, in _dispatch_request\n result = self._exec_func(method.func, request)\n File \"/home/go/.local/lib/python3.8/site-packages/pyln/client/plugin.py\", line 619, in _exec_func\n ret = func(*ba.args, **ba.kwargs)\n File \"/mnt/md0/C-Lightning/plugins.bak/rebalance/rebalance.py\", line 212, in rebalance\n getroute = plugin.getroute\nAttributeError: 'Plugin' object has no attribute 'getroute'\n"
}
or
lightning-cli fill 123456x1464x1 50
{
"code": -32600,
"message": "Error while processing fill: unsupported operand type(s) for +: 'int' and 'str'",
"traceback": "Traceback (most recent call last):\n File \"/home/go/.local/lib/python3.8/site-packages/pyln/client/plugin.py\", line 639, in _dispatch_request\n result = self._exec_func(method.func, request)\n File \"/home/go/.local/lib/python3.8/site-packages/pyln/client/plugin.py\", line 616, in _exec_func\n ret = func(*ba.args, **ba.kwargs)\n File \"/mnt/md0/C-Lightning/plugins.bak/drain/drain.py\", line 458, in fill\n payload = read_params('fill', scid, percentage, chunks, retry_for, maxfeepercent, exemptfee)\n File \"/mnt/md0/C-Lightning/plugins.bak/drain/drain.py\", line 350, in read_params\n payload['my_id'] = plugin.rpc.getinfo().get('id')\n File \"/home/go/.local/lib/python3.8/site-packages/pyln/client/lightning.py\", line 806, in getinfo\n return self.call(\"getinfo\")\n File \"/home/go/.local/lib/python3.8/site-packages/pyln/client/lightning.py\", line 354, in call\n this_id = self.get_json_id(method, cmdprefix)\n File \"/home/go/.local/lib/python3.8/site-packages/pyln/client/lightning.py\", line 338, in get_json_id\n this_id = cmdprefix + '/' + this_id\nTypeError: unsupported operand type(s) for +: 'int' and 'str'\n"
}
Log for rebalance.py failure:
2022-09-28T13:28:00.450Z INFO plugin-rebalance.py: Traceback (most recent call last):
2022-09-28T13:28:00.450Z INFO plugin-rebalance.py: File \"/home/go/.local/lib/python3.8/site-packages/pyln/client/plugin.py\", line 639, in _dispatch_request
2022-09-28T13:28:00.450Z INFO plugin-rebalance.py: result = self._exec_func(method.func, request)
2022-09-28T13:28:00.450Z INFO plugin-rebalance.py: File \"/home/go/.local/lib/python3.8/site-packages/pyln/client/plugin.py\", line 619, in _exec_func
2022-09-28T13:28:00.450Z INFO plugin-rebalance.py: ret = func(*ba.args, **ba.kwargs)
2022-09-28T13:28:00.450Z INFO plugin-rebalance.py: File \"/home/go/.local/lib/python3.8/site-packages/pyln/client/plugin.py\", line 961, in _init
2022-09-28T13:28:00.450Z INFO plugin-rebalance.py: return self._exec_func(self.child_init, request)
2022-09-28T13:28:00.450Z INFO plugin-rebalance.py: File \"/home/go/.local/lib/python3.8/site-packages/pyln/client/plugin.py\", line 619, in _exec_func
2022-09-28T13:28:00.450Z INFO plugin-rebalance.py: ret = func(*ba.args, **ba.kwargs)
2022-09-28T13:28:00.450Z INFO plugin-rebalance.py: File \"/mnt/md0/C-Lightning/plugins.bak/rebalance/rebalance.py\", line 792, in init
2022-09-28T13:28:00.450Z INFO plugin-rebalance.py: config = plugin.rpc.listconfigs()
2022-09-28T13:28:00.450Z INFO plugin-rebalance.py: File \"/home/go/.local/lib/python3.8/site-packages/pyln/client/lightning.py\", line 911, in listconfigs
2022-09-28T13:28:00.450Z INFO plugin-rebalance.py: return self.call(\"listconfigs\", payload)
2022-09-28T13:28:00.450Z INFO plugin-rebalance.py: File \"/home/go/.local/lib/python3.8/site-packages/pyln/client/lightning.py\", line 354, in call
2022-09-28T13:28:00.450Z INFO plugin-rebalance.py: this_id = self.get_json_id(method, cmdprefix)
2022-09-28T13:28:00.450Z INFO plugin-rebalance.py: File \"/home/go/.local/lib/python3.8/site-packages/pyln/client/lightning.py\", line 338, in get_json_id
2022-09-28T13:28:00.450Z INFO plugin-rebalance.py: this_id = cmdprefix + '/' + this_id
2022-09-28T13:28:00.451Z INFO plugin-rebalance.py: TypeError: unsupported operand type(s) for +: 'int' and 'str'
2022-09-28T13:28:00.451Z INFO plugin-rebalance.py:
2
Any idea? I need to rebalance my channels... :(
Did you try to update your plugins? I mean to download the latest versions of the plugins?
I mean to download the latest versions of the plugins?
First thing I checked, cause it happened to me in the past that I was using an outdated plugin.
They are up to date.