RTL crash when parsing JSON-RPC response

[BTCBellyButton]

I was trying to use the Rebalance plugin but after restarting Lightningd, RTL stops working. Below there is the Systemctl status output. Before adding the Rebalance plugin this line for RTL was present: ├─35732 node /home/go/c-lightning-REST/plugin.js So it seems that there is a conflict between the above and this one: ├─35208 python3 /home/go/.lightning/plugins/rebalance/rebalance.py

● lightningd.service - C-Lightning daemon
     Loaded: loaded (/etc/systemd/system/lightningd.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2020-08-18 23:02:04 CDT; 39s ago
    Process: 35200 ExecStart=/usr/local/bin/lightningd --daemon --pid-file=/run/lightningd/lightningd.pid (code=exited, status=0/SUCCESS)
   Main PID: 35201 (lightningd)
      Tasks: 16 (limit: 14187)
     Memory: 274.1M
     CGroup: /system.slice/lightningd.service
             ├─35201 /usr/local/bin/lightningd --daemon --pid-file=/run/lightningd/lightningd.pid
             ├─35202 /usr/local/bin/../libexec/c-lightning/plugins/autoclean
             ├─35203 /usr/local/bin/../libexec/c-lightning/plugins/bcli
             ├─35204 /usr/local/bin/../libexec/c-lightning/plugins/fundchannel
             ├─35205 /usr/local/bin/../libexec/c-lightning/plugins/keysend
             ├─35206 /usr/local/bin/../libexec/c-lightning/plugins/pay
             ├─35208 python3 /home/go/.lightning/plugins/rebalance/rebalance.py
             ├─35223 /usr/local/libexec/c-lightning/lightning_hsmd
             ├─35224 /usr/local/libexec/c-lightning/lightning_connectd
             ├─35236 /usr/local/libexec/c-lightning/lightning_gossipd
             ├─35255 /usr/local/libexec/c-lightning/lightning_channeld
             ├─35256 /usr/local/libexec/c-lightning/lightning_channeld
             ├─35258 /usr/local/libexec/c-lightning/lightning_channeld
             ├─35259 /usr/local/libexec/c-lightning/lightning_channeld
             └─35263 /usr/local/libexec/c-lightning/lightning_channeld

Aug 18 23:02:10 BTCPayServer lightningd[35207]: **Error: Unexpected "T" at position 10 in state STOP**
Aug 18 23:02:10 BTCPayServer lightningd[35207]:     at Parser.proto.charError (/home/go/c-lightning-REST/node_modules/jsonparse/jsonparse.js:90:16)
Aug 18 23:02:10 BTCPayServer lightningd[35207]:     at Parser.proto.write (/home/go/c-lightning-REST/node_modules/jsonparse/jsonparse.js:267:27)
Aug 18 23:02:10 BTCPayServer lightningd[35207]:     at LightningClient._handledata (/home/go/c-lightning-REST/lightning-client-js.js:143:29)
Aug 18 23:02:10 BTCPayServer lightningd[35207]:     at Socket.LightningClient.client.on.data (/home/go/c-lightning-REST/lightning-client-js.js:77:46)
Aug 18 23:02:10 BTCPayServer lightningd[35207]:     at Socket.emit (events.js:198:13)
Aug 18 23:02:10 BTCPayServer lightningd[35207]:     at addChunk (_stream_readable.js:288:12)
Aug 18 23:02:10 BTCPayServer lightningd[35207]:     at readableAddChunk (_stream_readable.js:269:11)
Aug 18 23:02:10 BTCPayServer lightningd[35207]:     at Socket.Readable.push (_stream_readable.js:224:10)
Aug 18 23:02:10 BTCPayServer lightningd[35207]:     at Pipe.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)

[BTCBellyButton]

Opened issue at RTL as well: https://github.com/Ride-The-Lightning/RTL/issues/439

[cdecker]

It seems unlikely that this is a conflict, rather the JSON parser in c-lightning-rest seems to be crashing while attempting to parse something. Sadly I'm not very experienced with JS code, so I can't really help with this one, but it is unlikely to be at all related to the rebalance plugin (it's not doing anything on its own, just waiting for lightning-cli rebalance calls).

[BTCBellyButton]

Yeah, I figured it could be the RTL use of c-lightning-REST the source of the problem. That's why I opened an issue with them afterward. They marked it as a bug to be fixed in a future release. Thanks for taking the time to look into this!

[darosior]

I think it should be directed to the C-lightning-REST repo as it is neither related to rebalance nor RTL.

The (common) issue comes from the jsonparse library used by C-lightning-REST, to which the data is written without checking its state (which is apparently stopping at this moment. Weird.

As a quick-fix you could use RTL as a plugin instead, which uses its own brewed JSON parser and thus would not run into this specific error.

[BTCBellyButton]

Thanks. It's a minor inconvenience. I can just rename the Plugins directory when I need to rebalance, ignore RTL and then rename it *.bak when I'm done. Not sure how to use RTL as a plugin though.

[darosior]

Wait, this happens each time you try to rebalance ?

Not sure how to use RTL as a plugin though.

https://github.com/Ride-The-Lightning/c-lightning-REST#option-2-run-as-c-lightning-plugin (not sure if this can be done directly from RTL though)

[BTCBellyButton]

When I load the plugin, the plugin works fine but RTL stops working. So once I'm done rebalancing I rename the plugins directory to plugins.bak and restart lightnind so it doesn't load rebalance.py and I can use RTL until the next rebalancing. I can't have both at the same time but I am not rebalancing so frequently that it is unbearable.

I'll try that one as soon as I can! Thanks!!

On Fri, Aug 28, 2020 at 1:05 PM Darosior [email protected] wrote:

Wait, this happens each time you try to rebalance ?

Not sure how to use RTL as a plugin though.

https://github.com/Ride-The-Lightning/c-lightning-REST#option-2-run-as-c-lightning-plugin (not sure if this can be done directly from RTL though)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/lightningd/plugins/issues/134#issuecomment-683010608, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQLFTROB75MYLSTFYB45UFTSC7WWHANCNFSM4QEQGLZA .

[darosior]

Ah! That's more inconvenient than i thought it was. Pinging @saubyk .

[BTCBellyButton]

RTL will fix it in the next release so I won't touch anything in order to be able to let them know if the fix actually worked on my installation. Hopefully it won't be too long... :)

I'll keep you posted.

[saubyk]

Wait, this happens each time you try to rebalance ?

Not sure how to use RTL as a plugin though.

https://github.com/Ride-The-Lightning/c-lightning-REST#option-2-run-as-c-lightning-plugin (not sure if this can be done directly from RTL though)

Hi @BTCBellyButton are you running C-Lightning-REST as a plugin or a standalone API server?

If you are running it as a separate API server, can you try running it as a C-Lightning plugin and see if you run into the same problem?

I believe the option which @darosior suggested is to run c-lightning-rest as the plugin, not RTL.

Thanks.

[BTCBellyButton]

Hi @saubyk. It is running as a plugin. I just double checked.

------------------------------
**~/.lightning/config**
alias=BitcoinBellyButton
bind-addr=127.0.0.1:9735
proxy=127.0.0.1:9050
announce-addr=xxxxxx.onion:9735

plugin=/home/go/c-lightning-REST/plugin.js
rest-port=3001
rest-docport=4001
rest-protocol=https
--------------------------------

BUT, I also tried to add the rebalance.py using the config file:

alias=BitcoinBellyButton
bind-addr=127.0.0.1:9735
proxy=127.0.0.1:9050
announce-addr=xxxxxxx.onion:9735

plugin=/home/go/c-lightning-REST/plugin.js
rest-port=3001
rest-docport=4001
rest-protocol=https
plugin=/home/go/.lightning/plugins.bak/rebalance/rebalance.py

instead of just putting it in the plugins directory as I did when I had the issue. (I followed this guide: https://github.com/lightningd/plugins/blob/master/README.md#automatic-plugin-initialization)

This way (plugin in config) it seems to be working:

● lightningd.service - C-Lightning daemon
     Loaded: loaded (/etc/systemd/system/lightningd.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2020-08-28 23:48:26 CDT; 8min ago
    Process: 222540 ExecStart=/usr/local/bin/lightningd --daemon --pid-file=/run/lightningd/lightningd.pid (code=exited, status=0/SUCCESS)
   Main PID: 222541 (lightningd)
      Tasks: 33 (limit: 14187)
     Memory: 329.1M
     CGroup: /system.slice/lightningd.service
             ├─222541 /usr/local/bin/lightningd --daemon --pid-file=/run/lightningd/lightningd.pid
             ├─222542 /usr/local/bin/../libexec/c-lightning/plugins/autoclean
             ├─222543 /usr/local/bin/../libexec/c-lightning/plugins/bcli
             ├─222544 /usr/local/bin/../libexec/c-lightning/plugins/fundchannel
             ├─222545 /usr/local/bin/../libexec/c-lightning/plugins/keysend
             ├─222546 /usr/local/bin/../libexec/c-lightning/plugins/pay
             ├─222547 node /home/go/c-lightning-REST/plugin.js
             ├─222548 python3 /home/go/.lightning/plugins.bak/rebalance/rebalance.py
             ├─222563 /usr/local/libexec/c-lightning/lightning_hsmd
             ├─222564 /usr/local/libexec/c-lightning/lightning_connectd

So the issue seems to be related to the use of the "Automatic plugin initialization"

[BTCBellyButton]

NOPE. Same problem. Using the config file does not solve the issue.

As soon as you reload the page the bug is back.

[cdecker]

Thanks for taking the time to look into this!

Anytime, and sorry for the delay, hoping this gets resolved soon :+1: