Could not run as a plugin, but works perfectly as a standalone service

[sethforprivacy]

Hi there, just finished building out a Core LN + c-lightning-REST + RTL setup, and ran into this error when trying to use c-lightning-REST as a plugin for lightningd:

plugin-plugin.js: Killing plugin: exited before replying to getmanifest

The plugin works perfectly when run as it's own service (systemd script here, config file here), but no matter what I tried lightningd would crash instantly on trying to load the plugin.

Here are the commands I ran before adding it to the lightningd config file:

    5  wget https://github.com/Ride-The-Lightning/c-lightning-REST/releases/download/v0.7.2/v0.7.2.tar.gz.asc
    6  tar -xvf v0.7.2.tar.gz
    7  cd c-lightning-REST-0.7.2
    8  npm install --only=prod

Then I added via full path as specified here: https://github.com/Ride-The-Lightning/c-lightning-REST#option-2-run-as-c-lightning-plugin

Not sure how to get better logs, but happy to collect what I can for you!

[saubyk]

Hi @sethforprivacy is it possible for you to share your core lightning config settings? It should be in the .lightning folder.

[sethforprivacy]

Hi @sethforprivacy is it possible for you to share your core lightning config settings? It should be in the .lightning folder.

Here it is:

https://paste.sethforprivacy.com/?3e2080ae7cbf5ad0#BMcHpNDXn4rtv8SjQ8sd2HNTxxjMbqkYyLisHGdYgEU

When I was trying c-lightning-rest as a plugin I called it via full-path, i.e.:

important-plugin=/home/lightningd/c-lightning-REST-0.7.2/plugin.js
rest-port=3001
rest-docport=4001
rest-protocol=https

My best guess would be it being related to lightningd not properly recognizing the location of the node binary, but nothing I could find to resolve that worked.

[saubyk]

what all do you have in the lightningd folder? The only difference that I could find in my settings is that I am keeping the c-lightning-REST folder separate from all of core lightning's and plugin folders.

Keep in mind that C-Lightning-REST is not an "official" plugin, so not sure if there is manifest where it needs to be registered. But that's not required for it to run as a plugin.

May be, I need to get it registered as an official plugin (not sure if that's a thing either) 😄

[devastgh]

Adding my 2c, maybe it helps: C-ln tries to load every executable it finds in ~/.lightning/plugins/* . So having plugin-plugin.js: Killing plugin: exited before replying to getmanifest is expected, if you have any executable in that path, that is not a c-ln plugin. I think loading the plugin might fail, cause it's not an executable. Could you check what permissions you have on plugin.js ? If it does not have executable flag on the file, you need to chmod +x plugin.js

[litch]

I just got bit by the same exact issue - where the plugin config in the .lightning/config seems incorrect.

May 21 19:45:22  lightningd[871350]: 2022-05-21T19:45:22.631Z INFO    plugin-plugin.js: Killing plugin: exited before replying to getmanifest
May 21 19:45:22  lightningd[871350]: lightningd: /home/litch/.lightning/config line 21: rest-port=3002: unrecognized option

[saubyk]

I just got bit by the same exact issue - where the plugin config in the .lightning/config seems incorrect.

May 21 19:45:22  lightningd[871350]: 2022-05-21T19:45:22.631Z INFO    plugin-plugin.js: Killing plugin: exited before replying to getmanifest
May 21 19:45:22  lightningd[871350]: lightningd: /home/litch/.lightning/config line 21: rest-port=3002: unrecognized option

Hi @litch can you share your config file settings?

[litch]

The config file sections were exactly as in the readme:

mainnet
bitcoin-rpcuser=<>
bitcoin-rpcpassword=<>
bitcoin-rpcconnect=<>
bitcoin-rpcport=8332

proxy=127.0.0.1:9050
announce-addr=<>:9735

large-channels
experimental-dual-fund
funder-policy=match
funder-policy-mod=100
lease-fee-base-msat=500sat
lease-fee-basis=52
channel-fee-max-base-msat=5sat
channel-fee-max-proportional-thousandths=1


clboss-auto-close=true
clboss-min-onchain=10000000

plugin=/home/<>/c-lightning-REST/plugin.js
rest-port=3002
rest-docport=4001
rest-protocol=https

Similarly to @sethforprivacy - it works swimmingly if I run it otherwise.

[sethforprivacy]

@saubyk - I don't have any plugins in there, just this structure:

tiny in tiny in ~ on  master [!?] took 5s ❯ sudo tree /etc/lightningd
/etc/lightningd
├── lightningd.conf
├── lightningd.log
└── lightning-rpc

0 directories, 3 files
tiny in tiny in ~ on  master [!?] ❯ sudo tree /home/lightningd/.lightning
/home/lightningd/.lightning
└── bitcoin
    ├── backup.lock
    ├── crash.log.20220517195817
    ├── crash.log.20220522165328
    ├── data.clboss
    ├── gossip_store
    ├── hsm_secret
    ├── keys.clboss
    ├── lightningd.sqlite3
    └── lightning-rpc

1 directory, 9 files

I was loading the c-lightning-REST plugin from a directory in /home/lightningd (where it's running from now as a standalone app without issues under the same user as lightningd).

@devastgh - plugin.js is properly permissioned for the user, and has the execute bit set:

-rwxrwxr-x   1 lightningd lightningd   1698 May  9 22:31 plugin.js

Happy to help however I can!

[sethforprivacy]

Still the same issue with v0.8.0:

2022-08-15T12:34:26.952Z DEBUG   lightningd: Opened log file /etc/lightningd/lightningd.log
2022-08-15T12:34:27.319Z INFO    plugin-plugin.js: Killing plugin: exited before replying to getmanifest
2022-08-15T12:34:27.319Z **BROKEN** plugin-plugin.js: Plugin marked as important, shutting down lightningd!

Config tested:

# c-lightning-REST settings
important-plugin=/home/lightningd/c-lightning-REST-0.8.0/plugin.js
rest-port=3001
rest-docport=4001
rest-protocol=http
rest-domain=cln.sethforprivacy.com

Folder structure for cln-REST:

sethforprivacy in apps in ~ on  master [!?] via  v16.16.0 ❯ sudo tree -L 1 /home/lightningd/c-lightning-REST-0.8.0/
/home/lightningd/c-lightning-REST-0.8.0/
├── amd64.Dockerfile
├── app.js
├── arm32v7.Dockerfile
├── arm64v8.Dockerfile
├── certs
├── cl-rest-config.json
├── cl-rest.js
├── controllers
├── docapp.js
├── docker-entrypoint.sh
├── docs
├── LICENSE
├── lightning-client-js.js
├── methods.json
├── node_modules
├── package.json
├── package-lock.json
├── plugin.js
├── README.md
├── routes
├── sample-cl-rest-config.json
├── screenshots
├── test
└── utils

[devastgh]

Could you just check it without any options specified in c-lightning config? If it works, set the options one by one, to check which one doesn't work. I assure you it loads fine as a plugin for me without any options specified.

[sethforprivacy]

Could you just check it without any options specified in c-lightning config? If it works, set the options one by one, to check which one doesn't work. I assure you it loads fine as a plugin for me without any options specified.

Same issue, doesn't help:

# c-lightning-REST settings
important-plugin=/home/lightningd/c-lightning-REST-0.8.0/plugin.js
#rest-port=3001
#rest-docport=4001
#rest-protocol=http
#rest-domain=cln.sethforprivacy.com

sethforprivacy in apps in ~ on  master [!?] via  v16.16.0 ❯ tail -f /etc/lightningd/lightningd.log
2022-08-15T12:56:55.282Z DEBUG   lightningd: Opened log file /etc/lightningd/lightningd.log
2022-08-15T12:56:55.427Z INFO    plugin-plugin.js: Killing plugin: exited before replying to getmanifest
2022-08-15T12:56:55.427Z **BROKEN** plugin-plugin.js: Plugin marked as important, shutting down lightningd!




2022-08-15T12:56:55.747Z DEBUG   lightningd: Opened log file /etc/lightningd/lightningd.log
2022-08-15T12:56:55.893Z INFO    plugin-plugin.js: Killing plugin: exited before replying to getmanifest
2022-08-15T12:56:55.893Z **BROKEN** plugin-plugin.js: Plugin marked as important, shutting down lightningd!

[devastgh]

Maybe try it with just plugin= not important. Also, try to load it while the file cl-rest-config.json is not present in c-lightning-REST directory. If it still fails to load, then i'm out of ideas.

[jochemin]

Same error on fresh install, it works when run as it's own service but fails trying to run it as plugin.

[ShahanaFarooqui]

Could you run the plugin after adding rest-execmode=test in the config. It might provide us better insights on the failure reason.

I tried all above mentioned scenarios on my local Ubuntu 22 and they all are working successfully. It seems like some local setting issue.

[sethforprivacy]

I'll test again ASAP.

[sethforprivacy]

@ShahanaFarooqui:

Nov 04 16:15:22 lightningd[2517231]: # Fatal error in , line 0
Nov 04 16:15:22 lightningd[2517231]: # Check failed: reservation_.SetPermissions(protect_start, protect_size, permission).

[sethforprivacy]

Still fails to start the plugin, above seems the error returned with execmode=test set.

[sethforprivacy]

The only thing I can think of that may be causing this is the recommended hardening in the lightningd systemd script:

# Hardening measures
####################

# Provide a private /tmp and /var/tmp.
PrivateTmp=true

# Disallow the process and all of its children to gain
# new privileges through execve().
NoNewPrivileges=true

# Use a new /dev namespace only populated with API pseudo devices
# such as /dev/null, /dev/zero and /dev/random.
PrivateDevices=true

# Deny the creation of writable and executable memory mappings.
MemoryDenyWriteExecute=true

Testing without it now.

[sethforprivacy]

Disabling those protections resolved the issue, will nail down which one is to blame shortly.

[sethforprivacy]

Narrowed down to this:

# Deny the creation of writable and executable memory mappings.
MemoryDenyWriteExecute=true

If the user has this hardening measure enabled in their systemd script, c-lightning-REST cannot function as a plugin.

Disabling the hardening has everything working perfectly!

Ideally there is some way you can have both this hardening and the plugin, but that's far above my pay grade.

[ShahanaFarooqui]

@sethforprivacy Thank you for the quick testing and solution.

@jochemin & @litch Could you also verify your permissions or add rest-execmode=test to get some failure insights. I am keeping this issue open till then.

[litch]

I was also running the "hardening" systemd script, and that makes perfect sense as to the cause.

I echo the, "I would prefer to run this and have the thing hardened" but likewise don't have enough insight into the changes required to have a strong opinion about whether that would be better than, perhaps a log message that detects that flag and tells the user that that is the cause of the error.

[jochemin]

No luck for me. This is my systemd service:

`# /etc/systemd/system/lightningd.service

[Unit] Description=Core Lightning daemon Requires=bitcoind.service After=bitcoind.service Wants=network.target After=network.target

[Service] ExecStart=/home/jochemin/clightning/usr/bin/lightningd --daemon ExecStop=/bin/sh -c '/home/jochemin/clightning/usr/bin/lightning-cli stop'

`# Process management #################### Type=forking Restart=on-failure TimeoutSec=300 RestartSec=30

`# Directory creation and permissions #################################### User=jochemin Group=adm Environment=PATH=/home/jochemin/clightning/usr/bin:/home/jochemin/bitcoin/bin

# Hardening measures #################### # Provide a private /tmp and /var/tmp. PrivateTmp=true

`# Mount /usr, /boot/ and /etc read-only for the process. ProtectSystem=full

# Disallow the process and all of its children to gain # new privileges through execve(). NoNewPrivileges=true

# Use a new /dev namespace only populated with API pseudo devices # such as /dev/null, /dev/zero and /dev/random. PrivateDevices=true

# Deny the creation of writable and executable memory mappings. #MemoryDenyWriteExecute=true

[Install] WantedBy=multi-user.target`

I've commented MemoryDenyWriteExecute but same error:

2022-11-04T18:21:08.430Z INFO plugin-plugin.js: Killing plugin: exited before replying to getmanifest

[sethforprivacy]

Try also commenting out this line, I realized I had it commented out already for other issues:

ProtectSystem=full

[ShahanaFarooqui]

@jochemin Also add rest-execmode=test in your lightning config file to see the failure log from this plugin.

[jochemin]

nov 04 19:32:02 jbtc lightningd[537230]: /usr/bin/env: 'node': No such file or directory nov 04 19:32:02 jbtc lightningd[537213]: lightningd: /home/jochemin/.lightning/config line 16: rest-execmode=test: unrecognized option

It seems a problem with node executable.

I've checked and I have node executable in /user/bin

Out of ideas

[jochemin]

Try also commenting out this line, I realized I had it commented out already for other issues:

ProtectSystem=full

Nope, same error.

[ShahanaFarooqui]

@jochemin It either looks like an incorrect naming issue which can be resolved via symlink OR nodejs-legacy installation might work.

Below are some links for your reference related to these solutions: https://github.com/nodejs/node-v0.x-archive/issues/3911#issuecomment-8956154 https://github.com/nodejs/node-v0.x-archive/issues/3911#issuecomment-18951288 https://askubuntu.com/questions/1044692/usr-bin-env-node-no-such-file-or-directory-even-though-it-is-in-path https://stackoverflow.com/questions/30281057/node-forever-usr-bin-env-node-no-such-file-or-directory

[jochemin]

/usr/bin/env: 'node': No such file or directory

Thanks @ShahanaFarooqui , no luck for me. My executable name is node so no naming isuue, I installed nodejs from Ubuntu repositories with sudo apt-install nodejs.

I've tried to install last LTS version but the error continues.

Mi SO is Ubuntu Server 22.04 and node version is 18.12.1

I'll keep trying.

[ShahanaFarooqui]

@jochemin Thank you for the update. Please post your solution, if you find any, as it might help other users.

I am closing this issue now as there is no action required from us anymore.