Can't call methods if socket file is a symbolic link

[wanxewoj]

I'm using the p5 release binary on a armhf system.

I have the following line in my lightning.conf:

sparko-keys=masterkey:+getinfo

However curl gives my a 405 Method Not Allowed response:

$ curl -ki http://127.0.0.1:19737/rpc/ -d '{"method": "getinfo"}' -H 'X-Access masterkey'
HTTP/1.1 405 Method Not Allowed
Content-Length: 0

No errors in the lightning debug.log file. The only sparko related line is:

DEBUG plugin-manager started(4530) /mnt/data/test_lightningd/.lightning/plugins/sparko_linux_arm

That's the only line. No other output from the sparko plugin.

[wanxewoj]

Probably related: Spark wallet app doesn't work either. In the developer tools of the browser I can see the rpc requests fail with 500 Internal Server Error:

Request URL:http://localhost:19737/rpc
Request method:POST
Remote address:127.0.0.1:19737
Status code: 500
Version:HTTP/1.1

Response headers:
-------------------------
Referrer Policy:no-referrer-when-downgrade
HTTP/1.1 500 Internal Server Error
Set-Cookie: user=MTU3MDEyNjI3NXxCd3dBQkhWelpYST18ayBMYx7BxtJ-AvskPGHSjnVvwbvpOKLjNWDviK2iQCY=; Max-Age=2592000; HttpOnly; Secure; SameSite=Strict
Vary: Accept-Encoding
Content-Length: 0

Other requests like index.html, app.js, style.css, etc. work fine.

[fiatjaf]

Try -H 'X-Access: masterkey' with the :.

[wanxewoj]

$ curl -ki http://127.0.0.1:19737/rpc/ -d '{"method": "getinfo"}' -H 'X-Access: masterkey'
HTTP/1.1 405 Method Not Allowed
Content-Length: 0

It's the same error in Postman.

[fiatjaf]

Wait, please try now /rpc without the ending slash.

[wanxewoj]

$ curl -ki http://127.0.0.1:19737/rpc -d '{"method": "getinfo"}' -H 'X-Access: masterkey'
HTTP/1.1 500 Internal Server Error
Vary: Accept-Encoding
Content-Length: 0

[fiatjaf]

Ok, did a test with config:

sparko-host=127.0.0.1
sparko-port=19737
sparko-keys=masterkey:+getinfo

fiatjaf@etleneum ~> curl -ki http://127.0.0.1:19737/rpc -d '{"method": "getinfo"}' -H 'X-Access: masterkey'
HTTP/1.1 200 OK
Content-Type: application/json
Vary: Accept-Encoding
Date: Fri, 04 Oct 2019 12:18:14 GMT
Content-Length: 583

{"id":"02bed1812d3824f7cc4ccd38da5d66a29fcfec146fe95e26cd2e0d3f930d653a8d","alias":"etleneum.com","color":"c2f1bb","num_peers":10,"num_pending_channels":0,"num_active_channels":10,"num_inactive_channels":0,"address":[{"type":"ipv4","address":"172.81.178.38","port":9736}],"binding":[{"type":"ipv4","address":"0.0.0.0","port":9736},{"type":"ipv6","address":"::","port":9735}],"version":"v0.7.2.1-291-g14b55d0","blockheight":597802,"network":"bitcoin","msatoshi_fees_collected":844,"fees_collected_msat":"844msat","warning_lightningd_sync":"Still loading latest blocks from bitcoind."}

I've used the binary from the p5 release, but I used the linux/amd64 binary as I don't have an arm system to test.

Now I don't know how to proceed in debugging this. Maybe you could share all your sparko-relevant config entries here.

Or perhaps if you are able to build go code you could insert a bunch of log.Print() statements at the plugin code and compile yourself.

[fiatjaf]

Actually, I've just done the above myself and here's the binary with tons of ugly debugging statements. Hopefully that will give us more information: https://file.io/ujCEfA

sha256 of the file: 164031ce22f83d172f2458d4bac4290f4dcbb1077acf2914686235fd6f54bcf6

[wanxewoj]

I downloaded your file and put it in .lightning/plugins/sparko_linux_arm_debug, but it's the same with the p5 file. Not a single debug output in debug.log except for:

DEBUG plugin-manager started(1269) /mnt/data/test_lightningd/.lightning/plugins/sparko_linux_arm_debug

And testing the rpc with curl gives again:

$ curl -ki http://127.0.0.1:19737/rpc -d '{"method": "getinfo"}' -H 'X-Access: masterkey'
HTTP/1.1 500 Internal Server Error
Vary: Accept-Encoding
Content-Length: 0

[wanxewoj]

My c-lightning version is: v0.7.2.1-259-g6a8d18c

[wanxewoj]

Ah sorry... the output from your plugin doesn't appear in .lightning/debug.log but in journalctl instead:

got request at /rpc
not using the login/password access key
called /rpc
X-Access is 'masterkey'
permissions:
key 'masterkey' has {map[getinfo:true] map[]}
checking PermissionSet {map[getinfo:true] map[]}
calling method getinfo
ok
error calling node unable to dial socket /mnt/data/test_lightningd/.lightning/run/test_lightningd/lightning-rpc:dial unix /mnt/data/test_lightningd/.lightning/run/test_lightningd/lightning-rpc: connect: no such file or directory
rpc client info: &{<nil> 0 /mnt/data/test_lightningd/.lightning/run/test_lightningd/lightning-rpc   false}

So the issue is with the socket path which is a symbolic link on my system:

$ ls -la .lightning/lightning-rpc
lrwxrwxrwx 1 test_lightningd bitcoin 34 Sep  13 11:41 .lightning/lightning-rpc -> /run/test_lightningd/lightning-rpc

Seems like the plugin doesn't resolve the symbolic link correctly.

[wanxewoj]

Here's how c-lightning daemon is started:

/usr/local/bin/lightningd --daemon --lightning-dir=/mnt/data/test_lightningd/.lightning --pid-file=/run/test_lightningd/lightningd.pid --rpc-file=/run/test_lightningd/lightning-rpc

[wanxewoj]

Confirmed. If I change my setup so that .lightning/lightning-rpc is not a symbolic link everything works:

$ curl -ki http://127.0.0.1:19737/rpc -d '{"method": "getinfo"}' -H 'X-Access: masterkey'
HTTP/1.1 200 OK
Content-Type: application/json
Vary: Accept-Encoding
Content-Length: 602

[fiatjaf]

I don't know what I'm supposed to do here. Should I follow the link before opening the socket? That thing is in a mounted directory from a remote machine?

[wanxewoj]

That thing is in a mounted directory from a remote machine?

No, it's just a symbolic link to a different location on the same local file system.

Should I follow the link before opening the socket?

No, I'm pretty sure this would be handled on the OS level. I think the issue is, that the link is "wrongly" expanded in GO. The path should just be: /mnt/data/test_lightningd/.lightning/lightning-rpc and not /mnt/data/test_lightningd/.lightning/run/test_lightningd/lightning-rpc. There is no /run directory beneath the .lightning dir. So for some reason the symbolic link is resolved and treated (concatenated) as a sub-directory in the implementation.