lightningd
sauron: Add mutinynet support
Adds mutinynet support to sauron. Mutinynet.com uses the mempool.space REST API, which has a slightly different spec from blockstream's Esplora so there is new code to handle that.
We pinned pyln-client to <=24.5 because version 24.8 has some extra requirements that sauron does not meet, we should probably fix this but for now pinning 24.5 should work fine.
You need to pin the version of pyln-client in requirements.txt as
pyln-client==24.5 as otherwise a newer version gets installed which causes this error when starting the plugin:
lightningd: --sauron-api-endpoint=https://mutinynet.com/api/: unknown option
You might want to add some instructions in the README for how to start sauron with Mutinynet. Below command worked for me once I had pyln-client==24.5 installed (see comment above):
lightningd --signet --disable-plugin bcli --plugin $PWD/sauron.py --sauron-api-endpoint https://mutinynet.com/api/
Btw @sip-21 noticed that this PR breaks mainnet on blockstream.info. Can you look into this @ca-ruz @iorch ?
@sip-21 Can you please give me more details about the tests you wrote and what were you trying to do when Sauron failed? I am trying to replicate it to fix it. Thanks!
@ca-ruz The failing GitHub actions/tests are here: https://github.com/sip-21/plugins/actions/runs/11116007452/job/30885476517?pr=15
[gw0] [ 20%] PASSED test_sauron.py::test_rpc_getchaininfo
[gw1] [ 40%] ERROR test_sauron.py::test_rpc_sendrawtransaction_invalid
[gw2] [ 60%] ERROR test_sauron.py::test_rpc_getrawblockbyheight
[gw3] [ 80%] ERROR test_sauron.py::test_rpc_getutxout
[gw1] [ 80%] ERROR test_sauron.py::test_rpc_sendrawtransaction_invalid
[gw2] [ 80%] ERROR test_sauron.py::test_rpc_getrawblockbyheight
[gw4] [100%] ERROR test_sauron.py::test_rpc_estimatefees
[gw3] [100%] ERROR test_sauron.py::test_rpc_getutxout
[gw4] [100%] ERROR test_sauron.py::test_rpc_estimatefees
The tests run fine together locally, but in GitHub action lightningd doesn't seem to not shutdown properly after a test. This is why in the logs you then see
lightningd-0 2024-10-01T00:02:01.508Z **BROKEN** connectd: Failed to listen on socket 127.0.0.1:9735: Address already in use
Time-out: can't find [re.compile('Server started with public key')] in logs
lightningd seems to randomly be able to shutdown properly, though, and two subsequent tests pass as in this case: https://github.com/sip-21/plugins/actions/runs/11089120680/job/30809842202
@sip-21 Thanks!! I recreated the bug and now it's fixed! Can you please run the tests again and let me know?
Waiting on @sip-21's PR here https://github.com/ca-ruz/plugins/pull/2#issuecomment-2405818832
@sip-21
After adding back concurrency, the tests are failing because they're trying to listen on port 9735... that actually doesn't seem necessary. Can you fix the tests not to require listening on port 9735? (or maybe just listen on a random port)
Please don't override the fixtures and the node class unless you absolutely need to. options passed to NodeFactory.gey_node() is where you can disable one and enable the other. Furthermore, the tests differ only in the backend used, so you could use pytest's parametrized Tests to reuse the implementation but get the backend as a parameter (they'd run once for each backend then). Finally, you are doing format checking which is great, but we could go one further and actually use the JSON schemas from CLN to ensure all fields are present and types match up.
@cdecker do you have an example of any pytests that use the JSON schema from CLN to check the fields and types? That sounds amazing!
It's done automatically by instrumenting the pyln-client library to load and check the schemas. Anything going through pyln-client in a test environment should be checked afaik. I don't think notifications and hooks are checked yet (there are no schemas yet), but that'd be an amazing extension covering the docs vs implementation too
@ca-ruz some tests still failing, looks like blockstream is ratelimiting us
@chrisguida @sip-21 Skipped the sendrawtransaction_invalid test across all backends since we were intentionally sending an invalid transaction to test the API call and response. Esplora had been rate-limiting us, likely due to the high frequency of invalid requests. We know the call works because we get an expected response from the API.
@ca-ruz just make an issue to refactor these tests to repeat ourselves less, and clean up the commits, then let's merge