lightning
lightning copied to clipboard
ElementsProject
Core Lightning is incompatible with the Lightning Network - lacks support for payments
I am not sure if it is just me or others also, but I have noticed that ever since updating to version 24 most payments fail.
I get alot of these messages, the htlc and node number varies though.
htlc 16 failed from 0th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)
then payment deadline expires.
I have even attempted to pay various invoices even with small payments, I also have some large channels (millions of sats) open with large and well connected nodes.
Posting this for visibility in case if others are experiencing this also. If there is something I can check to confirm if it is an issue with my node or corelightning I can do so.
To clarify, it is uncertain if this is a problem wiht my node or something to do with core lightning.
What happens if you put disable-mmp inside your config file and restart your node?
@vincenzopalazzo I was away from computer but I can try that now. Before I do that I want to know what that setting does. I am not finding any documentation on it. What is that setting?
disable-mmp is not a recgonized command or config setting by my Core Lightning.
@nakoshi-satamoto is
disable-mppsorry
Did this change anything for you? @nakoshi-satamoto
@kilrau I actually did not need to.
Ever since I upgraded to core lightning verison 24.02 I kept getting alot of these errors in my logs for various channels. This also was an issue on V24.02.1
2024-04-15T04:27:36.026Z UNUSUAL lightningd: Bad gossip order: could not find channel 696011x284x1 for peer's channel update
After updating to the latest version of core lightning onto version 43.02.2 those gossip issues went away, transactions inbound and outbound are not failing anymore. Maybe it was related to gossip not working?
Did 24.02.2 fix this for you? I upgraded to that and I'm still having the same issue. (Pages of Bad gossip order....)
On 2024-05-05 18:52, BitcoinMechanic wrote:
Did 24.02.2 fix this for you? I upgraded to that and I'm still having the same issue. (Pages of
Bad gossip order....)
I am on 24.02.2 and I still get those bad gossip order messages. I am able to do payments though.
also on 24.02.02 with high - payment failure rate and bad gossip order messages
I'm also on 24.02.02 (using the BTCPay Docker image), and 100% payment failures unless paying direct channel partners, and tons of bad gossip order messages. Cannot query remote routes at all, even to targets that are direct partners of channel partners. Incoming payments and some through-routes seem to work though (the through routes may only be working to next-node termination though).
Example log entries:
2024-05-27T18:19:39.925Z INFO plugin-pay: cmd 34 partid 0: Payment fee constraint 2500msat is below exemption threshold, allowing a maximum fee of 5000msat 2024-05-27T18:21:25.798Z INFO plugin-pay: cmd 34 partid 0: Payment fee constraint 2500msat is below exemption threshold, allowing a maximum fee of 5000msat 2024-05-27T18:47:11.456Z INFO plugin-pay: cmd 34 partid 0: Payment fee constraint 2500msat is below exemption threshold, allowing a maximum fee of 5000msat
2024-05-27T18:21:25.803Z INFO plugin-clnrest.py: RPC Error: {'code': 210, 'message': 'Destination 027cd974e47086291bb8a5b0160a889c738f2712a703b8ea939985fd16f3aae67e is not reachable directly and all routehints were unusable.', 'attempts': [{'status': 'failed', 'failreason': 'Destination 027cd974e47086291bb8a5b0160a889c738f2712a703b8ea939985fd16f3aae67e is not reachable directly and all routehints were unusable.', 'partid': 0, 'amount_msat': 500000msat}]} 2024-05-27T18:47:11.484Z INFO plugin-pay: cmd 34 partid 0: Destination 03b428ba4b48b524f1fa929203ddc2f0971c2077c2b89bb5b22fd83ed82ac2f7e1 is not reachable directly and all routehints were unusable. 2024-05-27T18:47:11.486Z INFO plugin-clnrest.py: RPC Error: {'code': 210, 'message': 'Destination 03b428ba4b48b524f1fa929203ddc2f0971c2077c2b89bb5b22fd83ed82ac2f7e1 is not reachable directly and all routehints were unusable.', 'attempts': [{'status': 'failed', 'failreason': 'Destination 03b428ba4b48b524f1fa929203ddc2f0971c2077c2b89bb5b22fd83ed82ac2f7e1 is not reachable directly and all routehints were unusable.', 'partid': 0, 'amount_msat': 500000msat}]}
2024-05-27T18:47:55.749Z UNUSUAL lightningd: Bad gossip order: could not find channel 844051x1140x1 for peer's channel update 2024-05-27T18:47:55.749Z UNUSUAL lightningd: Bad gossip order: could not find channel 786203x525x0 for peer's channel update 2024-05-27T18:49:25.765Z UNUSUAL lightningd: Bad gossip order: could not find channel 746879x1261x2 for peer's channel update 2024-05-27T18:49:25.775Z UNUSUAL lightningd: Bad gossip order: could not find channel 820913x3568x0 for peer's channel update
UPDATE: I added disable-mpp to the startup file, with no effect.
I said that I can still do payments, I am ammending my claim. I can receive inbound payments no problem but I do have a high rate of failure for outbound payments unless it is a direct peer. Ever since I upgraded to version 24 I have noticed this and it also coinciding with the gossip error messages. I delete gossip database and it still keeps getting gossip errors.
DO NOT delete the gossip store, you will not be able to make payments any more. You've made the network disappear, and it can take a while to refill.
Are you missing gossip? A healthy node should look like this:
$ lightning-cli -H listchannels | grep -c ^short_channel_id
85921
$ lightning-cli -H listnodes | grep -c ^nodeid
9062
That is, about 85000 public channels and 9000 nodes.
Well… in an attempt to make my core lightning node functional again I did delete the gossip store (but did save a backup! (This was… a week ago?)) should I just continue as is or cp gossip_store.backup to replace the “new” one? Have you not noticed any payment issues Rusty?
On Mon, Jun 3, 2024, at 9:26 PM, Rusty Russell wrote:
DO NOT delete the gossip store, you will not be able to make payments any more. You've made the network disappear, and it can take a while to refill.
Are you missing gossip? A healthy node should look like this:
$ lightning-cli -H listchannels | grep -c ^short_channel_id 85921 $ lightning-cli -H listnodes | grep -c ^nodeid 9062That is, about 85000 public channels and 9000 nodes.— Reply to this email directly, view it on GitHub https://github.com/ElementsProject/lightning/issues/7180#issuecomment-2146579773, or unsubscribe https://github.com/notifications/unsubscribe-auth/AND3BCSLAHAFJBKAP3S2W4LZFU6XHAVCNFSM6AAAAABFLB774SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBWGU3TSNZXGM. You are receiving this because you commented.Message ID: @.***>
On 2024-06-04 06:26, Rusty Russell wrote:
DO NOT delete the gossip store, you will not be able to make payments any more. You've made the network disappear, and it can take a while to refill.
Are you missing gossip? A healthy node should look like this:
$ lightning-cli -H listchannels | grep -c ^short_channel_id 85921 $ lightning-cli -H listnodes | grep -c ^nodeid 9062That is, about 85000 public channels and 9000 nodes.
What is wrong with deleting gossip? I have done that often times in version 24 to fix gossip issues. It re-populates fast. I deleted gossip yesterday after I upgraded to version 24.05 which claims to fix the gossip issues.
I just checked and I have 92910 channels, 16063 nodes.
On 2024-06-07 02:33, blapcity wrote:
Did this resolve your issue with payments ?
I had the opportunity to test some payments today. I have 12 chans and almost 10,000,000 of outbound liquidity combined. Many of the peers I'm connected to are well connected also. I tried three random payments of less than 100,000 to different destinations and all of them failed. I still see gossip errors sometimes also, even after upgrading to 24.05 and rebuliding gossip. I'd say my gossip database is healthy also as I have 14427 nodes, 92599 chans in my view.
As of now, Core Lightning cannot send payments unless it is a directly connected peer. This is not functional.
On 2024-06-10 19:46, Satoshi wrote:
I had the opportunity to test some payments today. I have 12 chans and almost 10,000,000 of outbound liquidity combined.
10 millions sats.
Any chance we could get some more logs to try and hone in on the cause for the increased failure rate? From the above they all seem to come more or less directly after deleting gossip_store (fails immediately due to lack of network view, rather than taking some time and performing real attempts).
The gossip order is very likely to be unrelated, so ignore that maybe for a bit. It is mostly a warning that the peer has given us gossip messages in the incorrect order (update before announcement, or node before any of it's channels).
One more thing, how many here are running btcpayserver? As it has been mentioned a number of times, I'd like to exclude that as a potential source.
I have similar problems since upgrading to 24.05 without running btcpayserver. I'm only connected to boltz with a channel capacity of 1.000.000 sats. The logs below are from a failed transaction to phoenix wallet. Sending funds to wallet of satoshi was also not possible (same amount).
2024-06-15T21:25:26.486Z INFO plugin-pay: cmd 34 partid 0: Initial limit on max HTLCs: 1305, Destination 027cd974e47086291bb8a5b0160a889c738f2712a703b8ea939985fd16f3aae67e has 87 channels, assuming 15 HTLCs per channel
2024-06-15T21:25:26.486Z INFO lightningd: Sending 35000035msat in onion to deliver 35000000msat
2024-06-15T21:25:29.544Z INFO 02d96eadea3d780104449aca5c93461ce67c1564e2e1d73225fa67dd3b997a6018-chan#28: htlc 144 failed from 0th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)
2024-06-15T21:25:29.557Z INFO plugin-pay: cmd 34 partid 0: failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
2024-06-15T21:25:29.589Z INFO lightningd: Sending 35001105msat in onion to deliver 35000000msat
2024-06-15T21:25:32.612Z INFO 02d96eadea3d780104449aca5c93461ce67c1564e2e1d73225fa67dd3b997a6018-chan#28: htlc 145 failed from 0th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)
2024-06-15T21:25:32.623Z INFO plugin-pay: cmd 34 partid 1: failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
2024-06-15T21:25:32.657Z INFO lightningd: Sending 35001071msat in onion to deliver 35000000msat
2024-06-15T21:25:35.432Z INFO 02d96eadea3d780104449aca5c93461ce67c1564e2e1d73225fa67dd3b997a6018-chan#28: htlc 146 failed from 0th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)
2024-06-15T21:25:35.443Z INFO plugin-pay: cmd 34 partid 2: failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
2024-06-15T21:25:35.481Z INFO lightningd: Sending 35004070msat in onion to deliver 35000000msat
2024-06-15T21:25:38.787Z INFO 02d96eadea3d780104449aca5c93461ce67c1564e2e1d73225fa67dd3b997a6018-chan#28: htlc 147 failed from 0th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)
2024-06-15T21:25:38.798Z INFO plugin-pay: cmd 34 partid 3: failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
2024-06-15T21:25:38.836Z INFO lightningd: Sending 35003455msat in onion to deliver 35000000msat
2024-06-15T21:25:44.903Z INFO 02d96eadea3d780104449aca5c93461ce67c1564e2e1d73225fa67dd3b997a6018-chan#28: htlc 148 failed from 1th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)
2024-06-15T21:25:44.915Z INFO plugin-pay: cmd 34 partid 4: failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
2024-06-15T21:25:44.955Z INFO lightningd: Sending 35003455msat in onion to deliver 35000000msat
2024-06-15T21:25:51.109Z INFO 02d96eadea3d780104449aca5c93461ce67c1564e2e1d73225fa67dd3b997a6018-chan#28: htlc 149 failed from 1th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)
2024-06-15T21:25:51.120Z INFO plugin-pay: cmd 34 partid 5: failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
2024-06-15T21:25:51.170Z INFO lightningd: Sending 35003821msat in onion to deliver 35000000msat
2024-06-15T21:25:54.760Z INFO 02d96eadea3d780104449aca5c93461ce67c1564e2e1d73225fa67dd3b997a6018-chan#28: htlc 150 failed from 0th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)
2024-06-15T21:25:54.772Z INFO plugin-pay: cmd 34 partid 6: failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
2024-06-15T21:25:54.820Z INFO lightningd: Sending 35001040msat in onion to deliver 35000000msat
2024-06-15T21:25:57.924Z INFO 02d96eadea3d780104449aca5c93461ce67c1564e2e1d73225fa67dd3b997a6018-chan#28: htlc 151 failed from 0th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)
2024-06-15T21:25:57.933Z INFO plugin-pay: cmd 34 partid 7: failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
2024-06-15T21:25:57.981Z INFO lightningd: Sending 35001590msat in onion to deliver 35000000msat
2024-06-15T21:26:02.739Z INFO 02d96eadea3d780104449aca5c93461ce67c1564e2e1d73225fa67dd3b997a6018-chan#28: htlc 152 failed from 0th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)
2024-06-15T21:26:02.751Z INFO plugin-pay: cmd 34 partid 8: failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
2024-06-15T21:26:02.797Z INFO lightningd: Sending 35004170msat in onion to deliver 35000000msat
2024-06-15T21:26:05.668Z INFO 02d96eadea3d780104449aca5c93461ce67c1564e2e1d73225fa67dd3b997a6018-chan#28: htlc 153 failed from 0th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)
2024-06-15T21:26:05.679Z INFO plugin-pay: cmd 34 partid 9: failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
2024-06-15T21:26:05.740Z INFO lightningd: Sending 35003455msat in onion to deliver 35000000msat
2024-06-15T21:26:23.011Z INFO 02d96eadea3d780104449aca5c93461ce67c1564e2e1d73225fa67dd3b997a6018-chan#28: htlc 154 failed from 2th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)
2024-06-15T21:26:23.040Z INFO plugin-pay: cmd 34 partid 10: failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
2024-06-15T21:26:23.131Z INFO lightningd: Sending 18167140msat in onion to deliver 18167122msat
2024-06-15T21:26:23.153Z INFO lightningd: Sending 16833926msat in onion to deliver 16832878msat
2024-06-15T21:26:29.574Z INFO 02d96eadea3d780104449aca5c93461ce67c1564e2e1d73225fa67dd3b997a6018-chan#28: htlc 155 failed from 0th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)
2024-06-15T21:26:29.585Z INFO plugin-pay: cmd 34 partid 11: failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
2024-06-15T21:26:29.585Z INFO plugin-pay: cmd 34 partid 11: Payment deadline expired, not retrying (partial-)payment 21229a3a6e8bb8d7001101d258bcf1f23ce2df2fc37406b25b1ad3a065193a4c/11
2024-06-15T21:26:30.010Z INFO 02d96eadea3d780104449aca5c93461ce67c1564e2e1d73225fa67dd3b997a6018-chan#28: htlc 156 failed from 0th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)
2024-06-15T21:26:30.021Z INFO plugin-pay: cmd 34 partid 12: failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
2024-06-15T21:26:30.021Z INFO plugin-pay: cmd 34 partid 12: Payment deadline expired, not retrying (partial-)payment 21229a3a6e8bb8d7001101d258bcf1f23ce2df2fc37406b25b1ad3a065193a4c/12
2024-06-15T21:26:30.022Z UNUSUAL plugin-clrest.js: {\"type\":\"lightning\",\"name\":\"LightningError\",\"message\":\"Ran out of routes to try after 13 attempts: see `paystatus`\",\"code\":210,\"attempts\":[{\"status\":\"failed\",\"failreason\":\"failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)\",\"partid\":0,\"amount_msat\":35000000},{\"status\":\"failed\",\"failreason\":\"failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)\",\"partid\":1,\"amount_msat\":35000000,\"parent_partid\":0},{\"status\":\"failed\",\"failreason\":\"failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)\",\"partid\":2,\"amount_msat\":35000000,\"parent_partid\":1},{\"status\":\"failed\",\"failreason\":\"failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)\",\"partid\":3,\"amount_msat\":35000000,\"parent_partid\":2},{\"status\":\"failed\",\"failreason\":\"failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)\",\"partid\":4,\"amount_msat\":35000000,\"parent_partid\":3},{\"status\":\"failed\",\"failreason\":\"failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)\",\"partid\":5,\"amount_msat\":35000000,\"parent_partid\":4},{\"status\":\"failed\",\"failreason\":\"failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)\",\"partid\":6,\"amount_msat\":35000000,\"parent_partid\":5},{\"status\":\"failed\",\"failreason\":\"failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)\",\"partid\":7,\"amount_msat\":35000000,\"parent_partid\":6},{\"status\":\"failed\",\"failreason\":\"failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)\",\"partid\":8,\"amount_msat\":35000000,\"parent_partid\":7},{\"status\":\"failed\",\"failreason\":\"failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)\",\"partid\":9,\"amount_msat\":35000000,\"parent_partid\":8},{\"status\":\"pending\",\"failreason\":\"failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)\",\"partid\":10,\"amount_msat\":35000000,\"parent_partid\":9},{\"status\":\"failed\",\"failreason\":\"failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)\",\"partid\":11,\"amount_msat\":18167122,\"parent_partid\":10},{\"status\":\"failed\",\"failreason\":\"failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)\",\"partid\":12,\"amount_msat\":16832878,\"parent_partid\":10}],\"fullType\":\"lightning\"}
2024-06-17T17:52:39.935Z UNUSUAL lightningd: Bad gossip order: could not find channel 812591x937x0 for peer's channel update
If this issue is related, I'm happy to share more information.
Ok, I think I see what's happening. Just to be sure, can you provide the couple more lines above the snippet you posted? It contains the channel hints that we initialize based on our local view of the channels (listpeerchannels) which we then use to compute the actual route as an overlay on the gossip. It seems that we are picking a local channel that is either not fully reestablished or does not have sufficient balance. Then we somehow track the limits wrong and retry the same channel over and over again. At least skimming the logs, that's my best theory yet. More data and logs will help us pin this down and fix it.
Thanks for the logs, they are incredibly helpful 👍
Thank your for taking the time. I set log levels to debug and did three transactions.
1 sat --> succeeded 10 sats --> succeeded 1000 sats --> failed I had a 128000 sats transaction in the past which succeeded as well.
cln_debug_failed_1000sats_primal.txt cln_debug_succeeded_10sats_phoenix.txt cln_debug_succeeded_1sat_primal.txt
My friend runs an LND node and I'm his only channel. I have 25 large channels and I cannot send a payment to Strike but he can with 100% success make payments to strike despite his only channel being with me. Not sure if this is helpful but it confirms that there is an issue.
I recognize this issue as well. When local balance in direct channel isn't enough payments are failing most of the times. @cdecker is this the part you are looking for:
Multiple times it has occured that local balance in the WoS channel is not sufficient, but in the Kappa channel (which is also WoS) is. But in such cases payments to WoS invoices are failing.
2024-06-22T07:10:13.415Z DEBUG plugin-pay: cmd 34 partid 0: Received getchaininfo blockcount=848999, headercount=848999
2024-06-22T07:10:13.416Z DEBUG plugin-pay: cmd 34 partid 0: waitblockheight reports syncheight=848999
2024-06-22T07:10:13.425Z DEBUG plugin-pay: cmd 34 partid 0: Added a channel hint for 796747x1740x0/1: enabled true, estimated capacity 5195608458msat
2024-06-22T07:10:13.426Z DEBUG plugin-pay: cmd 34 partid 0: Added a channel hint for 829208x14004966x61721/1: enabled false, estimated capacity 0msat
2024-06-22T07:10:13.426Z DEBUG plugin-pay: cmd 34 partid 0: Added a channel hint for 5960895x10194222x36605/1: enabled false, estimated capacity 4294967295msat
2024-06-22T07:10:13.426Z DEBUG plugin-pay: cmd 34 partid 0: Added a channel hint for 848103x2170x1/0: enabled true, estimated capacity 0msat
2024-06-22T07:10:13.427Z DEBUG plugin-pay: cmd 34 partid 0: Added a channel hint for 10454817x11377454x13607/0: enabled false, estimated capacity 0msat
2024-06-22T07:10:13.427Z DEBUG plugin-pay: cmd 34 partid 0: Added a channel hint for 819140x2281x1/1: enabled true, estimated capacity 54359915msat
2024-06-22T07:10:13.427Z DEBUG plugin-pay: cmd 34 partid 0: Added a channel hint for 5591485x9388989x13342/1: enabled false, estimated capacity 0msat
2024-06-22T07:10:13.427Z DEBUG plugin-pay: cmd 34 partid 0: Added a channel hint for 826396x2260x0/1: enabled true, estimated capacity 1248975146msat
2024-06-22T07:10:13.428Z DEBUG plugin-pay: cmd 34 partid 0: Added a channel hint for 4282556x1477417x5590/1: enabled false, estimated capacity 0msat
2024-06-22T07:10:13.428Z DEBUG plugin-pay: cmd 34 partid 0: Added a channel hint for 847889x1670x0/0: enabled true, estimated capacity 1581645457msat
2024-06-22T07:10:13.428Z DEBUG plugin-pay: cmd 34 partid 0: Added a channel hint for 14226214x12490668x34238/0: enabled false, estimated capacity 0msat
2024-06-22T07:10:13.428Z DEBUG plugin-pay: cmd 34 partid 0: Added a channel hint for 831632x831x8/1: enabled true, estimated capacity 197645333msat
2024-06-22T07:10:13.429Z DEBUG plugin-pay: cmd 34 partid 0: Added a channel hint for 12419924x6069763x832/1: enabled false, estimated capacity 0msat
2024-06-22T07:10:13.429Z DEBUG plugin-pay: cmd 34 partid 0: Added a channel hint for 830097x1343x0/0: enabled true, estimated capacity 3583467msat
2024-06-22T07:10:13.429Z DEBUG plugin-pay: cmd 34 partid 0: Added a channel hint for 5448693x1437927x55335/0: enabled false, estimated capacity 0msat
2024-06-22T07:10:13.429Z DEBUG plugin-pay: cmd 34 partid 0: Added a channel hint for 794802x1236x1/1: enabled true, estimated capacity 56440577msat
2024-06-22T07:10:13.430Z DEBUG plugin-pay: cmd 34 partid 0: Added a channel hint for 15232671x14304892x37732/1: enabled false, estimated capacity 0msat
2024-06-22T07:10:13.437Z DEBUG plugin-pay: cmd 34 partid 0: Added a channel hint for 836553x1098x0/1: enabled true, estimated capacity 12981990msat
2024-06-22T07:10:13.438Z DEBUG plugin-pay: cmd 34 partid 0: Added a channel hint for 15804156x10261040x52402/1: enabled false, estimated capacity 0msat
2024-06-22T07:10:13.438Z DEBUG plugin-pay: cmd 34 partid 0: Added a channel hint for 825037x421x0/1: enabled true, estimated capacity 6291173857msat
2024-06-22T07:10:13.438Z DEBUG plugin-pay: cmd 34 partid 0: Added a channel hint for 4909945x337710x31253/1: enabled false, estimated capacity 0msat
2024-06-22T07:10:13.438Z DEBUG plugin-pay: cmd 34 partid 0: Added a channel hint for 806422x1945x1/1: enabled true, estimated capacity 8341msat
2024-06-22T07:10:13.439Z DEBUG plugin-pay: cmd 34 partid 0: Added a channel hint for 5015597x10726383x19215/1: enabled false, estimated capacity 0msat
2024-06-22T07:10:13.439Z DEBUG plugin-pay: cmd 34 partid 0: Added a channel hint for 846557x3093x0/1: enabled false, estimated capacity 0msat
2024-06-22T07:10:13.439Z DEBUG plugin-pay: cmd 34 partid 0: Added a channel hint for 1880118x5887875x11970/1: enabled false, estimated capacity 0msat
2024-06-22T07:10:13.439Z DEBUG plugin-pay: cmd 34 partid 0: Added a channel hint for 826544x3884x1/1: enabled true, estimated capacity 4673825msat
2024-06-22T07:10:13.440Z DEBUG plugin-pay: cmd 34 partid 0: Added a channel hint for 15140828x5950323x21109/1: enabled false, estimated capacity 0msat
2024-06-22T07:10:13.440Z DEBUG plugin-pay: cmd 34 partid 0: Added a channel hint for 774864x1716x1/1: enabled true, estimated capacity 250150121msat
2024-06-22T07:10:13.440Z DEBUG plugin-pay: cmd 34 partid 0: Added a channel hint for 8852482x10869402x60338/1: enabled false, estimated capacity 0msat
2024-06-22T07:10:13.440Z DEBUG plugin-pay: cmd 34 partid 0: Added a channel hint for 776826x3067x1/1: enabled true, estimated capacity 10719488msat
2024-06-22T07:10:13.441Z DEBUG plugin-pay: cmd 34 partid 0: Added a channel hint for 8061376x5028423x11465/1: enabled false, estimated capacity 0msat
2024-06-22T07:10:13.441Z DEBUG plugin-pay: cmd 34 partid 0: Added a channel hint for 844759x290x1/0: enabled true, estimated capacity 59431000msat
2024-06-22T07:10:13.441Z DEBUG plugin-pay: cmd 34 partid 0: Added a channel hint for 4959791x10771686x64007/0: enabled false, estimated capacity 0msat
2024-06-22T07:10:13.552Z DEBUG plugin-pay: cmd 34 partid 0: After filtering routehints we're left with 0 usable hints
2024-06-22T07:10:13.553Z DEBUG plugin-pay: cmd 34 partid 0: Not using a routehint
2024-06-22T07:10:13.553Z DEBUG plugin-pay: cmd 34 partid 0: The destination is directly reachable including attempts without routehints
2024-06-22T07:10:13.650Z INFO plugin-pay: cmd 34 partid 0: Initial limit on max HTLCs: 23475, Destination 035e4ff418fc8b5554c5d9eea66396c227bd429a3251c8cbc711002ba215bfc226 has 1565 channels, assuming 15 HTLCs per channel
2024-06-22T07:10:13.650Z DEBUG plugin-pay: cmd 34 partid 0: Created outgoing onion for route: 774864x1716x1 -> 841217x2174x0 -> 841809x2912x2 -> 830185x454x0
2024-06-22T07:10:13.653Z INFO lightningd: Sending 241409341msat in onion to deliver 241408000msat
2024-06-22T07:10:22.444Z INFO 033d38e07e2541628214c8184cf6c562311059872363e9f538b2f45434418b5bc5-chan#4: htlc 14690 failed from 2th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)
