lnd
lnd copied to clipboard
lightningnetwork
[bug]: Fee policies not broadcasting, odd behavior
I recently fired up a second LND node which is demonstrating odd fee policy behavior. I opened several channels three days ago, and have attempted to change the fee policy via lncli and thunderhub. Fee policy changes are made successfully, verified via lncli. However, over the past three days, the node has not broadcasted the fee policy changes, as reported by all LN network browsers, as well as my original node which has a channel with my second node. Oddly enough, the second node appears to be receiving gossip normally, as I can adjust channel fees from my original node, which are quickly reflected in the channel policies on the second node.
I've reviewed the lnd logs and see no errors. It appears to be updating the network graph / update as expected.
I've made fee rate changes many times over the past couple of days and attempted restarts of lnd; none of which seem to effect the issue.
Your environment
LND 15.3 Ubuntu Bitcoind
- any other relevant environment details
Steps to reproduce
As above
Expected behaviour
Fee policy changes should be accurately updating by peers
Actual behaviour
Second node seems to be receiving fee rate updates appropriately but may not be broadcasting its own policiies?
I've made fee rate changes many times over the past couple of days and attempted restarts of lnd; none of which seem to effect the issue.
Your fee updates could have been rate limited, if you sent out too frequently. I suggest to pause updating fees a bit (two days or so) and then test again.
Your fee updates could have been rate limited, if you sent out too frequently. I suggest to pause updating fees a bit (two days or so) and then test again.
I'd suggest this is not applicable as the original fee change from the opening policy in lnd.conf was allowed ~24 hours to propagate without success. Subsequent changes have been allowed minimum 6-24 hours before making another attempt.
It looks like a rate limiting thing but:
- Does this behavior happen with all the channels, or only with one of them?
- How many peers does your node have?
- Did any of your changes propagate?
- When was the last time that you updated your fees?
- Does this behavior happen with all the channels, or only with one of them?
All channels. Only one rate change was propagated from the node. Currently the node has a channel with my original node, and two others. After channel opening I was able to change fees once on the channel with my original node. However, no changes have since been broadcast for that channel. The other two channels have never appropriately updated fee policy changes.
- How many peers does your node have?
Currently 7. Three are peers with channels.
- Did any of your changes propagate?
As above, only one.
- When was the last time that you updated your fees?
I've attempted again this morning; on average every 24 hours. This morning I switched to tor only, assuming my hydrib/VPN setup could be causing an issue, however, it does not seem to have affected the issue. I'd imagine my hybrid/VPN setup is not to blame as I've been able to successfully route payments via these channels, and the node is appropriately seeing fee policy changes from its peers with which it has active channels
Yes, I also doubt is related to your hydrib/VPN setup.
You coud try:
- Asking some other operator in the network (better if he is connected to your primary node) what fees does he see.
- Updating the fee from your primary node, does that fee get propagated?
- Updating the fee from you secondary node with another channel, do LN Browsers reflect that fee?
I appreciate the ideas!
https://amboss.space/edge/759586x1943x2 This is one of the channels that has not updated fees... It still thinks I'm charging 3000ppm, which is specified on channel open via lnd.conf. The fees seem consistent among several LN browsers...
YES! Oddly enough, when I change fees from my primary node, it is seen by the secondary node. Here is the channel between the two: https://amboss.space/edge/759589x2173x1 ; As you can see, I was able to change fees ONCE on this channel with the secondary node, from 3000 > 1000 ppm; no others have been broadcast. Worth clarifying; the other two channels that the secondary node has with Moon and Boltz havent broadcast a single rate change.
LN browsers have not reflected the fee changes, as you can see from links provided
Oddly enough they finally updated... Only took 3 days. Any other ideas why I could have run into the delay?
My vote goes for rate limiting. Hard to say if it was something else without further investigation. I will close the issue, let us know if it happens again :+1:
I am running into similar issues on testnet. Very hard to get policy updates propagated to explorers. Also with two independent nodes connected to the same external node (opennode in my case, I believe also running lnd), the update doesn't reach the other node.
I understand rate limiting, but if only lnd nodes are involved, isn't that supposed to be only for a minute?
I am running into similar issues on testnet. Very hard to get policy updates propagated to explorers. Also with two independent nodes connected to the same external node (opennode in my case, I believe also running lnd), the update doesn't reach the other node.
I understand rate limiting, but if only lnd nodes are involved, isn't that supposed to be only for a minute?
It is definitely not rate limiting... It took 3 days to change from the original fee policy set on channel opening.
I have a feeling that there is a bug somewhere in the gossip code. In a different context, I did some investigation previously: https://github.com/lightningnetwork/lnd/issues/6531#issuecomment-1125986653
Can confirm that after applying https://github.com/bottlepay/lnd/commit/41564ff4588b62bc517b85f9adad05f8124a5817, my updates propagate to mempool.space promptly. So definitely not rate limiting indeed.
@joostjager how many peers does your node have? Have you modified any of the default gossip config params?
AFAICT, all your diff above does it always send the gossip messages to peers that are your active syncers. Otherwise, we'll treat them only as source of new inbound updates and use the other peers to send out outbound updates.
EDIT: Looking at the diff again, I wonder if there's a variable loop binding bug there, so a lack of:
msgChunk := msgChunk
at the very top of the loop.
At the moment it has 3 peers in ACTIVE_SYNC, but I remember that I tried adding more peers when I couldn't get my updates propagated.
I've seen the same problem in the controlled environment mentioned above. It's also reproducible there. Just need to update the docker lnd rev to a branch that doesn't include the patch.
@Roasbeef the analysis in the link above (horizon not set), does that make any sense?
@mcsnubbs
I'd imagine my hybrid/VPN setup is not to blame as I've been able to successfully route payments via these channels
Just wanted to add that I also started experiencing this issue shortly after switching to hybrid mode using tunnelsats
