erlang-libp2p
erlang-libp2p copied to clipboard
helium
IP change not detected
Many EU ISPs change consumer IP addresses every 24-48 hours.
There have been a number of reports where the IP address changes, and while the API has a recent peerbook record the IP address was not updated to reflect the change either never, or it took a very long time which put it past the time for the next IP update.
Restarting the hotspot seems to fix it every time this was observed.
The theory is that IP change detection is failing, not being done often enough or not being done at all?
This occurs in the helium OG, and rak's. This happens on occasion. With about 50+ miners this has only occurred a few times with me. I have use a wifi outlet that I can remotely turn off and on. It is a minor inconvenience for me, but for others this might be worse as the hotspots aren't able to be adjusted properly.
NO BEACONS
I've recently experienced the exact problem with 2 of my hosts and miners where they did not send any beacons for over 9 days which was concerning, when we rebooted their routers beacons started to happen again in the next few hours.
This problem has now become more of a recent report of events by several people in the South African Helium Community, as our ISP also changes IPs every 24-48 hours.
And there are now several hotspots where you would see they have not sent beacons at all in the last week or more and they are not even aware of this problem or how to resolve it.
This is becoming a problem now for us as we are starting to witness less as beacons are not being sent due to this problem.
And we have seen this with Bobcat and Sensecap miners.
Hope this will get as much attention to be resolved as soon as possible, please.
Can anyone affected help me troubleshoot this? I need access to an affected hotspot, or at least the logs from it.
what exactly is needed, i can provide it because i am affected by this problem.
@chillbreak a hotspot name would be helpful. It’s easiest for the core team if it’s an OG or Rak
I've seen the issue and reproduced it with the following test:
- Hotspot behind CG NAT (shared IP) and thus relayed
- Route hotspot through a VPN to a public IP
- Forward port 44158
- Peerbook "listen ip" is updated from the relay address to the CG NAT shared IP, and not the VPN public IP that it's actually using. It is as if the hotspot caches the IP it initially discovered and never bothers to check it again.
I know what the problem is, I just don't know why it happens. Prehistoric pearl dinosaur is not a hotspot I have support access too, unfortunately.
A simple reboot of the miner/hotspots resolves the issue, as during reboot it checks for an public ip change. This has to be implemented periodically, as it is nothing high sophisticated... But this issue is quite critical, as in some cases this would result in an unreachable miner for several days/months.
I know what the problem is, I just don't know why it happens.
Prehistoric pearl dinosauris not a hotspot I have support access too, unfortunately.
Perhaps try to reproduce it by changing the outward facing IP of a hotspot or test setup? Are you having trouble reproducing the issue?
@Vagabond if you DM me and we have a willing volunteer from https://github.com/NebraLtd/helium-miner-software/issues/108 I can give you logs / info about the HS. I know @dakky221 has this issue
Yeah, I'm having the same issue, and I "fixed" it by plugging the miner into a smart-plug and making it reboot the miner on IP change, but that will only decrease the lifespan of the hotspot and must be other way around, like @westerwalder said - add periodically check for public ip, for example every 5 minutes, it would fix the issue entirely.
btw @shawaj unfortunatley my Nebra is now behind CGNAT on a semi-static IP so my logs won't really help. I can share the Sensecap's logs though if there is a way to get them.
Yep, plugged my hs on a timer as well to reboot every 24h. Currently takes 2-3 hrs to reflect updated IP, but lag seems to increase (week before: ~1hr) due to network growth(?).
By (unsafely) rebooting it that often, you can directly say goodbye to the sd-card/filesystem..
I have cloned SD-Cards as backup should that happen. But I'm about to switch to tunneling a static IP instead.
By (unsafely) rebooting it that often, you can directly say goodbye to the sd-card/filesystem..
Well, hopefully the libp2p gets updated to periodically check for external IP before the sd-card goes kaboom ;)
I should have a fix for this today or tomorrow that we can test.
That is fantastic, will this fix be rolled out in an OTA on all miners or how will the update be released?
I should have a fix for this today or tomorrow that we can test.
I can renew the IP as many times as you wish on my VDSL2 connection, but the hotspot is Sensecap, so if they can deploy the updated firmware we can test.
There is no way to be sure if the fix worked or it didn't. Listen addr doesn't change on external IP change, but I guess that's normal because it gets it from the API and API is too slow to be of any use. I also guess manufacturers should add to the dashboard some kind of a log if the hotspots sent the new IP to the peers to be sure...
My isp updated the ip address about 10 hours ago, but looking at the address on the api it still shows the previous one. but when I just checked inside peerbook it shows the correct ip address, of course I had rebooted the hotspot before that so not sure if it was the reboot that made the peerbook change. but as of the time of the comment the api still shows the wrong address.
Rebooting the hotspot will find the new address, yes. The API may not have a perfectly up to date view of the system so it may take a while to see the IP change (if the hotspot notices it).
OK, just to add: I didn't reboot the hotspot since the IP change. 10 hours already passed and it's not updated on peerbook or API. Listen addr. is still the yesterday's IP. Will reboot hotspot now, I guess 10 hours is enough of waiting to see that fix isn't working.
It pains me to say that my Sensecap on firmware 2021.08.24.4 still shows the same behaviour and has not updated its IP address in several days (API and peerbook). Consequently the last beacon it performed was 5 days ago (compounded by the block halt issue).
Ok, I guess I can't fix it blind. I need details on these affected hotspots (manufacturer and hotspot name) so I can try to get some logs from them. You can privately provide me details via email to [email protected]
All of my sensecaps (all on newest fw) are still affected. Only a reboot will result in an updated listen-ip information due to a changed ISP public ip change.