disco4g
disco4g copied to clipboard
uavpal
Software lock up after temporary signal loss
Hello. Flying a Disco with a large capacity battery mod. Using the Huawei E3372 modem with a 128Gb microSD for video over Verizon in the US. Here's the problem I'm facing. On two separate long range flights so far, I've had the 4g mods "lock up". Here's what happens: Somewhere along the flight I hit a spot where the 4g signal is lost (due to distance from an antenna, aircraft orientation, whatever). Normally, when that happens, when the craft gets back in a good reception area (say in a minute or two), telemetry and FPV comes back and my glympse feed starts updating again. But twice now, I've had where it never reconnects. The first time, the Disco was over an hour away. For an hour I waited watching the skies until it came home. Something in the Disco had locked up to the point that, even when it got back to me and began loitering, I could not reconnect on the direct wifi in order to land the drone. I tried controller off and on, etc. Nothing worked. After the 5 minute loiter, the Disco auto-landed with me never regaining control. I powered the Disco off and back on and, as soon as I did so, everything worked including 4g connection and Glympse. Yesterday, I had a similar experience when I was over an hour and a half into the flight. I had had two momentary signal losses where they had regained. On the third one, I never regained telemetry, FPV or glympse. The drone came all the way home (through the exact same areas it had passed through outbound that have great coverage) with the connection never regaining, with the exception that I got one solitary glympse update about half way back. This time, I was able to take control via direct wifi when it got back. Again, after I powered the drone off and back on, everything worked fine. I have eliminated heat as the likely cause (good ventilation, cool weather, etc). I am running continual video recording onto the microSD in the Huawaii modem. On the first flight (the one where even wifi connection didn't work), the video file that was being written at the time of the loss of 4g control was not closed properly--clear that all connection to the modem had stopped. I don't think it was a cabling issue, since a power off and on immediately restored connection. On the second flight (where 4g and glympse quit but I could control via wifi), the video continued to record to the SD card. Any thoughts on what is occurring?
Hi Steven. I replied on Slack to you on Monday already. Guess you did not see it. There have been occasional reports where a reconnection via 4G is not possible. Can you try to enable the debug logging, this might help to investigate what's going on. Note down the exact time when a disconnection happens the next time. See https://uavpal.com/disco/faq#debug
Are you on the latest firmware versions of Disco/SC2 and the modem?
I never had this issue myself.
SoftIce
S0ftice. Thanks for replying. Yes, I'm on the latest firmware on both. I have enabled debugging. Should be able to do a flight today.
Hi guys, has there been any further developments on this issue ? I had exactly the same issue, my disco came home and loitered till the battery was low and did an auto land. I changed the battery, reconnected after 3 power cycles, it all finally connected again. I than tried to commence another flightplan, and an error message on FFP indicated something along the lines of not being able to load the flight plan.
I'll try running a debug as well.
Cheers.
Dru.
No update no. Waiting for people to share debug logs. Just to make sure you understood: The Skycontroller 2 will not fall-back to Wi-Fi automatically. It needs to be put into Wi-Fi mode manually (double-press Settings again). It can also be powered off and on again to reconnect via Wi-Fi.
The problem described here implies that even that did not work, correct?
Yes, correct, I'll email you the debuts next time I fly. Cheers.
On Mon, 16 Sep. 2019, 11:37 pm SoftIce, [email protected] wrote:
No update no. Waiting for people to share debug logs. Just to make sure you understood: The Skycontroller 2 will not fall-back to Wi-Fi automatically. It needs to be put into Wi-Fi mode manually (double-press Settings again). It can also be powered off and on again to reconnect via Wi-Fi.
The problem described here implies that even that did not work, correct?
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/uavpal/disco4g/issues/64?email_source=notifications&email_token=AMVLJDKQ35GUV5RTCDAKZDTQJ6D2RA5CNFSM4IS4FJDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6ZFDVA#issuecomment-531780052, or mute the thread https://github.com/notifications/unsubscribe-auth/AMVLJDL7XTAK3PZZ7RDIPW3QJ6D2RANCNFSM4IS4FJDA .
So, I had it happen again (I think) on a flight that lasted 4 hours and 5 minutes. Unfortunately, my battery table was a little off and the drone ended up running out of power and crashing in the woods on the way home. I did manage to get the debug file off of it and will send it along with the time at which I lost control signal. I definitely was in good 4G reception. Unlike the other time that I lost control, I continued to receive my Glympse feed and, when I retrieved the video, it continued to record video (the first time, Glympse stopped, wifi wouldn't reconnect when home, and video recording to SD in Huwaei modem quit at the same moment, leaving a corrupted video file). On this last occasion, I can't say for sure whether or not the wifi connection ability locked out. I tried to drive to rendezvous with the disco on its auto-return and think I got close enough to connect via wifi, but never could get the connection. However, it's possible that I was not close enough. I can say that the live video feed locked up even though I got telemetry for a while and then all control locked up, but I still had good 4g connection (according to Glympse feed). Of course, it's possible that was all do to zerotier problems.
I'll send the times and debug file.
If the logs are showing that zerotier have issue, would be nice to be added a VPN (client) connection option (conf.file) to Disco to make connection in same LAN Network with SC2 . If im not mistakeing Huawei e3372 have built in VPN client ...probably no config on disco will be needed...i have to test that.
Hi Nick, if a direct connection is possible, zerotier will not relay the connection via their cloud infrastructure. It will just facilitate the initial establishing of connection but all traffic is directly routed between the two devices. It can also do NAT traversal which is required for most carriers due to their internal network design. A normal VPN client does not do that.
Feel free to use your own VPN client, but it will not be as easy as you described it.
Now question to all who reported this issue, do you guys see a [D]irect or [R]elayed connection in Glympse?
On my last 4G-flight my Disco experienced exactely the same error, as originally described by StevenLawyer: When it lost the connection to the 4G network it was not able to reconnect at all, also when the network coverage was good enough again (during flight-to-home). Additionally it was not transmitting anymore to Glympse. Even connecting directely to the SC2 by wifi, when it finally showed up at the beginning coordinates did not work anymore (Fortunately it made an automatic circular landing which went well). A reboot of the drone fixed the problem. I will run a debug on my next flight in case it happens again.
Yes, please post the debug log if you can. The important question that needs to be clarified is, what is the cause and what is the consequence.
While what you guys describe is a loss of 4G connection causing the Wifi interface to shutdown, might as well be: Loss of all network interfaces that makes 4G disconnect in the first place.
Could it be that the battery used does not provide enough power? Are you all flying on stock battery? The reason why I believe this is not a software issue: No reports of the same on the Bebop2 so far, which uses the same softmod code and modules.
I will.
Hmm, interesting point. Indeed, I am using a battery mod as well and it is possible that it can't handle power peaks as well as the stock battery (higher inner resistance). What I do not understand though, how can this only lead to the shutdown of the wirless devices and not of the whole Chuck?
no idea :) but if you guys can find something that all of your setups have in common, we might get a lead. I never had that issue myself.
Thinking further about this issue, led me to the idea that it could maybe be due to overheating of the Chuck. This could be the case, because I was flying against rather strong wind for more than 30 minutes. Is there a way to log the core temperature of the Chuck?
I’m confident it’s not overheating. Been very cool the times I’ve had it happen.
OK. So yesterday it happened again. Somehow. Not sure if the problem was the same or if the network coverage was insufficient. The situation was the follwoing:
I flew close to a mountain and tried to cross it, but by that time the battery was already half empty. When I stared to fly back home, it suddenly started to loose connection. At first it sort of connected again several times, but bandwith was insufficient to regain control and the video was very very choppy. After some reconnections it disconnected permanently. Because I have an additional GPS tracker installed, I saw that the Disco fought against the wind to return home, but I underestimated the wind at this high altituede and was now unable to pilot the plane to a lower height, so it was not long until the battery was depleated and it stopped returning. Instead it was carried out again by the wind and went down somewhere in the woods. Luckily I was able to retrieve it, because of the additonal GPS tracker.
So now the question is, was it unable to reconnect because of the bad network coverage at this high altitude or because of something else? Like insufficient power? An interesting detail is, that the Chuck was running until the crash (logged everything also the route).
Please have a look at the attached debug log file. I lam looking forward to read your interpretation...
Great, I will look into the log later. One question, when you found the Disco and CHUCK was still running. Did you try to reconnect your SC2 while in Wi-Fi mode? That would give an indication whether the reconnection was not possible at all, or just the 4G reconnection.
On that point, I can tell you that when it happened to me, I could not reconnect via wifi even though the chuck kept flying the aircraft.
Steven, yes I know, not just for you, some more reports as well. Just trying to figure out if marph’s issue is the same or something else. By the way, when you answer via e-mail - due to your signature - your home address and phone number gets immortalized on the public Internet...
@s0ftice : Sorry, I still owe you an answer. I can not tell you for sure if the Chuck was still running, because I retrieved it 10 hrs after it crashed. But as I mentioned, the additional GPS tracker was able to transmit the position during the whole flight, but it's only 2G and another carrier, than the 4G modem, so the coverage might be different. On the other hand it did not reconnect, during its descent either, although the coverage should have been sufficient. So my gut feeling tells me, that it was the same reconnection error all along.
Anyways. Did you discover something in the log?
i did some test`s (about 6 times) on the ground..and i notice that while in 4G mod, if connection on disco is lost for 10 to 20 minutes ...wifi and 4g mod are not accessible anymore, CHUCK is still runing but you cant do nothing to regain control. Il try to extract log file.
That's very useful. Thanks, I will try that. If I can reproduce it on mine, it will be much easier than log files.
@marph43 not yet.
@Nicktk23 I can't reproduce your test. I tried the following
- Disco and SC2 end-to-end connected over 4G - FreeFlight Pro showing the video stream (on the ground).
- I simply turned off SC2
- From my PC, which is connected to the Disco's Wi-Fi, I'm pinging the Disco to see if the Wi-Fi interface is still up (192.168.42.1)
After 1 hour, the Disco is still reachable. At the same time, I also had a telnet session on the Disco open (PC via Wi-Fi) to interactively see the ulogcat - maybe this session prevented the interfaces from going down?
after your last post i did the test once again , i give more details about the test run. 1st step : plug the power from battery and insert 4G modem Huawei 3372h into CHUCK useing OTG cable and turn Disco CHUCK On
2nd step : turn on SC2 and connect normaly on Disco`s WiFI. 3rd step : double press Settings Button on SC2...after few moments SC2 and Disco are connected (as expected in 4G mod)
4th step : double press Settings button on SC2 to switch back on WiFi mod 5th step : turn off SC2....after 14 minutes Huawei e3372h was blinking (showing is disconnect from 4G network) , on WiFi Network scan Disco Parrot wasn`t available anymore.
Maybe these steps help`s .. this glitch works on my parrot every time (sadly).
PS. i didnt keep alive a Telnet session.
Why the 4th step? That would not be a real-life scenario, as when you lose 4G when flying, Wi-Fi will hardly be in range to reconnect immediately.
Can you repeat your test with out 4. ?
I will repeat mine without a telnet session.
I have been following your steps and after 30 minutes, I can still see the SSID in a Wi-Fi scan and the modem shows solid LED. I also ran your step 4, even though it does not make much sense to me as explained above. I also did not even enable telnet with the double-press on CHUCK's power button in today's test.
Maybe someone else wants to try?
after your last post i did the test once again , i give more details about the test run. 1st step : plug the power from battery and insert 4G modem Huawei 3372h into CHUCK useing OTG cable and turn Disco CHUCK On
2nd step : turn on SC2 and connect normaly on Disco`s WiFI. 3rd step : double press Settings Button on SC2...after few moments SC2 and Disco are connected (as expected in 4G mod)
4th step : double press Settings button on SC2 to switch back on WiFi mod 5th step : turn off SC2....after 14 minutes Huawei e3372h was blinking (showing is disconnect from 4G network) , on WiFi Network scan Disco Parrot wasn`t available anymore.
Maybe these steps help`s .. this glitch works on my parrot every time (sadly).
PS. i didnt keep alive a Telnet session.
It seems you have everything working, but manage to obtain disconnection from the Disco (after a while, every time) while other users are not... do you have Glympse installed and active? (Or anything else able to keep the connection active?...). Could you redo your test with Glympse and push notifications tool?
I have been following your steps and after 30 minutes, I can still see the SSID in a Wi-Fi scan and the modem shows solid LED. I also ran your step 4, even though it does not make much sense to me as explained above. I also did not even enable telnet with the double-press on CHUCK's power button in today's test.
Maybe someone else wants to try?
Ok... I made the try... no telnet enabled. But I have the glimpse and push notifications options installed. I followed the steps (including the 4th), and after 90mn was still receiving glimpse updates, my disco modem light was still solid, and I reached 12% on the Disco battery. Then, I started the Skycontroller2 and the hotspot on my Samsung S6 edge+, achieving : solid blue light as usual, controller seen in Auriga/FreeFlight, BUT no telemetry and no video streaming... I didn’t touch the disco but switched the Skycontroller2 to WiFi, which gave me back telemetry/video streams. Then I switched the Skycontroller2 to 4g and obtained again solid blue light, but this time I had telemetry and video streaming as usual and everything was working well. My thoughts: the step 4 made the disco looking for WiFi and not 4g, while being visible for zero tier. Until the Skycontroller2 discussed with it in WiFi and told to it to go looking for data on the 4g subsystem. I can confirm that the 4g modem remained connected during the whole test, maybe because it saw no reason to disconnect or maybe because glimpse kept it “alive”.
This happened to me too today... Luckily It was not very far and somehow I managed to get a WiFi connection and land without video feed (controls worked even if video didn't)
