Allow flashing autopilot from file-on-SD-card

[peterbarker]

Longer-term it would be far nicer to be able to use the HAL. This might allow us to use "blackbox' devices (and perhaps serial-based-loggers) for flashing the autopilot.

Note that this is a little incomplete - no renaming of files ATM.

[durka]

Would be happy to test this out / provide feedback (btw, I'm @aburka on Discord). I'd be using a CubeOrange or PH4-Mini.

2 questions:

1. Is there documentation on usage?
2. If it goes sideways what do I need to recover? An ST-Link?

[peterbarker]

Would be happy to test this out / provide feedback (btw, I'm @aburka on Discord). I'd be using a CubeOrange or PH4-Mini.

Strongly suggest testing this on something other than a CubeOrange. If something goes wrong with flashing then you will end up with an uphill battle recovering that board.

I can't lay my hands on the PH4-Mini I have floating around here ATM and can't recall how recoverable that is.

Ideally you use something with proper debug connectors, or at least something that exposes boot0. And maybe something less expensive than a CubeOrange....

1. Is there documentation on usage?

Nope :-)

I think it would roughly be:

• Rebuild the bootloader for the board using the relevant script which also copies it into the appropriate place to be included in the main firmware image
• rebuild the main firmware
• flash that onto the board
• tell that running ArduPilot image to flash the bootloader
• build an .abin for your autopilot (you can see a patch in here which means whenever I build Pixhawk6x I get a .abin for it.
• Put that file in the root directory of your SD card called ardupilot.abin. every time you reboot it will flash that to the autopilot.

2. If it goes sideways what do I need to recover? An ST-Link?

That comes under the "proper debug connectors" as above. boot0 and dfu-util are other options. Be comfortable that you can recover the autopilot before trying this stuff :-)

Always good to try to tell people how to do this sort of thing. Perhaps we should look for any of the vehicle names instead of requiring it be called "ardupilot.abin" - so look for "arduplane.abin" then "arducopter.abin" etc etc

[magicrub]

This is a great feature, thanks @peterbarker ! Once we get this in, we stopped consider making this a standard feature for bootloader's that can fit it.. like all 128k BLs in H743

[peterbarker]

I've now adjusted this so the file is renamed as we progress through the flash-from-sdcard sequence.

I think this is now feature-complete.

[peterbarker]

This is a great feature, thanks @peterbarker ! Once we get this in, we stopped consider making this a standard feature for bootloader's that can fit it.. like all 128k BLs in H743

I definitely think it's a candidate for that. Problem is that it might not actually fit on that 128kB once we get @bugobliterator 's RemoteID stuff (https://github.com/ArduPilot/ardupilot/pull/21456) in!

I've marked this as consideration for ArduPilot 4.4 - will need to rebase this on Sid's work as it's important for flight in India FWIU.

[peterbarker]

Adds ~21kB where it is used:

Orange:
bootloader/AP_Bootloader     38166       200   130872                 38366           92704
bootloader/AP_Bootloader     17792        92   130976                 17884          113188

Pixhawk6X:
bootloader/AP_Bootloader     38122       200   130876                 38322           92744
bootloader/AP_Bootloader     17732        92   130980                 17824          113244

Pixhawk1-1M:
bootloader/AP_Bootloader     16272        64   196544                 16336              48
bootloader/AP_Bootloader     16272        64   196544                 16336              48

Durandal:
bootloader/AP_Bootloader     17508        92   130980                 17600          113468
bootloader/AP_Bootloader     17508        92   130980                 17600          113468

[peterbarker]

I've now tested this end-to-end on CubeOrange and on a Pixhawk6X.

Testing is straight-forward; get the bootloader onto a board, then (e.g.):

ftp put /home/pbarker/rc/ardupilot/build/Pixhawk6X/bin/arduplane.abin /ardupilot.abin

If someone would like a firmware to try which incorporates the new bootloader, ping me.

[rmackay9]

I think this could be useful for OTA (over the air) updates.

[DavideLentini]

Hello everyone ! I find this feature incredibly interesting and useful especially in vision to upload the ardupilot.abin firmware with MAVFTP to the SD card and then upload the new firmware directly to the drone ... a real OTA update . I would like to understand the status of the development and if it is possible to test it on my drone . Thank you all for your efforts !

[pompecukor]

This would be such an amazing feature. I would love to have it working. @peterbarker ARACE will buy you some a crate of beer if you get it in plane and copter. Just to add some extra (fun) motivation. :)

[DavideLentini]

Hello @tridge @peterbarker . I would like to congratulate Peter and all the developers who dedicate patience . I would like to ask a question , I have tried all Peter's procedure successfully , but loading ardupilot.abin to the SD card via MavFtp , via COM or via UDP with Mission Planner , takes about 2 minutes . I asked a colleague and according to him the loading time is strangely long . I would like to ask you if the time seems reasonable or not . Thank you all .And I also have another problem , I can't rename the file via MavFtp , it keeps giving a Fail .

[DavideLentini]

I think this could be useful for OTA (over the air) updates.

Really interesting ! I too was just thinking that this feature has the potential to become a new method(the first?) for OTA. Actually in a sense it already has an OTA feature , the procedure would be to : 1)Turn on the vehicle(with appropriate bootloader) 2)Via FTP delete the firmware.abin(if there are any), upload the firmware.abin to the SD card via MavFTP 3)When the upload is finished , send a command to reboot the drone(reboot and stay in bootloader) . Upon reboot the drone will upload the firmware.abin

[khancyr]

2 min isn't that long. I won't be much surprised considering that MAVFTP don't have priority over the other tasks running on the Autopilot. We could probably make a new FTP path that is handled as the log downloading to allow a fast upload

[ntamas]

2 min isn't that long. I won't be much surprised considering that MAVFTP don't have priority over the other tasks

I think the key bottleneck (compared to the performance that you see with standard TCP file transfers over the Internet) is that the GCS needs to wait for an ACK for every chunk of a file that was sent via MAVFTP to the flight controller, even if the transfer happens over a reliable link (USB cable or TCP stream). So, the GCS spends most of its time waiting for an ACK before it can send the next chunk. MAVFTP has a BurstRead operation for downloads that can be used to work around the long waiting times, but no BurstWrite.

On the upside of things, the fact that the GCS spends most of its time waiting means that you can parallelize the firmware upload process to multiple drones without noticeable performance loss.

[DavideLentini]

2 min isn't that long. I won't be much surprised considering that MAVFTP don't have priority over the other tasks

I think the key bottleneck (compared to the performance that you see with standard TCP file transfers over the Internet) is that the GCS needs to wait for an ACK for every chunk of a file that was sent via MAVFTP to the flight controller, even if the transfer happens over a reliable link (USB cable or TCP stream). So, the GCS spends most of its time waiting for an ACK before it can send the next chunk. MAVFTP has a BurstRead operation for downloads that can be used to work around the long waiting times, but no BurstWrite.

On the upside of things, the fact that the GCS spends most of its time waiting means that you can parallelize the firmware upload process to multiple drones without noticeable performance loss.

This is very good news @ntamas ! I mean the parallelization of firwmare upload. To answer @khancyr , I used MavProxy via CLI , and the ftp put command , and strangely instead of taking 2 minutes as on Mission Planner , the same file took only 1 minute to be moved .

[peterbarker]

Since @DavideLentini has successfully used this PR I'm going to mark it again for DevCall.

If anyone else can take the time to prove it Works For Them, that would be great - please put your results in here :-)

(someone else on the DevTeam said they wanted to test it themselves before merging - but they're rather busy and/or slightly forgetful ;-) )

[peterbarker]

This would be such an amazing feature. I would love to have it working. @peterbarker ARACE will buy you some a crate of beer if you get it in plane and copter. Just to add some extra (fun) motivation. :)

:-)

Not a big beer drinker (might be a couple of decades now....). Chocolate OTOH ;-)

[DavideLentini]

@peterbarker @tridge An excellent job , and I confirm that it works perfectly . But for a future inclusion in the official firmware ardupilot I would propose the creation of a script (perhaps integrated with MavProxy) that can help the user to make an OTA update of the firmware , then with the need to only turn on the drone once . The procedure could look like this :

1. You turn on the drone (with the correct bootloader for OTA).
2. You connect to the drone AP network.
3. Starting the script it would delete the previous firmware.abin(if it exists in the sd card)
4. At this point it would upload the firmware.abin selected by the user again via FTP
5. And at the end of the upload(if everything was successful) , the script would send a mavlink command to the drone requesting a "reboot and stay in the bootloader" command , at this point the real firwmare uploading procedure would start and it would end with the drone always on and with the new firmware loaded .

I have tried creating a prototype of this system , by running the command (ftp put "example.abin" /"example.abin") with the --cmd argument of MavProxy.exe , but the problem is that the command starts right at the beginning and I think it has to wait for the full connection with the drone . Just thoughts . great job anyway . 😄

[peterbarker]

Can be merged after a quick size-check on master

[peterbarker]

I'm not sure we should be merging this without external flash support?

Why not? So it can't work on boards with external flash (which are, FWIU essentially non-existent right now), but that can be fixed without breaking non-exernal-flash boards. Non-external-flash boards can have their hwdefs modified if feature is desired on those boards - it's an "off by default" sort of thing as I haven't gone through and modified the hwdefs.

[peterbarker]

Size check is fine:

Board              AP_Periph  blimp  bootloader  copter  heli  iofirmware  plane  rover  sub
CubeOrange                    16     20500       16      16                16     16     16
Durandal                      *      *           *       *                 *      *      *
Hitec-Airspeed     0                                                                     
KakuteH7-bdshot               *      *           *       *                 *      *      *
MatekF405                     *      *           *       *                 *      *      *
Pixhawk1-1M                   *      *           *       *                 *      *      *
f103-QiotekPeriph  *                                                                     
f303-Universal     *                                                                     
iomcu                                                          *                         
revo-mini                     *      *           *       *                 *      *      *
skyviper-v2450                                   *                                       

[peterbarker]

Discussed at DevCall. abin format would probably need changing the external flash support.