OTA updates: Upgrading ESP32-S2 child nodes through an ESP32 Root

[peterNodens]

• Development Kit: Custom boards based on ESP32 Wroom and ESP32 S2 Wroom
• MDF version: ESP-IDF v4.3-beta3

I have seen similar questions, but not with this specific scenario: https://github.com/espressif/esp-mdf/issues/147 https://github.com/espressif/esp-mdf/issues/60 https://github.com/espressif/esp-mdf/issues/142

The scenario I have is that we use a fixed ESP32-wroom root node and our sensors (the child nodes)are ESP32S2-Wroom based. We have found this setup to the best for us due to the limited memory at runtime of the ESP32S2's.

I am working on implementing an OTA upgrade using the MUPGRADE api. As I understand, the procedure is that the root node downloads the new firmware from the HTTP server and must upgrade itself using mupgrade_firmware_download, before sending the firmware onwards to the child nodes. However, if we wish to upgrade our child nodes (the S2 devices) the mupgrade_firmware_download command will of course fail with the error: image has invalid chip ID, expected at least 0, found 2

I have tried to bypass the mupgrade_firmware_download call by first calling mupgrade_firmware_init followed by mupgrade_firmware_send. This fails on the line

MDF_ERROR_CHECK(g_upgrade_config->status.error_code != MDF_ERR_MUPGRADE_FIRMWARE_FINISH,
                    MDF_ERR_MUPGRADE_FIRMWARE_INCOMPLETE, "mupgrade_firmware_download");

I have simply commented that error check out, which allows me to send the firmware to the child nodes, but that then fails on the child nodes with the following:

I (484315) [mupgrade_node, 220]: Write total_size: 1191136, written_size: 1191136, spend time: 11s
E (484320) esp_image: image at 0x1f0000 has invalid magic byte
W (484321) esp_image: image at 0x1f0000 has invalid SPI mode 255
W (484327) esp_image: image at 0x1f0000 has invalid SPI size 15
W (484334) [mupgrade_node, 233]: <ESP_ERR_OTA_VALIDATE_FAILED> esp_ota_set_boot_partition
W (484343) [mupgrade_node, 274]: <ESP_ERR_OTA_VALIDATE_FAILED> mupgrade_handle
W (484351) [my_mesh, 387]: <ESP_ERR_OTA_VALIDATE_FAILED> mupgrade_handle

Question: Is there a way (at least in theory) to send an S2 firmware to the child nodes through an S1 root node, without updating the root node first? Am I on the right track?

[will-emmerson]

When you call mupgrade_firmware_send you give it a list of mac addresses you want to send the firmware to. So yes you have to send the firmware to the root node and it gets saved to an ota partition but you don't have to actually update the firmware on that node, you can just use it as somewhere to save the firmware before sending it to your chosen nodes. I am using this method and it works for me, but it would be interesting to hear from someone at espressif if this is actually sensible. The only issues are that the node firmware can't be bigger than the root firmware partition size but this is very unlikely as nodes are usually much simpler. You might also run into problems if you're using ota rollback or factory reset on the root node as it will possibly roll back into the node firmware in the other OTA slot.

On the subject of limited memory for the S2, I recently managed to increase the amount of ram by a) turning all tasks into timers and b) using certain settings such as:

CONFIG_MBEDTLS_DYNAMIC_BUFFER=y
CONFIG_MBEDTLS_DYNAMIC_FREE_PEER_CERT=y

All the espressif examples seem to use tasks which all use seperate areas of ram but most tasks can be turned into timers which all use the same area of ram.

[peterNodens]

Hi @will-emmerson . Thanks for the reply and the information. It was helpful to know that it was possible as it gave me the confidence to keep going with it. Based on what you said, I just commented out a few bits of mupgrade_root.c to stop it checking the firmware validity when running mupgrade_firmware_download_finished. Probably not the best approach and I haven't really tested it thoroughly to see if my changes have introduced any issues but I have at least been able to update my child nodes, so its a step in the right direction. I agree, it would be good to have some further guidance on an approach for this.

thanks for the tip on the memory, i will have try with the timer technique. I managed to free up a decent amount of space (but still not enough for a reliable root node) by turning off some of the WiFi IRAM optimisations, without any obvious impact.

[Jiangyafeng]

@peterNodens You can do such a test to shield it from the inspection of the firmware header, so that the upgrade can proceed normally. But there may be a certain risk in this way, that is, if you use the esp32s2 firmware to upgrade the esp32, the upgrade will be normal, but there will be a crash.

The specific location can be added as follows:

esp-mdf/esp-idf/components/app_update/esp_ota_ops.c

#if 0
     if (esp_image_verify(ESP_IMAGE_VERIFY, &part_pos, &data) != ESP_OK) {
         ret = ESP_ERR_OTA_VALIDATE_FAILED;
         goto cleanup;
     }
#endif

[will-emmerson]

@peterNodens Sorry I misread your initial question, I see what the problem is now. The sensible thing would be for esp-mdf to have CONFIG_FIRMWARE_VALIDATE or something which you could set to false to do your own validation. Why are you using S2's as the nodes just out of interest?

[peterNodens]

Hi @Jiangyafeng , Thanks for your reply. That works well and I will perhaps deal with checking the firmware somehow else to prevent that scenario.

@will-emmerson , no problem, you pointed me to the correct place so that was very helpful. We designed this set of prototype boards before implementing the esp-mdf in our sensors and weren't aware of how much memory the mesh used as we hadn't implemented it at the time. We were quite interested in using the Time of Flight features that the S2 has, which was another reason for it. Our normal sensors (non-mesh) have plenty of memory left when using the S2, but once they are configured with a mesh, the memory gets quite low (at least too low to be reliable as a root node). Our next set of prototypes will certainly use either the ESP32 or the new S3 when it eventually is released. Hindsight is a wonderful thing, but if we designed them again we'd definitely be using the ESP32 rather than the S2, but now we are just having to work around the problem as best as we can.

[mmrein]

You might also run into problems if you're using ota rollback or factory reset on the root node as it will possibly roll back into the node firmware in the other OTA slot.

It seems to me that one possibility would be, after nodes update proces is finished, not only prevent the root from restarting, but also erase otadata partition (and maybe even ota_1 partition if it was used to store the update firwmare). That should (it seems) result in booting the default application in ota_0 partition.

Or, in case you want to be able to OTA update the root too, use the esp_ota_set_boot_partition(const esp_partition_t *partition) function to explicitly set the boot parition for root?

@will-emmerson Have you tried something like that to prevent the root from booting node's upadte firmware?

[will-emmerson]

@mmrein I haven't tried that as it was a bit of a toy project and didn't go into production in the end, but something like that would be a good idea. I think you'd want to use esp_ota_get_next_update_partition to find where the nodes firmware is being stored and erase it.

When I did it I didn't have to prevent the root from rebooting into the node's firmware or have any problems with validation. If the root isn't included in the list of devices to send firmware to then neither of those issues should be a problem as nothing will be trying to reboot the root to a new partition. And if you want to upgrade just the root then you send a firmware to the root alone, it shouldn't be more complicated than that.

[mmrein]

If the root isn't included in the list of devices to send firmware to then neither of those issues should be a problem as nothing will be trying to reboot the root to a new partition.

I'll have to check if it actually works like that, but it make sense.

I'm worried about reset from 'another' source anytime after the update so I want to be sure the boot partition does not switch accidentally. But I'm also still learning how all this actually works so it may not even be a problem. Thanks for the ideas!

[will-emmerson]

Just because there is something in the non-boot partition, doesn't mean it's going to reboot into it. The OTA process involves downloading the new update to the non-boot partition, checking it (possibly), then and only then setting a value in ota_data to boot to the new partition. But this last part won't happen on the root if the firmware is just "resting" there temporarily. But you might want to double check that before deploying thousands of devices! :fearful: :fearful: :fearful:

[mmrein]

So I can confirm this is true if root's address is not used when calling mupgrade_firmware_send():

Just because there is something in the non-boot partition, doesn't mean it's going to reboot into it. The OTA process involves downloading the new update to the non-boot partition, checking it (possibly), then and only then setting a value in ota_data to boot to the new partition. But this last part won't happen on the root if the firmware is just "resting" there temporarily.