Allow user to pass partition label to the handleUpdate method of the HTTPUpdate class.

[zekageri]

Related area

HTTPUpdate

Hardware specification

Every esp32

Is your feature request related to a problem?

I'm facing an issue where i have custom partitions and i want to update only one of them with HTTPUpdate. My idea is that i want a "user" partition and a "system" partition. The system partition would hold the web related files, the user partition would hold user configuration and things like that. On a firmware update, the esp would download a new firmware.bin and a new littlefs.bin but it would spaw only my system partition so the user configuration will remain intact.

Describe the solution you'd like

Currently there is no possibility to update only one particular partition. The HTTPUpdater picks the last partition labeled as "spiffs" and wants to put the downloaded bin file to this partition. ( see line 263 in HTTPUpdate.cpp )

It would be good if we could pass a partition label to updateSpiffs method.

Current: Updater.updateSpiffs(_wifi_client, url) Possible: Updater.updateSpiffs(_wifi_client, url,"system")

Describe alternatives you've considered

Alternatively i'm doing this right now:

In HTTPUpdate.cpp at line 183 there is a method handleUpdate. In this method i replaced this line const esp_partition_t* _partition = esp_partition_find_first(ESP_PARTITION_TYPE_DATA, ESP_PARTITION_SUBTYPE_DATA_SPIFFS, NULL); with this line: const esp_partition_t* _partition = esp_partition_find_first(ESP_PARTITION_TYPE_DATA, ESP_PARTITION_SUBTYPE_DATA_SPIFFS, "system"); So it will search for my custom partition.

After that still in HTTPUpdate.cpp at line 398 there is a method runUpdate. In this method i replaced this line if(!Update.begin(size, command, _ledPin, _ledOn)) { with this line: if(!Update.begin(size, command, _ledPin, _ledOn,"system")) { so i passed my custom partition to the begin method of the Update class.

Now in my sketch if i call Updater.updateSpiffs(_wifi_client, url); it will update my custom partition.

I have checked existing list of Feature requests and the Contribution Guide

• [x] I confirm I have checked existing list of Feature requests and Contribution Guide.

[zekageri]

Can this be implemented? I just updated the core and i had to do these changes again.

[lbernstone]

Can you submit a PR with your changes? I recommend not using a partition for your read-only files, but rather including them in your firmware as C-string arrays. You can use xxd -i to convert files in the file system, and script a method to include them as header files. example

[zekageri]

I can create a PR but that would require much more analysis. Currently i just hardcoded my partition label but in order for this to work for everyone we have to rewrite a bunch of functions so we can pass partition label.

I can't include them easily because I'm using a webserver library built on top of the IDF webserver and my partition contains a frontend application. The webserver handles routes with LittleFS. I don't want to build a routing library just for this so it can pick the files up from the c-string arrays. There are several nested routes handles with urls and things like that. Also because the frontend app is bundled with webpack i don't even know the file names which will be requested by the frontend.

[lbernstone]

Ok. Whoever has to support this is going to looove you.

[zekageri]

Well, it was already a bad design from the start. If we can create multiple partitions why can't we update them individually from the start? I mean, the option is there but the implementation left it out.

[lbernstone]

I don't mean the feature request, which seems like a fine idea, but your system, where the html app is going to be able to get out of sync with the firmware version.

[zekageri]

Ah it cant. With each update it downloads both files at once from a server :D

[zekageri]

Each version comes with a backend and a frontend.

[lbernstone]

How about if we supercede the updateSpiffs idea, and if you call HTTPUpdate(const char* partition), it will search for that partition, attempt to determine the partition type, and call the appropriate Update?

[zekageri]

That would be nice

[zekageri]

That way we dont have to modify the existing functions

[zekageri]

It isnt even spiffs anymore so

[lbernstone]

Looks like where I am headed is making a completely alternate route in Update::begin, with a command called U_AUTO, which will use the label name. That way it is a non-breaking change. If it is good enough at the end of the day, it may be able to be a full replacement.

[zekageri]

I had to specify the label at two different places in order to work. Will check on it tomorrow

[zekageri]

updateSpiffs(WiFiClient& client, const String& url, const String& currentVersion, HTTPUpdateRequestCB requestCB) calls handleUpdate(HTTPClient& http, const String& currentVersion, bool spiffs, HTTPUpdateRequestCB requestCB) which then calls runUpdate(Stream& in, uint32_t size, String md5, int command). I don't see what you see in here.

[zekageri]

If we create a new function like update(HTTPClient& http,const char* partitionLabel,HTTPUpdateRequestCB requestCB) we still need to call handleUpdate and runUpdate ?!

[zekageri]

What i can see here is that we must modify the handleUpdate and runUpdate methods like this

t_httpUpdate_return handleUpdate(HTTPClient& http, const String& currentVersion, bool spiffs = false, const char* partitionLabel = NULL, HTTPUpdateRequestCB requestCB = NULL);
bool runUpdate(Stream& in, uint32_t size, String md5, int command = U_FLASH, const char* partitionLabel = NULL,);

That way we can make a method which a user can call like this

HTTPUpdateResult HTTPUpdate::updatePartition(WiFiClient& client, const String& url, const char* partitionLabel, HTTPUpdateRequestCB requestCB)
{
    HTTPClient http;
    if(!http.begin(client, url))
    {
        return HTTP_UPDATE_FAILED;
    }
    return handleUpdate(http, currentVersion, true, partitionLabel, requestCB);
}

[zekageri]

This would looke like that:

Methods

t_httpUpdate_return handleUpdate(HTTPClient& http, const String& currentVersion, bool spiffs = false, const char* partitionLabel = NULL, HTTPUpdateRequestCB requestCB = NULL);
bool runUpdate(Stream& in, uint32_t size, String md5, int command = U_FLASH, const char* partitionLabel = NULL);
HTTPUpdateResult updatePartition(WiFiClient& client, const String& url, const char* partitionLabel, HTTPUpdateRequestCB requestCB);

updatePartition

HTTPUpdateResult HTTPUpdate::updatePartition(WiFiClient& client, const String& url, const char* partitionLabel, HTTPUpdateRequestCB requestCB)
{
    HTTPClient http;
    if(!http.begin(client, url))
    {
        return HTTP_UPDATE_FAILED;
    }
    return handleUpdate(http, currentVersion, true, partitionLabel, requestCB);
}

handleUpdate

/**
 *
 * @param http HTTPClient *
 * @param currentVersion const char *
 * @return HTTPUpdateResult
 */
HTTPUpdateResult HTTPUpdate::handleUpdate(HTTPClient& http, const String& currentVersion, bool spiffs, const char* partitionLabel, HTTPUpdateRequestCB requestCB)
{

    HTTPUpdateResult ret = HTTP_UPDATE_FAILED;

    // use HTTP/1.0 for update since the update handler not support any transfer Encoding
    http.useHTTP10(true);
    http.setTimeout(_httpClientTimeout);
    http.setFollowRedirects(_followRedirects);
    http.setUserAgent("ESP32-http-Update");
    http.addHeader("Cache-Control", "no-cache");
    http.addHeader("x-ESP32-STA-MAC", WiFi.macAddress());
    http.addHeader("x-ESP32-AP-MAC", WiFi.softAPmacAddress());
    http.addHeader("x-ESP32-free-space", String(ESP.getFreeSketchSpace()));
    http.addHeader("x-ESP32-sketch-size", String(ESP.getSketchSize()));
    String sketchMD5 = ESP.getSketchMD5();
    if(sketchMD5.length() != 0) {
        http.addHeader("x-ESP32-sketch-md5", sketchMD5);
    }
    // Add also a SHA256
    String sketchSHA256 = getSketchSHA256();
    if(sketchSHA256.length() != 0) {
      http.addHeader("x-ESP32-sketch-sha256", sketchSHA256);
    }
    http.addHeader("x-ESP32-chip-size", String(ESP.getFlashChipSize()));
    http.addHeader("x-ESP32-sdk-version", ESP.getSdkVersion());

    if(spiffs) {
        http.addHeader("x-ESP32-mode", "spiffs");
    } else {
        http.addHeader("x-ESP32-mode", "sketch");
    }

    if(currentVersion && currentVersion[0] != 0x00) {
        http.addHeader("x-ESP32-version", currentVersion);
    }
    if (requestCB) {
        requestCB(&http);
    }

    if (!_user.isEmpty() && !_password.isEmpty()) {
        http.setAuthorization(_user.c_str(), _password.c_str());
    }

    if (!_auth.isEmpty()) {
        http.setAuthorization(_auth.c_str());
    }

    const char * headerkeys[] = { "x-MD5" };
    size_t headerkeyssize = sizeof(headerkeys) / sizeof(char*);

    // track these headers
    http.collectHeaders(headerkeys, headerkeyssize);


    int code = http.GET();
    int len = http.getSize();

    if(code <= 0) {
        log_e("HTTP error: %s\n", http.errorToString(code).c_str());
        _lastError = code;
        http.end();
        return HTTP_UPDATE_FAILED;
    }


    log_d("Header read fin.\n");
    log_d("Server header:\n");
    log_d(" - code: %d\n", code);
    log_d(" - len: %d\n", len);

    String md5;
    if (_md5Sum.length()) {
        md5 = _md5Sum; 
    } else if(http.hasHeader("x-MD5")) {
        md5 = http.header("x-MD5");
    }
    if(md5.length()) {
        log_d(" - MD5: %s\n",md5.c_str());
    }

    log_d("ESP32 info:\n");
    log_d(" - free Space: %d\n", ESP.getFreeSketchSpace());
    log_d(" - current Sketch Size: %d\n", ESP.getSketchSize());

    if(currentVersion && currentVersion[0] != 0x00) {
        log_d(" - current version: %s\n", currentVersion.c_str() );
    }

    switch(code) {
    case HTTP_CODE_OK:  ///< OK (Start Update)
        if(len > 0) {
            bool startUpdate = true;
            if(spiffs) {
                const esp_partition_t* _partition = esp_partition_find_first(ESP_PARTITION_TYPE_DATA, ESP_PARTITION_SUBTYPE_DATA_SPIFFS, partitionLabel);
                if(!_partition){
                    _lastError = HTTP_UE_NO_PARTITION;
                    return HTTP_UPDATE_FAILED;
                }

                if(len > _partition->size) {
                    log_e("spiffsSize to low (%d) needed: %d\n", _partition->size, len);
                    startUpdate = false;
                }
            } else {
                int sketchFreeSpace = ESP.getFreeSketchSpace();
                if(!sketchFreeSpace){
                    _lastError = HTTP_UE_NO_PARTITION;
                    return HTTP_UPDATE_FAILED;
                }

                if(len > sketchFreeSpace) {
                    log_e("FreeSketchSpace to low (%d) needed: %d\n", sketchFreeSpace, len);
                    startUpdate = false;
                }
            }

            if(!startUpdate) {
                _lastError = HTTP_UE_TOO_LESS_SPACE;
                ret = HTTP_UPDATE_FAILED;
            } else {
                // Warn main app we're starting up...
                if (_cbStart) {
                    _cbStart();
                }

                WiFiClient * tcp = http.getStreamPtr();

// To do?                WiFiUDP::stopAll();
// To do?                WiFiClient::stopAllExcept(tcp);

                delay(100);

                int command;

                if(spiffs) {
                    command = U_SPIFFS;
                    log_d("runUpdate spiffs...\n");
                } else {
                    command = U_FLASH;
                    log_d("runUpdate flash...\n");
                }

                if(!spiffs) {
/* To do
                    uint8_t buf[4];
                    if(tcp->peekBytes(&buf[0], 4) != 4) {
                        log_e("peekBytes magic header failed\n");
                        _lastError = HTTP_UE_BIN_VERIFY_HEADER_FAILED;
                        http.end();
                        return HTTP_UPDATE_FAILED;
                    }
*/

                    // check for valid first magic byte
//                    if(buf[0] != 0xE9) {
                    if(tcp->peek() != 0xE9) {
                        log_e("Magic header does not start with 0xE9\n");
                        _lastError = HTTP_UE_BIN_VERIFY_HEADER_FAILED;
                        http.end();
                        return HTTP_UPDATE_FAILED;

                    }
/* To do
                    uint32_t bin_flash_size = ESP.magicFlashChipSize((buf[3] & 0xf0) >> 4);

                    // check if new bin fits to SPI flash
                    if(bin_flash_size > ESP.getFlashChipRealSize()) {
                        log_e("New binary does not fit SPI Flash size\n");
                        _lastError = HTTP_UE_BIN_FOR_WRONG_FLASH;
                        http.end();
                        return HTTP_UPDATE_FAILED;
                    }
*/
                }
                if(runUpdate(*tcp, len, md5, command, partitionLabel)) {
                    ret = HTTP_UPDATE_OK;
                    log_d("Update ok\n");
                    http.end();
                    // Warn main app we're all done
                    if (_cbEnd) {
                        _cbEnd();
                    }

                    if(_rebootOnUpdate && !spiffs) {
                        ESP.restart();
                    }

                } else {
                    ret = HTTP_UPDATE_FAILED;
                    log_e("Update failed\n");
                }
            }
        } else {
            _lastError = HTTP_UE_SERVER_NOT_REPORT_SIZE;
            ret = HTTP_UPDATE_FAILED;
            log_e("Content-Length was 0 or wasn't set by Server?!\n");
        }
        break;
    case HTTP_CODE_NOT_MODIFIED:
        ///< Not Modified (No updates)
        ret = HTTP_UPDATE_NO_UPDATES;
        break;
    case HTTP_CODE_NOT_FOUND:
        _lastError = HTTP_UE_SERVER_FILE_NOT_FOUND;
        ret = HTTP_UPDATE_FAILED;
        break;
    case HTTP_CODE_FORBIDDEN:
        _lastError = HTTP_UE_SERVER_FORBIDDEN;
        ret = HTTP_UPDATE_FAILED;
        break;
    default:
        _lastError = HTTP_UE_SERVER_WRONG_HTTP_CODE;
        ret = HTTP_UPDATE_FAILED;
        log_e("HTTP Code is (%d)\n", code);
        break;
    }

    http.end();
    return ret;
}

runUpdate

/**
 * write Update to flash
 * @param in Stream&
 * @param size uint32_t
 * @param md5 String
 * @return true if Update ok
 */
bool HTTPUpdate::runUpdate(Stream& in, uint32_t size, String md5, int command, const char* partitionLabel)
{

    StreamString error;

    if (_cbProgress) {
        Update.onProgress(_cbProgress);
    }

    if(!Update.begin(size, command, _ledPin, _ledOn, partitionLabel)) {
        _lastError = Update.getError();
        Update.printError(error);
        error.trim(); // remove line ending
        log_e("Update.begin failed! (%s)\n", error.c_str());
        return false;
    }

    if (_cbProgress) {
        _cbProgress(0, size);
    }

    if(md5.length()) {
        if(!Update.setMD5(md5.c_str())) {
            _lastError = HTTP_UE_SERVER_FAULTY_MD5;
            log_e("Update.setMD5 failed! (%s)\n", md5.c_str());
            return false;
        }
    }

// To do: the SHA256 could be checked if the server sends it

    if(Update.writeStream(in) != size) {
        _lastError = Update.getError();
        Update.printError(error);
        error.trim(); // remove line ending
        log_e("Update.writeStream failed! (%s)\n", error.c_str());
        return false;
    }

    if (_cbProgress) {
        _cbProgress(size, size);
    }

    if(!Update.end()) {
        _lastError = Update.getError();
        Update.printError(error);
        error.trim(); // remove line ending
        log_e("Update.end failed! (%s)\n", error.c_str());
        return false;
    }

    return true;
}

[zekageri]

The problem with this is that in the rest of the handleUpdate calls we have to modify from that

handleUpdate(http, currentVersion, true, requestCB);

to that

handleUpdate(http, currentVersion, true, NULL, requestCB);