[REQUEST] grab OTA files from https://github.com/Koenkk/zigbee-OTA/blob/master/index.json

[grischard]

@Koenkk maintains a nice convenient list of OTA firmwares at https://github.com/Koenkk/zigbee-OTA

Would it be possible to automatically support grabbing OTA updates from https://github.com/Koenkk/zigbee-OTA/blob/master/index.json ?

[Hedda]

FYI, there was a previous discussion about Zigbee device OTA firmware updates in https://github.com/zigpy/zigpy/issues/154 which might be relative or supersede this?

[grischard]

Not superseded, but definitely related indeed. It's good that the tickets are separate but linked.

[almico]

I tried to download images from https://github.com/Koenkk/zigbee-OTA/tree/master/images/Xiaomi and all of them turned to be invalid OTA files according to zigpy:

DEBUG (SyncWorker_2) [zigpy.ota.provider] File '/config/zigpy_ota/OTA_lumi.flood.agl02_V1.0.18_20190814.20191008104903.ota' doesn't appear to be a OTA image

Did I miss something?

[puddly]

They parse fine for me. What is the full error traceback?

[almico]

It should be this:

2021-06-29 05:24:11 DEBUG (SyncWorker_13) [zigpy.ota.provider] File '/config/zigpy_ota/1600167449-si32_zg_uart_connect_sleep_ZS5_ty_OTA_1.1.5.bin' doesn't appear to be a OTA image
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/zigpy/ota/image.py", line 262, in parse_ota_image
    return HueSBLOTAImage.deserialize(data)
  File "/usr/local/lib/python3.8/site-packages/zigpy/ota/image.py", line 201, in deserialize
    header, remaining_data = OTAImageHeader.deserialize(data)
  File "/usr/local/lib/python3.8/site-packages/zigpy/ota/image.py", line 112, in deserialize
    raise ValueError(
ValueError: Wrong magic number for OTA Image: 168430090

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/zigpy/ota/provider.py", line 302, in scan_image
    parsed_image, _ = parse_ota_image(f.read())
  File "/usr/local/lib/python3.8/site-packages/zigpy/ota/image.py", line 264, in parse_ota_image
    return OTAImage.deserialize(data)
  File "/usr/local/lib/python3.8/site-packages/zigpy/ota/image.py", line 162, in deserialize
    hdr, data = OTAImageHeader.deserialize(data)
  File "/usr/local/lib/python3.8/site-packages/zigpy/ota/image.py", line 112, in deserialize
    raise ValueError(
ValueError: Wrong magic number for OTA Image: 168430090

[puddly]

I think you downloaded the HTML files instead of the actual OTA images:

# Your file
In [11]: (168430090).to_bytes(4, 'little')
Out[11]: b'\n\n\n\n'

# GitHub HTML
In [12]: requests.get('https://github.com/Koenkk/zigbee-OTA/blob/master/images/Xiaomi/20201218113852_lumi.relay.c2acn01_0.0.0_0046_20201216_6BB0FD.ota').content[:4]
Out[12]: b'\n\n\n\n'

# File in the repo
In [13]: pathlib.Path('~/Downloads/OTA_lumi.flood.agl02_V1.0.18_20190814.20191008104903.ota').expanduser().read_bytes()[:4]
Out[13]: b'\x1e\xf1\xee\x0b'

[almico]

I think you downloaded the HTML files instead of the actual OTA images

AAAAAAAAAAARGHHHHHH! 😆

[Adminiuga]

But how do you update xiaomi? Do they have it's cluster?

[MattWestb]

Some is having it like my old 2 way switch:

[github-actions[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[grischard]

Still an issue, dear little robot.

[Hedda]

Bump this discussion now that ITead has submitted an official OTA firmware for one of their Sonoff brander Zigbee devices to Koenkk’s repository as primary source for download. See:

Originally posted by @Daniel-zhan-itead in https://github.com/zigpy/zha-device-handlers/issues/1519#issuecomment-1132783043

We have raised the pull request for the OTA update image on github https://github.com/Koenkk/zigbee-OTA/pulls. Once it is approved, you can download the image and update the ZBMINI-L devices.

[MattWestb]

Our ZHA user have already start updating there device with the firmware itead was submitted and its looks working OK in ZHA with local OTA files.

The Z2M have some very serious issues then applaying bad frirmware for tuya TRVs and many Xiaomi devices that was not tested and was hard braking user devices so i think it better letting the user downloading there OTA files and doing it by hand and not linking to one external source that can having bad firmware for our devices (as IKEA was braking our controllers but it was all production firmware (and some test feed) that was bad).

[Hedda]

FYI, OpenHAB's Zigbee developers and users have also been discussed the idea of getting Zigbee OTA firmware images from GitHub repository/repositories and specifically parsing https://github.com/Koenkk/zigbee-OTA/blob/master/index.json and grabbing OTA files from https://github.com/Koenkk/zigbee-OTA/tree/master/images

https://community.openhab.org/t/ikea-tradfri-firmware-updates/132042/63

And now the developer of OpenHAB's ZigBee Binding (who is also the developer of com.zsmartsystems.zigbee https://github.com/zsmartsystems/com.zsmartsystems.zigbee/ Zigbee Cluster Library Java framework/libraries it depends on) is apparently working on a somewhat similar GitHub Zigbee Firmware Provider backed by GitHub repository/repositories, using a GitHub Library Reader to parse "index.json" and will by default point to Koenkk's https://github.com/Koenkk/zigbee-OTA repo (that is also used by Zigbee2MQTT and IoBroker via zigbee-herdsman):

https://github.com/openhab/org.openhab.binding.zigbee/pull/767

This PR implements a FirmwareProvider for Zigbee devices. It gets the firmware files from the web and caches them locally for use based on requests from the device. The provider gets the firmware IDs from the ZigBeeThingHandler based on these periodic requests from the device, and provides them as a list so the user can allow the upgrade via the standard OH firmware management system.

Requires an updated ZSS 1.4.7 library to provide callback when the client sends a request. This is a little cleaner than the other approach of adding a ZclCommandListener as it better integrates in with the server. Also requires further testing.

[Hedda]

FYI, pipiche38 look to be working on a similar GitHub OTA Firmware Provider for same"index.json" in Zigbee Plugin for Domoticz:

https://github.com/zigbeefordomoticz/Domoticz-Zigbee/issues/1293

https://github.com/zigbeefordomoticz/Domoticz-Zigbee/pull/1294

@pipiche38 Could it not benefit all projects dependent on zigpy if you implemented that GitHub OTA Firmware Provider in zigpy?

https://github.com/zigpy/zigpy/tree/dev/zigpy/ota

https://github.com/zigpy/zigpy/blob/dev/zigpy/ota/provider.py

[pipiche38]

@pipiche38 Could it not benefit all projects dependent on zigpy if you implemented that GitHub OTA Firmware Provider in zigpy?

We are only relying on RAW messages. Going to OTA level, will require to got to ZDO and ZCL from zigpy which is not the approach we have taken, as at the same time we support native zigate communication.

Now, as firmware update is concerned, my personal opinion is that I'll never make thing smooth and transparent for the end user. I will make sure that a manual action is done , so the user is acting on that. We are not the firmware provider and we cannot take any responsibility if that fails.

[mdeweerd]

FYI. I added the zha-toolkit implementation that uses the index.json from Zigbee-OTA to a discussion https://github.com/zigpy/zigpy/discussions/654#discussioncomment-3775270 .

[Hedda]

FYI, puddly has just now added a new feature to the "zigpy-cli" tool that manufacturers can use to produce a skeleton JSON index from a given set of OTA files, and that way make sure manufacturers OTA files are correctly-structured so that they will be compatible with zigpy out-of-the-box. See the README.md for the "zigpy-cli" tool:

https://github.com/zigpy/zigpy-cli/blob/dev/README.md#ota

PS: puddly mentioned when adding that feature for generating OTA index files via zigpy-cli was done in preparation for some OTA rewrites in the main zippy library that he is planning to do.

[Hedda]

FYI, be sure to check out puddly's draft PR here which will add support for new open standard of Generic OTA providers -> https://github.com/zigpy/zigpy/pull/1165

Implements the format specified in https://github.com/zigpy/zigpy/wiki/OTA-Information-for-Manufacturers:

zha:
  zigpy_config:
    ota:
      remote_providers:
        - url: "https://fw.zigbee.example.org/ota.json"
          manufacturer_ids: [0x1234, 0x5678]
        - url: "https://fw.zigbee.example.org/ota-beta.json"

This allows for new providers to be added to zigpy without extra code. This also allows for manufacturers to distribute test feed URLs to customers without deploying them globally.

[github-actions[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[Hedda]

Now that PR https://github.com/zigpy/zigpy/pull/1165 has been merged adding support for Generic OTA providers, can the wanted feature be achieved if would generate a JSON index file with zigpy-cli (https://github.com/zigpy/zigpy-cli/blob/dev/README.md#generate-ota-index-files) for the OTA images that are hosted on GitHub in Koenkk's zigbee-OTA repository at https://github.com/Koenkk/zigbee-OTA/ and if so then maybe Koenkk could be convinced to host such an alternative index in the same GitHub repository? An additional index file to https://github.com/Koenkk/zigbee-OTA/blob/master/index.json that instead follow zigpy's new JSON index file format as documented here in zigpy's wiki -> https://github.com/zigpy/zigpy/wiki/OTA-Information-for-Manufacturers

[puddly]

We are going to go with a different approach for OTA and there are no plans to use the Z2M OTA repo at the moment. You're free to generate your own index file (or just clone it and use the file-based OTA provider).