feat: improve OTA

[Nerivec]

Design changes

• https://github.com/Koenkk/zigbee-herdsman/issues/1608
• requires https://github.com/Koenkk/zigbee-herdsman/pull/1612/
• Update API to match new features
• Merge Zigbee/MQTT triggered update logic into single function for consistency
• Rework OTA support check (from definition) to allow custom file/URL to always go through (check/update/schedule)
• Add from/to file versions to update response payload to avoid weirdness with swBuildId/dateCode often unavailable
• Add latest_source (URL/filesystem), latest_release_notes to OTA state payload
• Allow passing data settings (timings/sizes) per request to override settings
• Tweak the published OTA state based on return of update (clear available if update returned "no image")
• Add ability to schedule with custom URL/filesystem
• Add ability to send hex-formatted image to Z2M which will be automatically written to data dir (for use with update or schedule)
  • Should allow using plain file over MQTT/WS assuming properly formatted upstream (e.g. frontend)
  • With limitation that MQTT/WS must allow payloads of sufficient size (though most cases should be <1MB)

Due to MQTT request payloads still allowing simple string for OTA, it duplicates the code a bit for now (marked deprecated for 3.0).

@Koenkk

• I'm thinking with this one, we might want to bump the settings version so we can trigger a migration to remove all cached OTA states?
• How do you prefer to deal with null source/release_notes in #getEntityPublishPayload? I'm hesitating between null or undefined so it's taken out on stringify.

TODO:

• [x] remove restart required from OTA data settings in schema
• [ ] change hex writes to ota subdir of data dir
• [ ] new tests for full coverage
• [ ] ZHC: cleanup
• [ ] docs for new features / MQTT API additions

[Koenkk]

I'm thinking with this one, we might want to bump the settings version so we can trigger a migration to remove all cached OTA states?

Agree

How do you prefer to deal with null source/release_notes in #getEntityPublishPayload? I'm hesitating between null or undefined so it's taken out on stringify.

I would propose to go for null (such that the attribute is always in the published payload)

[Nerivec]

@Koenkk can you take a closer look now that it's fully tested? In particular:

1. Left a few TODOs, not all quite related to this specifically.
2. I modified reInterview (previously only used if triggered by bridge request) to behave more like if it had been triggered from ZH Controller. Should provide better feedback and align the logic?
3. Image files as hex written to ota subdir of data dir, should be a bit cleaner. Files are auto-removed on unschedule, but not sure if we should also auto-remove on update end (tracing?)?

Note: we'll have to rebuild the all-settings page in docs for this one.

[Koenkk]

1. Left a few TODOs, not all quite related to this specifically.

Replied!

2. I modified reInterview (previously only used if triggered by bridge request) to behave more like if it had been triggered from ZH Controller. Should provide better feedback and align the logic?

Look good

3. Image files as hex written to ota subdir of data dir, should be a bit cleaner. Files are auto-removed on unschedule, but not sure if we should also auto-remove on update end (tracing?)?

Let's keep them for now, these OTAs are fairly small so shouldn't be a problem.

[Bjk8kds]

Thank you so much! Now it is easier to update custom firmware such as ZigbeeTlc @pvvx, etc.