Provide alternate http URL for Reolink camera callback

[ex-nerd]

The problem

Reolink cameras apparently need an http (not https) URL for ONVIF callbacks. However, my home-assistant server is configured with https … additionally, the cameras are in a DMZ that does not provide access to the DNS name that I use for "internal network" to allow my phones, etc. to access my Home Assistant server.

Please provide an advanced configuration option so that we can provide a callback URL for the integration to use with the Reolink cameras. Ideally, this form would look something like http://[ hostname and port input ]/callback-path so that users could also be informed about the callback path, in case http proxy rules need to be configured to ensure the cameras are not given access to any other sections of Home Assistant.

What version of Home Assistant Core has the issue?

core-2023.4.5

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant Container

Integration causing the issue

reolink

Link to integration documentation on our website

https://www.home-assistant.io/integrations/reolink/

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

Also tracking this conversation here: https://community.home-assistant.io/t/how-to-solve-reolink-webhook-url-uses-https-ssl-error/561934

[home-assistant[bot]]

Hey there @starkillerog, mind taking a look at this issue as it has been labeled with an integration (reolink) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of reolink can trigger bot actions by commenting:

• @home-assistant close Closes the issue.
• @home-assistant rename Awesome new title Renames the issue.
• @home-assistant reopen Reopen the issue.
• @home-assistant unassign reolink Removes the current integration label and assignees on the issue, add the integration domain after the command.

_{^{(message by CodeOwnersMention)}}

---

reolink documentation reolink source _{^{(message by IssueLinks)}}

[ex-nerd]

Even a yaml-only config option to override the host/IP would be great.

[starkillerOG]

See the discussion in this PR that would have implemented this: https://github.com/home-assistant/core/pull/89625 And also the discussion here: https://github.com/starkillerOG/home-assistant/commit/1f21a9b1ce0863659d27f461b323b4bcfb0fad42#commitcomment-99480652

As @frenck decided in that PR that was rejected:

Home Assistant doesn't support multiple Homes or advanced network scenarios, nor do we have the intention to do so at this point. We should, most certainly, not resolve such cases in each integration individually. That is out of scope for our project.

Bottom line: these kinds of advanced network configurations are out of scope for the HomeAssistant project as decided by the core team, therefore I am not allowed to add such options. You will have to use HTTP (no SSL) for your internal connection to HomeAssistant. You can use HTTPS (SSL) for your external connection using something like NGINX proxy manager.

As for the webhook path used by this integration you can find it in the source code (I know advanced user, but these kinds of network setups are not for regular users): https://github.com/home-assistant/core/blob/a511e7d6bcc88cebb6abd15083a06574e84c855c/homeassistant/components/reolink/host.py#L335-L336 https://github.com/home-assistant/core/blob/a511e7d6bcc88cebb6abd15083a06574e84c855c/homeassistant/components/webhook/init.py#L31 https://github.com/home-assistant/core/blob/a511e7d6bcc88cebb6abd15083a06574e84c855c/homeassistant/components/reolink/host.py#L316

So it would look something like this: "http://<192.168.1.IP>:8123/api/webhook/reolink_<MAC>_ONVIF" Where <192.168.1.IP> is the IP adress of your HA instance (taken from the local network URL), and <MAC> is the mac of the camera withouth ":"

[starkillerOG]

If you appreciate the reolink integration and want to support its development, please consider sponsering the upstream library or purchase Reolink products through this affiliate link.

[ex-nerd]

Modifying the code is unmaintainable, esp. with HA-OS or Docker on a regular auto-update. Is it really that much work to add a couple lines somewhere around https://github.com/home-assistant/core/blob/dev/homeassistant/components/reolink/host.py#L324 to just pull something from the global conflg like:

reolink:
    webhook_host_override: 192.168.100.1

Or is this the kind of thing the core team doesn't allow?

(something like this would be easy to set up on a reverse proxy to let the cameras speak https with nginx proxy handing the http->https internally)

My problem isn't just internal/external. My cameras are in a DMZ. They are external as far as the camera's own network stack. The internal URL won't work, even if it's http, because the DNS routes to an external IP that won't have http enabled.

Hmm, I even tried to disable https for the internal URL now and get:

Invalid local network URL You have configured an HTTPS certificate in Home Assistant. This means that your internal URL needs to be set to a domain covered by the certficate.

It looks like if I'm going to make this change I need to put the whole server behind a reverse proxy.

Anyway, thanks for the explanation. I'll keep poking around, but for now it looks like I'll just have to stick with the more rudimentary webhooks I already have set up through synology surveilance station.

[starkillerOG]

YAML config is not allowed at all anymore by core team. The options flow is easy (see PR linked above), but was rejected by core team as described above.

Can't you just move the cams to the VLAN of HomeAssistant, or move them to a seperate VLAN, but allow them to acces HA bidirectionally but not the internet?

Yes you need to remove the global SSL certificate and put everything behind a reverse proxy (NGINX proxy manger).

[ex-nerd]

YAML config is not allowed at all anymore by core team.

Ah, thanks for the clarification. That's unfortunate for edge case overrides.

Playing with VLANs gets messy, especially with wifi cameras and different DNS requirements. I'll figure something out. Just seemed weird that there wasn't a way to override values, especially for IoT devices that many of us prefer to keep completely separate from any internal trusted network.

Anyway, thanks for all of the details. Hopefully this info and the forum post will be enough to save other people at least a bit of frustration trying to track this down. Not sure how much help it would be, but it would be helpful to add a note/explanation about the http limitation for HA itself in the documentation.

[kevdliu]

I was actually just about to open my own PR to implement this exact feature, glad I thought to search before having the PR rejected 🤣

That's a very disappointing comment from frenck. I have my home assistant to always serve over HTTPS as I do not trust my LAN as a totally secure network, with all the IoT and other wifi connected devices running on it. Even though setting up Home Assistant to serve over HTTPS might be considered advanced usage it is more secure so it's sad to see that we can't workaround the reolink HTTPS issue.

I guess I'll just have to live with my modified reolink integration and remember to pull every time there's a change. Nevertheless @starkillerOG thank you for trying to implement these requested features so quickly!

Edit: @starkillerOG do you know if the cameras fail to send to https endpoints because it doesn't support https endpoints at all or because the endpoints have self-signed certs?

[craiggenner]

I have this exact issue @kevdliu and worked around it by patching the code to use http and putting a HTTP->HTTPS proxy in place for just the webhooks.

At best this is an unsuitable approach. At worst it 100% unmaintainable.

The core problem is that the firmware of the cameras needs to connect to a HTTP endpoint. While this is understandable from the view point of the resources on the camera, it's not in keeping with modern security practices either.

This is also true of the Home Assistant stance. I understand the response from frenck, but it's also short sighted to assume that people will only run HA on HTTP internally. I wouldn't say HTTPS internally isn't "an advanced home", but given the problems people have with certificates I can certainly see where frenck is coming from.

The only way I can think to resolve this is to have an 'advanced' (and also unsupported) version of the the Reolink integration in something like HACS that has these fixes/features/etc. That itself has problems maintaining two sets of code.

Either way we (the users of HA and buyers of the cameras) are stuffed as it currently stands.

Is there any ways of getting Reolink to do something in their firmware?

[kevdliu]

I have this exact issue @kevdliu and worked around it by patching the code to use http and putting a HTTP->HTTPS proxy in place for just the webhooks.

At best this is an unsuitable approach. At worst it 100% unmaintainable.

The core problem is that the firmware of the cameras needs to connect to a HTTP endpoint. While this is understandable from the view point of the resources on the camera, it's not in keeping with modern security practices either.

This is also true of the Home Assistant stance. I understand the response from frenck, but it's also short sighted to assume that people will only run HA on HTTP internally. I wouldn't say HTTPS internally isn't "an advanced home", but given the problems people have with certificates I can certainly see where frenck is coming from.

The only way I can think to resolve this is to have an 'advanced' (and also unsupported) version of the the Reolink integration in something like HACS that has these fixes/features/etc. That itself has problems maintaining two sets of code.

Either way we (the users of HA and buyers of the cameras) are stuffed as it currently stands.

Is there any ways of getting Reolink to do something in their firmware?

Yeah I'm planning on at some point to fork just the reolink component and periodically pull changes made to the upstream component. We could try submitting support requests to Reolink to add HTTPS support to the cameras, but we'll still have the problem with the self-signed CA, unless they also add support for uploading our own self-signed CA to the camera's CA store. Though at least with HTTPS support we can have the cameras go through nabu casa or duckdns. It's not ideal as it'll route through the internet but at least it'll function.

[craiggenner]

@kevdliu Look at https://github.com/home-assistant/core/pull/91137, this MR should provide a workaround for the HTTP/HTTPS issue in the next release (assuming it's not reverted for some reason).

Meanwhile I tried to get something added to documentation so that others were aware of this issue, but I err... ran into issues doing this. You can see my attempt at https://github.com/home-assistant/home-assistant.io/pull/27119

[magtimmermans]

Same issue here. Hopefully it can be solved.

[bdraco]

If the reolink cameras support pull point that could be one solution.

Onvif uses a dual pull point / webhook implementation https://github.com/home-assistant/core/blob/ac21f69ad0b85bff0a5c5d00e54e4e65c290ec31/homeassistant/components/onvif/event.py#L226 that falls back to pull point if webhooks are not available (is https)

[bdraco]

I tested two reolink cameras (The e1 pro and the doorbell). They both support pullpoint so this should be a good alternative if webhooks aren't an option.

You could also use them with the onvif integration

[starkillerOG]

@bdraco thank you for the suggestion, I was already thinking about something simular. Actually the motion/AI state are already polled every 60 seconds (in the big poll of the complete camera state) as a backup using the HTTP API. Polling over the HTTP API is I think faster and the messages contains less data (JSON) than using the ONVIF pullpoint (XML with lots of useless stuff), but I am aware that Reolink cameras do support ONVIF pullpoint.

I am thinking about polling only the motion/AI state over the HTTP API (takes less than 0.3 seconds) with a polling intervall of 2 seconds as long as no ONVIF push message has been received. Once the first ONVIF push messages comes in, I would stop polling since then I have the confirmation that push is working.

[starkillerOG]

Edit: @starkillerOG do you know if the cameras fail to send to https endpoints because it doesn't support https endpoints at all or because the endpoints have self-signed certs?

@kevdliu I think it fails because the cameras dont support HTTPS endpoints at all.

Is there any ways of getting Reolink to do something in their firmware?

@craiggenner you can request supporting HTTPS endpoints in the firmware of reolink through their support form: https://support.reolink.com/hc/en-us/requests/new/

[Schnabulation]

Same setup as op: Cameras in DMZ, external URL only via HTTPS. I am using this custom integration and it works flawlessly: https://github.com/JimStar/reolink_cctv

[starkillerOG]

@Schnabulation indeed JimStar/reolink_cctv will allow you to configure a custom HA URL used by the integration. However it is missing some of the newer features/entities that the build-in integration has (especially related to the reolink doorbell) and is missing a couple of important bug-fixes. So the build-in integration will probably be a lot more reliable.

[starkillerOG]

@bdraco I just implemented your suggestion for a dual polling/webhook subscription approach in which polling is used until the first push message is received and therefore push is confirmed to be working. The PR is here: https://github.com/home-assistant/core/pull/92837, would you be so kind to have a look?

[Markus9955]

I've got the same issue with the callback. Home assistant is connected to two VLANs, one that is internet connected where the HA is on 192.168.1.201 and another that has the cameras connected to it with no internet access on 192.168.3.201.

The default call back points to 192.168.1.201 so I have edited the code to point it at 192.168.3.201 and have concerns about maintainability which is what brought me here.

Looking forward to giving the proposed solutions a test

[joneshf]

@starkillerOG I noticed that https://github.com/home-assistant/core/pull/92837 was merged, but it doesn't look like it made it into the 2023.5.3 release, and it doesn't look like it's in the upcoming 2023.5.4 release. I mention the 2023.5.3 release because https://github.com/home-assistant/core/pull/92979 was merged around the same time, and it made it in the 2023.5.3 release. Is there more that needs to be done before https://github.com/home-assistant/core/pull/92837 can be released, or did it get missed?

[starkillerOG]

@joneshf No, https://github.com/home-assistant/core/pull/92837 is not a small bugfix but a larger change, therefore it will not be part of the patch releases. It will be in Home Assistant 2023.6.0. So unfortunatelly you will have to wait a bit for it to be available.

[joneshf]

Gotcha. Thanks for the explanation!

[joneshf]

Just came by to say that this landed in 2023.6.0, like you said. It was the final nudge I needed to move from the HACS integration to this integration. Thanks for providing a solution to this problem!

[starkillerOG]

@joneshf glad to hear that. Note that you will suffer a delay of up to 5 seconds when ONVIF WS base notifications are not working and you fall back on polling (every 5 seconds).

I am working on implementing ONVIF long polling as a intermidiate fallback which should allow for push like behaviour whithouth delays.

[starkillerOG]

If you appreciate the reolink integration and want to support its development, please consider sponsering the upstream library or purchase Reolink products through this affiliate link.

[Schnabulation]

I seem to not understand: I thought this issue is closed? Yet, I still can't add my Reolink cameras in another VLAN. Is there a config adjustment that I missed?

[ex-nerd]

@Schnabulation Even before this fix, my cameras were all auto-detected (home assistant vlan can "see into" the camera vlan, and related traffic then allowed back) and added. Lots of warnings about https, but since this latest release the polling code does update the sensors for person detection, etc. It's just slightly slow. From the sound of things, there is future fix (whatever ONVIF long polling actually entails) that will basically cut that delay to zero.

[Schnabulation]

Thank you for your reply - I was also able to make it work right now.

Update: I think the issue is because I'm trying to switch from the HACS integration to the stock one. When I delete the camera in the HACS integration and try to add it in the stock integration I always get an error "Handler is already defined!"

[starkillerOG]

@Schnabulation please first completly remove the custom integration and remove the config entry for the reolink integration. Make sure there is nothing left of reolink in the custom_components dir (inside your config dir). Then reboot Home Assistant. Only then attempt to configure the build in reolink integration.