home-assistant
Ingress proxy unpacks assest and work slowly
Describe the issue you are experiencing
Hello from zigbee2mqtt team!
Recently many users started complaining that z2m frontend open slowly.
https://github.com/zigbee2mqtt/hassio-zigbee2mqtt/issues/236 https://github.com/nurikk/zigbee2mqtt-frontend/issues/1124
After some debugging i can see that when frontend is loaded through ingress proxy, it's loaded very slowly: there are few symptoms:
- Assets aren't gzip compressed
- Missing caching headers
- Sometimes response comes in form of headers from underlying backend as is, kind of http response parsing issue
Here're some screenshots
This how it looks if we request frontend directly, without ingress First load (no cache)
After refresh
What is the used version of the Supervisor?
supervisor-2021.10.8
What type of installation are you running?
Home Assistant Supervised
Which operating system are you running on?
Debian
What is the version of your installed operating system?
Ubuntu 18.04.6 LTS
What version of Home Assistant Core is installed?
core-2021.11.5
Steps to reproduce the issue
- Install zigbee2mqtt addon
- Start it
- Open web ui
Anything in the Supervisor logs that might be useful for us?
no
Additional information
Attached HAR archive from chrome dev tools
Current workaround is to use websocat exported port for frontend and access directly on websocat port
right now the Z2M frontend looks like this every second time. If you press reload, everything is fine...
@frenck Hi, I am sorry that I "quoted" you directly in that issue, but I saw in the last HA shows that you use Z2M yourself. There is an issue with the frontend and ingress (which seems to be a part of HA supervisor). Maybe you face the problem yourself (you need to reload the frontend sometimes that it shows correctly) @nurikk made this issue about it but no one seems to pick it up. Do you have any hint how to proceed? I saw problems with ingress when HA started using it but now I am not sure if there is somebody who will cover the ingress part...
There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.
There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.
There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.
There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.
There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.
@balloob Hello Paul, normally HA issues are assigned by a bot, but here for the supervisor this doesn't seem to work. I don't know what to do, just keeping this issue open. Is there a better way or should i let the bot close the issue, as nobody seems to be able to take a look? Thx for a small hint about that as I am clueless here...
There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.
Hey sorry for the delay on this one @Bascht74 . I'm digging into this now.
So first thing I noticed is that the cache-control header seems to be the same for both ingress and non-ingress connections to Z2M. And in my testing just now I do actually see Z2M hitting the cache when using ingress after the first refresh. I realize that its been a while since this bug was first reported, do you see that change as well? Or are you still seeing Z2M skipping the cache?
Dev tools for Z2M direct access on second refresh with cache allowed:
Dev tools for Z2M ingress access on second refresh with cache allowed:
Now for the other two parts. The response actually gets unpacked twice along the way to the client unfortunately. Since ingress everything from the HA frontend (including ingress requests) go to core first. Core then sees it is an ingress request and so passes it to supervisor. Supervisor then checks that it has a valid ingress session and if so passes it on to the addon. The process then runs in reverse for the response.
The request handling makes sense but the response handling is a bit illogical. Core obviously has to process the response since it is often asked to be the TLS endpoint by users when they add their SSL certs to its HTTP config. But supervisor never has to handle SSL and is basically just pass through. When I asked about this it seems there's a limitation in aiohttp around this. We let aiohttp handle encoding and apparently it always decodes response, there is no way to pass a response straight through without decoding it. I can research that more as that seems like the obvious improvement, perhaps something has changed.
Since supervisor currently has to decode the response it is currently leveraging that to decide whether to stream it in chunks or not: https://github.com/home-assistant/supervisor/blob/0991f52100b49b3e5abd33d095ef38a915630d2d/supervisor/api/ingress.py#L189-L219 I'm suspicious that this is the source of your 0B responses but I'm having a hard time reproducing that. Do you still see that behavior?
The thing that seems like the top priority here is why isn't the response gzipped? I definitely see exactly what you're talking about there. My understanding is that this should be handled by aiohttp. The request headers include the accepted encodings so it should be encoding the output in one of the accepted formats. That doesn't seem to be happening which is confusing, will look into why.
There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.