tobychui
[BUG] Memory overflow with false forward-auth redirect (Authelia)
Describe the bug Zoraxy consumed the complete RAM of the Docker LXC (8GB) running in Proxmox and basically making Docker unresposive. The issue is not related to FASTGEOIP
The culprit what is causing this behavior seems to be a wrong setup of AUTHELIA forward auth.
Running HTOP in the docker host shows that memory consumption of Zoraxy is raising rapidly. If not stopping Zoraxy manually it would consume the full memory within a couple minutes. (see screenshot)
When this occurs the browser window keeps loading and will not end. Closing the browser window will not stop memory increase.
It can be stopped by a) shutting down the container b) changing entries to something that will not cause a loop and reopening the host (not 100% sure but this seemed to have stopped it)
Authelia log show such entries when this occurs:
time="2025-09-10T17:43:19+02:00" level=error msg="An unknown error occurred while handling a request from client." error="error when reading request headers: unsupported http request method "\x16\x03\x01\x00\xf6\x01\x00\x00\xf2\x03\x03)\x92\xf0/겵\xc4p\x10DJ{\x84\xe9o\xa6\x88\xcf%\xe4q\xe0\x16\r\xe5c{\x84\xadDY" in "\x16\x03\x01\x00\xf6\x01\x00\x00\xf2\x03\x03)\x92\xf0/겵\xc4p\x10DJ{\x84\xe9o\xa6\x88\xcf%\xe4q\xe0\x16\r\xe5c{\x84\xadDY \xff\xebʼ[\xbac\u05fca\xf5ݡX\xb3\xfb\xe12i\\x96\\xb0\\x97\\x89k\\xf8m\\xa0\\xab\\xdd\\xf9 \\x00\\x1c\\xc0+\\xc0/\\xc0,\\xc00̨̩\\xc0\\t\\xc0\\x13\\xc0\\n\\xc0\\x14\\xc0\\x12\\x13\\x01\\x13\\x02\\x13\\x03\\x01\\x00\\x00\\x8d\\x00\\v\\x00\\x02\\x01\\x00\\xff\\x01\\x00\\x01\\x00\\x00\\x17\\x00\\x00\\x00\\x12\\x00\\x00\\x00\\x05\\x00\\x05\\x01\\x00\\x00\\x00\\x00\\x00\\n\\x00\\n\\x00\\b\\x00\\x1d\\x00\\x17\\x00\\x18\\x00\\x19\\x00\\r\\x00\\x1a\\x00\\x18\\b\\x04\\x04\\x03\\b\\a\\b\\x05\\b\\x06\\x04\\x01\\x05\\x01\\x06\\x01\\x05\\x03\\x06\\x03\\x02\\x01\\x02\\x03\\x00\\x10\\x00\\x0e\\x00\\f\\x02h2\\bhttp/1.1\\x00+\\x00\\x05\\x04\\x03\\x04\\x03\\x03\\x003\\x00&\\x00$\\x00\\x1d\\x00 \\x8a\xa6\x15\x13\x97\t:\xd7\x04i\xc8<O\xa2\b\xaf\x1cT!jf\xa7\xb2(\x95f\xca\xc2\xefS|". Buffer size=251, contents: "\x16\x03\x01\x00\xf6\x01\x00\x00\xf2\x03\x03)\x92\xf0/겵\xc4p\x10DJ{\x84\xe9o\xa6\x88\xcf%\xe4q\xe0\x16\r\xe5c{\x84\xadDY \xff\xebʼ[\xbac\u05fca\xf5ݡX\xb3\xfb\xe12i\\x96\\xb0\\x97\\x89k\\xf8m\\xa0\\xab\\xdd\\xf9 \\x00\\x1c\\xc0+\\xc0/\\xc0,\\xc00̨̩\\xc0\\t\\xc0\\x13\\xc0\\n\\xc0\\x14\\xc0\\x12\\x13\\x01\\x13\\x02\\x13\\x03\\x01\\x00\\x00\\x8d\\x00\\v\\x00\\x02\\x01\\x00\\xff\\x01\\x00\\x01\\x00\\x00\\x17\\x00\\x00\\x00\\x12\\x00\\x00\\x00\\x05\\x00\\x05\\x01\\x00\\x00\\x00\\x00\\x00\\n\\x00\\n\\x00\\b\\x00\\x1d\\x00\\x17\\x00\\x18\\x00\\x19\\x00\\r\\x00\\x1a\\x00\\x18\\b\\x04\\x04\\x03\\b\\a\\b\\x05\\b\\x06\\x04\\x01\\x05\\x01\\x06\\x01\\x05\\x03\\x06\\x03\\x02\\x01\\x02\\x03\\x00\\x10\\x00\\x0e\\x00\\f\\x02h2\\bhttp/1.1\\x00+\\x00\\x05\\x04\\x03\\x04\\x03\\x03\\x003\\x00&\\x00$\\x00\\x1d\\x00 \\x8a\xa6\x15\x13\x97\t:\xd7\x04i\xc8<O\xa2\b\xaf\x1cT!jf\xa7\xb2(\x95f\xca\xc2\xefS|"" method=GET path=/ remote_ip=192.168.188.165 status_code=400
Authelia log from that time frame: (snippet) - not sure if related to the error
[2025-09-10 09:17:43.694066] [proxy] [system:error] Failed to assign an upstream for this request: no online upstream is available for origin: zoraxy.mydomain.com [2025-09-10 09:17:43.694079] [router:subdomain-http] [origin:] [client: 192.168.188.121] [useragent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:142.0) Gecko/20100101 Firefox/142.0] GET /api/stats/netstatgraph?array=true 521 [2025-09-10 09:17:44.698401] [router:subdomain-http] [origin:] [client: 192.168.188.121] [useragent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:142.0) Gecko/20100101 Firefox/142.0] GET /api/stats/netstatgraph?array=true 521 [2025-09-10 09:17:44.698421] [proxy] [system:error] Failed to assign an upstream for this request: no online upstream is available for origin: zoraxy.mydomain.com [2025-09-10 09:17:45.704564] [proxy] [system:error] Failed to assign an upstream for this request: no online upstream is available for origin: zoraxy.mydomain.com [2025-09-10 09:17:45.704590] [router:subdomain-http] [origin:] [client: 192.168.188.121] [useragent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:142.0) Gecko/20100101 Firefox/142.0] GET /api/stats/netstatgraph?array=true 521 [2025-09-10 09:17:46.774949] [router:subdomain-http] [origin:] [client: 192.168.188.121] [useragent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:142.0) Gecko/20100101 Firefox/142.0] GET /api/stats/netstatgraph?array=true 521 [2025-09-10 09:17:46.774973] [proxy] [system:error] Failed to assign an upstream for this request: no online upstream is available for origin: zoraxy.mydomain.com [2025-09-10 09:17:47.624789] [proxy] [system:error] Failed to assign an upstream for this request: no online upstream is available for origin: zoraxy.mydomain.com [2025-09-10 09:17:47.624812] [router:subdomain-http] [origin:] [client: 192.168.188.121] [useragent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:142.0) Gecko/20100101 Firefox/142.0] GET /api/stats/summary?fast=true 521 [2025-09-10 09:17:47.790836] [proxy] [system:error] Failed to assign an upstream for this request: no online upstream is available for origin: zoraxy.mydomain.com [2025-09-10 09:17:47.790863] [router:subdomain-http] [origin:] [client: 192.168.188.121] [useragent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:142.0) Gecko/20100101 Firefox/142.0] GET /api/stats/netstatgraph?array=true 521 [2025-09-10 09:17:48.796601] [proxy] [system:error] Failed to assign an upstream for this request: no online upstream is available for origin: zoraxy.mydomain.com [2025-09-10 09:17:48.796629] [router:subdomain-http] [origin:] [client: 192.168.188.121] [useragent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:142.0) Gecko/20100101 Firefox/142.0] GET /api/stats/netstatgraph?array=true 521 [2025-09-10 09:17:49.801942] [proxy] [system:error] Failed to assign an upstream for this request: no online upstream is available for origin: zoraxy.mydomain.com [2025-09-10 09:17:49.801971] [router:subdomain-http] [origin:] [client: 192.168.188.121] [useragent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:142.0) Gecko/20100101 Firefox/142.0] GET /api/stats/netstatgraph?array=true 521 [2025-09-10 09:17:50.808508] [router:subdomain-http] [origin:] [client: 192.168.188.121] [useragent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:142.0) Gecko/20100101 Firefox/142.0] GET /api/stats/netstatgraph?array=true 521 [2025-09-10 09:17:50.808562] [proxy] [system:error] Failed to assign an upstream for this request: no online upstream is available for origin: zoraxy.mydomain.com [2025-09-10 09:17:50.870362] [proxy] [system:error] Failed to assign an upstream for this request: no online upstream is available for origin: zoraxy.mydomain.com
To Reproduce Steps to reproduce the behavior:
- Configure forward auth address leaving out port and /api/ portion of the path: -> https://auth.mydomain.com/authz/forward-auth
- "HTTP Proxy" tab click on a protected host
Expected behavior A wrong configuration should result in a comprehensive error message and shut down the process.
Screenshots
(I stopped it before it completely filled up the RAM making the system unresponsive)
Browser (if it is a bug appears on the UI section of the system):
- OS: Windows 11
- Browser: Firefox
- Version 142.0.1
Host Environment (please complete following information, DO NOT REMOVE ANY FIELD(S)):
Host System: Proxmox: 9.0.6
System: Kernel: 6.14.11-1-pve arch: x86_64 bits: 64 compiler: gcc v: 14.2.0 Console: pty pts/0 Distro: Debian GNU/Linux 13 (trixie) Machine: Type: Desktop Mobo: Trigkey model: Green G4 v: 10 serial: N/A UEFI: American Megatrends LLC. v: ADLNV105 date: 12/14/2023 CPU: Info: quad core model: Intel N100 bits: 64 type: MCP arch: Alder Lake rev: 0 cache: L1: 384 KiB L2: 2 MiB L3: 6 MiB Speed (MHz): avg: 3380 min/max: 700/3400 cores: 1: 3380 2: 3380 3: 3380 4: 3380 bogomips: 6451 Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx Graphics: Device-1: Intel Alder Lake-N [UHD Graphics] driver: i915 v: kernel arch: Xe bus-ID: 00:02.0 API: EGL v: 1.5 drivers: iris,swrast platforms: active: gbm,surfaceless,device inactive: wayland,x11 API: OpenGL v: 4.6 compat-v: 4.5 vendor: mesa v: 25.0.7-2 note: console (EGL sourced) renderer: Mesa Intel Graphics (ADL-N), llvmpipe (LLVM 19.1.7 256 bits) Info: Tools: api: eglinfo,glxinfo x11: xdriinfo, xdpyinfo, xprop, xrandr Audio: Device-1: Intel Alder Lake-N PCH High Definition Audio driver: snd_hda_intel v: kernel bus-ID: 00:1f.3 API: ALSA v: k6.14.11-1-pve status: kernel-api Network: Device-1: Intel CNVi: Wi-Fi driver: iwlwifi Device-2: Realtek RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet driver: r8169
- Are you using Docker? Yes
- Docker version 28.4.0, build d8eb465
Additional context Authelia Config: (running outside docker in a Proxmox LXC)
Network setup:
Internal: OPNsense -> Adguard (on OPNsense) -> Unbound DNS (on OPNsense) Adguard forwards *.mydomain.com to Zoraxy
External: Cloudflare tunnel to cloudflared LXC on Proxmox forwarding all "*.mydomain.com requests to https://zoraxy.mydomain.com:443"
theme: dark
server.address: tcp://0.0.0.0:9091
log:
level: trace
file_path: /etc/authelia/authelia.log
totp:
issuer: .mydomain.com
period: 30
skew: 1
authentication_backend:
file:
path: /etc/authelia/users.yml
watch: true
access_control:
default_policy: one_factor
rules:
- domain:
- "auth.mydomain.com"
policy: bypass
- domain: # Proxies only requiring username and password
- "glance.mydomain.com"
- "proxmox.mydomain.com"
policy: one_factor
session:
name: authelia_session
secret: <snippet>
same_site: lax
inactivity: 5m
expiration: 1h
remember_me: 1M
cookies:
- domain: .mydomain.com
authelia_url: https://auth.mydomain.com
regulation:
max_retries: 5
find_time: 2m
ban_time: 10m
storage:
encryption_key: <snippet>
local:
path: /etc/authelia/db.sqlite
identity_validation:
reset_password:
jwt_secret: <snippet>
jwt_lifespan: 5 minutes
jwt_algorithm: HS256
notifier:
filesystem:
filename: /etc/authelia/emails.txt
@tobychui :As requested here a new bugreport for this issue as first posted in https://github.com/tobychui/zoraxy/issues/52#issuecomment-3274531531
BTW, I am still struggling with setting up Authelia, is there any working guide for this? (I have a working instance in combination with Nginx Proxy Manager but fail to get it working with Zoraxy)
I have a question, do you have zoraxy set to authenticate all requests via authelia, then have authelia routed through zoraxy as well?
@FirebladeBMW I am suspecting two modules that might causes this problem, since the request will pass through the forward-auth / sso module first before going into the dpcore, there are possible that either the forward auth redirect implementation caused a redirect loop (between zoraxy and external service maybe?) or there is something wrong with the redirection rewrite in dpcore (which after so many revisions in previous releases, I think this is less likely to happen).
I will label this as forward-auth and see if anyone can figure out something.
I have a question, do you have zoraxy set to authenticate all requests via authelia, then have authelia routed through zoraxy as well?
Not sure if I understand your question correctly but let me try to answer:
I haven't done any other settings in Zoraxy to route all traffic automatically to Authelia. (Not sure how I would do that. Is there a setting?) Authelia Proxy itself is setup by IP address.
All my settings can be seen here: https://github.com/tobychui/zoraxy/issues/793#issuecomment-3280259175
I am still trying to connect to my authelia instance and did not assign Forward Auth to other proxy hosts.
@FirebladeBMW I am suspecting two modules that might causes this problem, since the request will pass through the forward-auth / sso module first before going into the dpcore, there are possible that either the forward auth redirect implementation caused a redirect loop (between zoraxy and external service maybe?) or there is something wrong with the redirection rewrite in dpcore (which after so many revisions in previous releases, I think this is less likely to happen).
I will label this as forward-auth and see if anyone can figure out something.
Just to highlight, this happens if you put in the URL to the Forward Auth incorrectly. (I accidentially left out the port and the /api/ part of the URI path.
So, I would suspect, that the request does not even hit the auth but ends up somewhere in nirvana.
0x16 0x03 0x01is the start of a TLS handshake when Authelia is listening on plain TCP. This means the client is making TLS request when it shouldn't be. As far as why this causes memory usage I'm not entirely sure.
I have the same issue with memory exhaustion and I don't use any auth capabilities. My installation has only some proxy rules, stream rules and certificates. I migrated my setup from nginx proxy manager where it was running on a VM with 2GB memory. Now I have zoraxy with 4GB and after 2-3 days the whole memory is exhausted and the VM starts to heavily swapping. Restarting zoraxy (docker compose restart) fixes the issue to re-appear after 2-3 days again. I get around 20k requests per day, same with nginx proxy manger who was running fine with half the memory.
@baskinsy I don't think your issue is related to this issue but try to run it natively in your VM. Zoraxy is designed to run natively on all platforms and in theory, you don't need docker to run Zoraxy.
@tobychui I prefer to run it with docker for easy maintenance and it seems at least with this setup it is memory leaking. Will monitor it and maybe migrate to native install if I find the time to do it... just migrated from nginx proxy manager and it is a lot of work to migrate again for me... Thanks in any case.
@baskinsy Ok, but just to let you know that I also got a few nodes running Zoraxy on my side and it doesn't seems to have memory leak issue at all when running natively. I am suspecting your issue is related to your particular setup if it is not a docker related issue.
@tobychui it is just a standard Ubuntu 24.04 host with docker 28.2.2 provided by Ubuntu packages and the default compose file. The only difference is that I run the zoraxy container in "host mode" so I don't have to alter the compose file for every stream I add. Will update if I manage to find something.