tobychui
[ENHANCEMENTS] Enable HTTP/2 over cleartext TCP (h2c) support
The problem For full compatibility it would be nice if Zoraxy would support http2 communication with upstreams both via TLS (h2) and cleartext (h2c) as RFC 7540 outlines. Due to the lack of support from webbrowsers and the generally bad performance of it over wider networks, h2c support for traffic ingress from the outside is probably not needed.
The V3.1.7 release of Zoraxy already added undocumented experimental support for H2C transport between the proxy and the upstream.
However there seems to be no way to actually enabled this experimental feature for select upstreams or the server as a whole yet, despite the implementation seemingly being done (correct me if I am wrong here). This is afaik due to the UI not having been updated to support the new option.
Proposed solution If the feature is functional already, it would be cool to receive a quick UI update soon to enable it to actually test it out and report back with findings. Otherwise feel free to just ignore this issue until h2c support is properly working.
Further considerations I know this feature was already requested by @Morethanevil via email, but I felt like it would be good to have it tracked publicly too.
@DerLeole Didn't expect you discovered that, I intentionally leave it out in the change log and hope no one notice it XD
Yes, there is no way to enable it for now in the compiled binary. But if what you wanna do is test if this implementation works, you can change this line
if dpcOptions.UseH2CRoundTripper {
to the following.
if true{
This will force enable the h2c round tripper so you can test it on your upstream that requires h2c. Recently due to some issues, I am unable to spawn of testing VM for a while and I do not want to release not-tested code publicly. So it might need to take a while before you can actually use it in Zoraxy.
@DerLeole Didn't expect you discovered that, I intentionally leave it out in the change log and hope no one notice it XD
Yes, there is no way to enable it for now in the compiled binary. But if what you wanna do is test if this implementation works, you can change this line
if dpcOptions.UseH2CRoundTripper {
to the following.
if true{
This will force enable the h2c round tripper so you can test it on your upstream that requires h2c. Recently due to some issues, I am unable to spawn of testing VM for a while and I do not want to release not-tested code publicly. So it might need to take a while before you can actually use it in Zoraxy.
I just tried it, but Zoraxy crashes at startup:
Feb 10 14:35:01 mtex sudo[1973115]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Feb 10 14:35:01 mtex sudo[1973115]: pam_unix(sudo:session): session closed for user root
Feb 10 14:35:01 mtex zoraxy_linux_amd64[1972999]: 2025/02/10 14:35:01 Unable to read authkey at /var/lib/zerotier-one/authtoken.secret: exit status 1
Feb 10 14:35:01 mtex zoraxy_linux_amd64[1972999]: [2025-02-10 14:35:01.743074] [internal] [system:info] Failed to load ZeroTier controller API authtoken
Feb 10 14:35:01 mtex zoraxy_linux_amd64[1972999]: 2025/02/10 14:35:01 ZeroTier connection failed: Get "http://localhost:9993/status": dial tcp [::1]:9993: connect:>
Feb 10 14:35:01 mtex zoraxy_linux_amd64[1972999]: [2025-02-10 14:35:01.774032] [internal] [system:info] Starting ACME handler
Feb 10 14:35:01 mtex zoraxy_linux_amd64[1972999]: [2025-02-10 14:35:01.774270] [cert-renew] [system:info] ACME early renew set to 30 days and check interval set to >
Feb 10 14:35:01 mtex zoraxy_linux_amd64[1972999]: [2025-02-10 14:35:01.774906] [internal] [system:info] Serving inbound port 443
Feb 10 14:35:01 mtex zoraxy_linux_amd64[1972999]: [2025-02-10 14:35:01.774918] [internal] [system:info] TLS mode enabled. Serving proxy request with TLS
Feb 10 14:35:01 mtex zoraxy_linux_amd64[1972999]: [2025-02-10 14:35:01.774932] [internal] [system:info] Development mode enabled. Using no-store Cache Control policy
Feb 10 14:35:01 mtex zoraxy_linux_amd64[1972999]: [2025-02-10 14:35:01.774936] [internal] [system:info] Force latest TLS mode enabled. Minimum TLS LS version is set>
Feb 10 14:35:01 mtex zoraxy_linux_amd64[1972999]: [2025-02-10 14:35:01.774957] [internal] [system:info] Force HTTPS mode enabled
Feb 10 14:35:01 mtex zoraxy_linux_amd64[1972999]: [2025-02-10 14:35:01.774963] [internal] [system:info] Port 80 listener enabled
Feb 10 14:35:01 mtex zoraxy_linux_amd64[1972999]: panic: interface conversion: http.RoundTripper is *http2.Transport, not *http.Transport
Feb 10 14:35:01 mtex zoraxy_linux_amd64[1972999]: goroutine 84 [running]:
Feb 10 14:35:01 mtex zoraxy_linux_amd64[1972999]: imuslab.com/zoraxy/mod/dynamicproxy/dpcore.NewDynamicProxyCore(0xc045a30c60, {0x18?, 0xc000087c40?}, 0xc0354b34a0?)
Feb 10 14:35:01 mtex zoraxy_linux_amd64[1972999]: /home/mte/zoraxy/src/mod/dynamicproxy/dpcore/dpcore.go:118 +0x98
Feb 10 14:35:01 mtex zoraxy_linux_amd64[1972999]: imuslab.com/zoraxy/mod/dynamicproxy/loadbalance.(*Upstream).StartProxy(0xc045a46570)
Feb 10 14:35:01 mtex zoraxy_linux_amd64[1972999]: /home/mte/zoraxy/src/mod/dynamicproxy/loadbalance/upstream.go:41 +0x13e
Feb 10 14:35:01 mtex zoraxy_linux_amd64[1972999]: imuslab.com/zoraxy/mod/dynamicproxy.(*Router).PrepareProxyRoute(0xc04592ee60, 0xc045944f00)
Feb 10 14:35:01 mtex zoraxy_linux_amd64[1972999]: /home/mte/zoraxy/src/mod/dynamicproxy/router.go:24 +0x87
Feb 10 14:35:01 mtex zoraxy_linux_amd64[1972999]: main.LoadReverseProxyConfig({0xc045a0ee70?, 0x15?})
Feb 10 14:35:01 mtex zoraxy_linux_amd64[1972999]: /home/mte/zoraxy/src/config.go:78 +0x177
Feb 10 14:35:01 mtex zoraxy_linux_amd64[1972999]: main.ReverseProxtInit()
Feb 10 14:35:01 mtex zoraxy_linux_amd64[1972999]: /home/mte/zoraxy/src/reverseproxy.go:135 +0xa6d
Feb 10 14:35:01 mtex zoraxy_linux_amd64[1972999]: main.main.func1()
Feb 10 14:35:01 mtex zoraxy_linux_amd64[1972999]: /home/mte/zoraxy/src/main.go:120 +0xf
Feb 10 14:35:01 mtex zoraxy_linux_amd64[1972999]: created by main.main in goroutine 1
Feb 10 14:35:01 mtex zoraxy_linux_amd64[1972999]: /home/mte/zoraxy/src/main.go:119 +0x585
Feb 10 14:35:01 mtex systemd[1]: zoraxy.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Feb 10 14:35:01 mtex systemd[1]: zoraxy.service: Failed with result 'exit-code'.
Feb 10 14:35:01 mtex systemd[1]: zoraxy.service: Consumed 4.286s CPU time, 1.1G memory peak.
Hey @Morethanevil ,
Interesting, it works before I comment it out. Btw, do you have any recommendation for upstream where I can easily deploy and test for h2c? Something do not require dockers / VM will be nice.
Hey @Morethanevil ,
Interesting, it works before I comment it out. Btw, do you have any recommendation for upstream where I can easily deploy and test for h2c? Something do not require dockers / VM will be nice.
You mean a testing environment? A VPS would be good for this. I recommend Netcup or Contabo
They are cheap and powerful
@Morethanevil No I mean an open source project / testing tool for h2c connections. The best case would be a h2c server that response hello world when requested.
@Morethanevil No I mean an open source project / testing tool for h2c connections. The best case would be a h2c server that response hello world when requested.
Caddy is a web server which is used in docmost afaik. So you can use Caddy for those requests. In the docs, Caddy says not to use H2C unless you need it :D
Tried today again with the new 3.1.8 dev patch, Zoraxy starts but all my services throw 404, Zoraxy itself says they are online. Only thing I modified was to set if true{ in the file ./ zoraxy/src/mod/dynamicproxy/dpcore/dpcore.go
@Morethanevil I am still working on it (As you can see from the Docmost issue, I am still trying to figure out why the UI is not sending the correct header to Zoraxy in HTTP/2). This problem seems to be somewhat related to Docmost implementation and golang default http transport implementation just not compatible with it. This might take a bit more time for me to trace down the issue.
@Morethanevil I am still working on it (As you can see from the Docmost issue, I am still trying to figure out why the UI is not sending the correct header to Zoraxy in HTTP/2). This problem seems to be somewhat related to Docmost implementation and golang default http transport implementation just not compatible with it. This might take a bit more time for me to trace down the issue.
Ahhh okay, I thought I give it a try since the startup issue was fixed 😅
You can mention me again if we can test it, I will tty it out ☝🏻☺️
Since there is no update in this case, I decided to write a little workaround. Most people will use Docmost and Zoraxy in docker, so I used this for my workaround. We simply use NGINX as a middleware. Zoraxy -> NGINX -> Docmost.
I just published the workaround in the wiki, let me hear your thoughts.
The NGINX example even works without docker, the only thing to do would be changing the "proxy pass" to the right IP / hostname.
Thank you for providing a workaround. I hope this gets fixed soon, since Docmost is the only service I host that doesn't work with Zoraxy.
The workaround is nice. But it would be great to have the feature enabled.
I stumbled through the problem again while installing headscale. Sadly I can't figure out, how to configure H2c in the zoraxy GUI. Im using zoraxy v3.1.9. hopefully the upcoming v3.2.0 have the feature.
Coming from the same docmost <--> zoraxy h2c problem after installing this two services today. 😭
@Morethanevil No I mean an open source project / testing tool for h2c connections. The best case would be a h2c server that response hello world when requested.
Maybe this repo can serve as h2c hello world example.
https://github.com/thrawn01/h2c-golang-example
Maybe this repo can serve as h2c hello world example.
I am well aware of this example. Though it is not compatible with the current Zoraxy architecture, at least I failed to integrate it into Zoraxy last time I tried it.
That is why I am working on the plugin system right now and try to expose a scriptable routing interface for additional use cases where direct integration is not possible, and one of the first few use cases will be a h2c middleware.
Will follow up on this after the plugin system is completed.
Hiya there! Big fan of your work.
I've moved over the business' reverse proxy webserver from NPM to Zoraxy awhile back but noticed my test container for Docmost I'd had running previously broke when I swapped over.
Obviously this stuff takes time and you can only work on it as much and as best you can, but do you have any rough idea on how long till the plugin system may or may not be complete? That H2C middleware would be pretty sick to move our business away from Confluence.
Again, thank you for your incredible work on Zoraxy and I look forward to where it goes in the future.
The plugin system is currently mostly functional, though will be greatly extended in v3.2.6 via #753 and #746
Hey @AnthonyMichaelTDM , you wanna be the maintainer for the plugin interface module? I am a bit too busy to handle both the http proxy, UI and system integration. I guess I might need some help in finishing the plugin system. Drop me an email with your discord / telegram id and I will let you know the details :D
After a few months of experimenting, I checked "Disable Chunked Transfer Encoding" in Zoraxy v.3.2.5 today and now Docmost works for me via Zoraxy.
https://github.com/docmost/docmost/discussions/287#discussioncomment-14229710
It really isn't about H2C but just Docmost uses legacy HTTP server implementation. I guess this can be closed?
Closing this as the docmost issue has been solved by disabling chunked transfer encoding