Slow browsing on http2 enabled reverse proxy (apache2), long-polling `/user/events` blocks other requests

[sub0ne]

I currently have the following setup

• gitea 1.16.5
• mysql 8.0.28
• apache 2.4.38
• gitea ssl is disabled
• apache does forward http to https
• apache is configured to reverse proxy to gitea
• Google Chrome shows that all requests are done via h2

When I navigate the files of any of my repository and do "fast directory browsing" (e.g. folder A -> folder B -> up to folder A -> folder B). Something is getting hiccups and I get requests which take about 10 seconds before the folder content is displayed. Those 10 seconds are pretty constant and can be seen in the apaches logs as follow:

2022/03/29 13:56:24 Completed GET /user/events 200 OK in 10.003202925s

I spend hours trying to track this down to what the problem is, and finally I found that adding Protocols http/1.1 to the VirtualDirectory of my apache configuration seems to fix this problem. Pretty fast browsing with low requests roundtrips.

Is this problem known? What logs can I provide else?

[wxiaoguang]

/user/events is a long-polling end point. The log is changed in 1.17, if you don't like the log, use 1.17 instead.

[wxiaoguang]

Hmm ..... /user/events is a long-polling end point.

If "I get requests which take about 10 seconds before the folder content is displayed." is the case, I think it may be caused by http2's muxing ........

Indeed the long-polling seems not friendly to http2, it should be replaced by websocket

[gertvdijk]

Same issue here; thanks for the hint about HTTP/2. I can confirm that only enabling HTTP/1.1 on the public side of the Apache httpd fixes it. It looks like some concurrency issue with Apache. While using the Event worker MPM one should get 25 concurrent connections to the backends and this should not be an issue with just one user and one browser, I think. But it does block a lot.

Moved to nginx with HTTP/2 enabled and did not experience any issue.

[samangh]

I have the same issue, when using an Apache as a proxy. I can confirm that disabling HTTP/2 on the Apache config fixes this issue.

[wizpresso-steve-cy-fan]

We have also experienced such problem. We don't want to lose the benefit of HTTP/2 so we just disabled ui.notification, which means we set the following options in our helm chart values:

    ui.notification:
      EVENT_SOURCE_UPDATE_TIME: -1
      MAX_TIMEOUT: 1s
      MIN_TIMEOUT: -1
      TIMEOUT_STEP: 1s

Although this would mean at the cost of the real-time notifications, so far this eliminated the random long loading problem we have suffered through the last few months. We are still closely monitoring the situation especially regarding to other potential long polling stuff.

Potentially related issue: #17205 #12459

[hydrapolic]

Thanks for the heads up, I was also having the same issue. For the time being I've disable http2 and it's working just great. Reproducible with apache 2.4.57 / ssl / http2 / event npm.

[tdero]

Similar issues in gitea 1.21.4. Apache 2.4.57 setup with ssl http2 and event npm. Calls to /user/events lasting up to 15s, averaging at about 5s. Falling back to HTTP/1.1 resolved the hanging requests issue.

Added the above ui.notification settings to app.ini to disable it solved the issue with HTTP/2.

[jloup83]

Same issue here (gitea 1.21.7), works perfectly 80% of the time and then suddenly will randomly get stuck loading for about 10 seconds.

Environment details:

• docker gitea 1.21.7 + postgreSQL 14
• apache2 2.4.52 as a reverse-proxy
• HTTP for gitea, HTTPS for apache2

This has been going on for at least 2 years from what i can see. Can we get at least an explaination of what is going on, and an ETA for the fix? It seems some stuff has been merge, but so far not much has changed on my end....

Honestly gitea looks really awesome and feature packed, but this one bug kind of prevents me from committing long term to this tool.....

So how about that "disabling HTTP/2" fix ? is it secure if it applies to a gitea instance that is not exposed directly to the web? (it is exposed only by the reverse-proxy)

the domain name that I use is very exposed to attacks, so security is a big concern for me....

Thanks for your time guys

[hmoffatt]

So how about that "disabling HTTP/2" fix ? is it secure

Disabling HTTP/2 may reduce performance but it won't reduce security.

[wolfbeast]

I ran into a different issue from this same problem and that is a matter of scale/load, the long polling seems to keep an open connection to every client with a GET ping every 3 seconds or so; this is causing stack saturation in our nginx front-end with long-running sessions of the course of a few weeks, necessitating service restarts or we get really long/slow loads. So the issue isn't necessarily restricted to Apache, just manifesting itself in a different way. I think moving to websockets for this would be a Good Thing™ - pretty sure that's a textbook example of "when to use websockets" ;)

[seraphyn]

I encountered exactly this problem today and downgraded to http1.1. from h2/h2c Is there another solution for this? Do you already know more?

[lunny]

It's a known issue. Currently, some parts prevent Gitea works well on http2. Like the mentioned /usr/events.