Instructions for "more complex" setups

[wiswedel]

The setup wizard suggests to refer to the manual setup for more complex setups:

If your setup is more complex or involves any kind of load balancing you should follow the manual setup instruction on the README instead

However, the Manual Setup instructions don't go into that.

How would this be set up in a clustered environment where a number of web servers are behind an HA Proxy?

[Marcwa19197]

Hello,

also one note for SELinux enabled systems: I had to change the flag for the binary to "bin_t" for getting the systemd script working. This is only an enhancement for the "more complex" manual Setup instructions.

semanage fcontext -a -t bin_t /var/www/html/nextcloud/public_data/apps/notify_push/bin/x86_64/notify_push restorecon -v /var/www/html/nextcloud/public_data/apps/notify_push/bin/x86_64/notify_push

And if you are using http2 and apache proxy_http2_module you have to change the proxy setup. (Note the "h2c" for http2, you can also use "h2" for http2s)

	ProxyPass /push/ws ws://127.0.0.1:7867/ws
	ProxyPass /push/ h2c://127.0.0.1:7867/
	ProxyPassReverse /push/ h2c://127.0.0.1:7867/

[casparfriedrich]

Hi! My setup here used haproxy for load balancing and TLS termination. I would very much like to have some more real world example :-)

Caspar

[SimplyCorbett]

I am also using haproxy and would like some additional real-world examples.

[ghost]

I am using haproxy too and i am very interested in real world examples :-)

[SteelPC]

Same here :) I'd also like instruction with multiple nginx servers (we run 4 of them behind Haproxy)

[wiswedel]

ping @icewind1991

[elhananjair]

Hello,

also one note for SELinux enabled systems: I had to change the flag for the binary to "bin_t" for getting the systemd script working. This is only an enhancement for the "more complex" manual Setup instructions.

semanage fcontext -a -t bin_t /var/www/html/nextcloud/public_data/apps/notify_push/bin/x86_64/notify_push restorecon -v /var/www/html/nextcloud/public_data/apps/notify_push/bin/x86_64/notify_push

And if you are using http2 and apache proxy_http2_module you have to change the proxy setup. (Note the "h2c" for http2, you can also use "h2" for http2s)

	ProxyPass /push/ws ws://127.0.0.1:7867/ws
	ProxyPass /push/ h2c://127.0.0.1:7867/
	ProxyPassReverse /push/ h2c://127.0.0.1:7867/

Thanks a lot that helped me a lot, though I couldn't get what the difference between h2 and h2c is.

[DanScharon]

I'll try an answer with a setup, that we are currently testing on our test-environment:

notify_push

Installed on a separate host, which is different from our application servers (Apache+PHP-FPM)

with the following systemd unit definition:

[Unit]
Description = Push daemon for Nextcloud clients
Documentation=https://github.com/nextcloud/notify_push
After=network-online.target
Wants=network-online.target
RequiresMountsFor=/usr/local/etc/nextcloud

[Service]
ExecStart = /usr/local/bin/notify_push --socket-path /run/notify_push/notify_push.sock --socket-permissions 0660 /usr/local/etc/nextcloud/config.php
User=www-data
RuntimeDirectory=notify_push

[Install]
WantedBy = multi-user.target

Where /usr/local/etc/nextcloud is /var/www/nextcloud/config mounted via NFS.

Apache

on the same host as notify_push

# a2enmod ssl proxy proxy_http proxy_wstunnel

and the following configuration:

ServerName cloud-push-test.some.example.com
[...]
SSLEngine on
[...]
ProxyPass /push/ws  unix:/run/notify_push/notify_push.sock|ws://localhost/ws
ProxyPass /push/  unix:/run/notify_push/notify_push.sock|http://localhost/
ProxyPassReverse /push/  unix:/run/notify_push/notify_push.sock|http://localhost/

HAProxy

which already does the loadbalancing to the Nextcloud application servers

The following ACL in the frontend definition, so we don't proxy to the Nextcloud application servers:

acl notify_push path_beg /push
use_backend cloud-push if notify_push

And then the own backend definition:

backend cloud-push
  mode http
  timeout tunnel 3600s
  server push cloud-push-test.some.example.com:443 check ssl verify required ca-file ca-certificates.crt

This seems to work on our test setup, which we will continue to test. Feedback welcome.

EDIT: Watch out for your configuration in /etc/apache2/mods-enabled/mpm_event.conf as the value of MaxRequestWorkers defines how many clients can connect at the same time and is set to 150 by default on Debian and its derivates. More information on how to configure mpm_event can be found here: https://www.liquidweb.com/kb/apache-performance-tuning-mpm-directives/#eventworker EDIT 2: The Apache config will only work with Apache 2.4.47 and above, which is e.g. available in Debian bullseye or via buster-backports. EDIT 3: added timeout tunnel 3600s to haproxy.cfg

[SteelPC]

@DanScharon Did you run into the trusted proxy problem with this setup ? I can't get past it :(

• I set up a seperate server for notify_push (nothing else on it)
• I have the following on my nginx conf for the nextcloud instance (behing Haproxy for LB and SSL termination)

location ^~ /push/ {
        proxy_pass http://INTERNAL_PUSH_IP:7867/;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }

    location ^~ /metrics/ {
        proxy_pass http://INTERNAL_PUSH_IP:7868/metrics/;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }

/push/test/cookie and /metrics are working but I'm stuck with

push server is not a trusted proxy, please add 'INTERNAL_PUSH_IP' to the list of trusted proxies or configure any existing reverse proxy to forward the 'x-forwarded-for' send by the push server. See https://docs.nextcloud.com/server/latest/admin_manual/configuration_server/reverse_proxy_configuration.html#defining-trusted-proxies for how to set trusted proxies.
  The following trusted proxies are currently configured: "INTERNAL_PUSH_IP"
  The following x-forwarded-for header was received by Nextcloud: INTERNAL_PUSH_IP
    from the following remote: INTERNAL_PUSH_IP

[DanScharon]

@DanScharon Did you run into the trusted proxy problem with this setup ? I can't get past it :(

no, but we have set option forwardfor in the HAProxy frontend setting and on Apache we are using mod_remoteip

Could you try setting up nginx on the same server where notify_push is and then proxying from HAProxy to nginx and from nginx to local socket?

[SteelPC]

@DanScharon Did you run into the trusted proxy problem with this setup ? I can't get past it :(

no, but we have set option forwardfor in the HAProxy frontend setting and on Apache we are using mod_remoteip

Could you try setting up nginx on the same server where notify_push is and then proxying from HAProxy to nginx and from nginx to local socket?

I tried this too. No luck. Haproxy -> Nginx (on notify_push host) -> notify_push

[DanScharon]

Now that notify_push supports listening with TLS we could immensely simplify our setup and get rid of Apache as middleman.

We have the following now:

notify_push

Installed on a separate host, which is different from our application servers (Apache+PHP-FPM)

with the following systemd unit definition:

[Unit]
Description = Push daemon for Nextcloud clients
Documentation=https://github.com/nextcloud/notify_push
After=network-online.target
Requires=network-online.target
RequiresMountsFor=/var/www/nextcloud

[Service]
ExecStart = /var/www/nextcloud/apps/notify_push/bin/x86_64/notify_push --tls-cert /etc/ssl/certs/cloud-push.some.example.com.pem --tls-key /etc/ssl/private/cloud-push.some.example.com.key --glob-config /var/www/nextcloud/config/config.php
User=www-data

[Install]
WantedBy = multi-user.target

Where /var/www/nextcloud is the Nextcloud application folder mounted (readonly) via NFS.

HAProxy

which already does the loadbalancing to the Nextcloud application servers

The following ACL in the frontend definition, so we don't proxy to the Nextcloud application servers:

acl notify_push path_beg /push
use_backend cloud-push if notify_push

And then the own backend definition:

backend cloud-push
  mode http
  timeout tunnel 3600s
  server push cloud-push.some.example.com:7867 check ssl verify required ca-file ca-certificates.crt alpn http/1.1

HTH

[wonderfulShrineMaidenOfParadise]

Is it possible to have 2 notify_push servers and put them behind HAProxy and get better performance?

[Thaodan]

notify_push

Installed on a separate host, which is different from our application servers (Apache+PHP-FPM)

with the following systemd unit definition:

[Unit]
Description = Push daemon for Nextcloud clients
Documentation=https://github.com/nextcloud/notify_push
After=network-online.target
Wants=network-online.target
RequiresMountsFor=/usr/local/etc/nextcloud

[Service]
ExecStart = /usr/local/bin/notify_push --socket-path /run/notify_push/notify_push.sock --socket-permissions 0660 /usr/local/etc/nextcloud/config.php
User=www-data
RuntimeDirectory=notify_push

[Install]
WantedBy = multi-user.target

Should this also be in the readme? I believe for local connection a socket should be proffered.