Connection issue when running CoCalc docker behind a reverse proxy

[Grekkor]

When I access my cocalc docker instance in my local network the connection to the hub server is established, but when trying to access it from the outside via the configured FQDN (which passes first through my reverse proxy via HTTPS), then the connection to the hub server cannot be established.

[Grekkor]

Okay. I think I know why this happens. The reverse proxy was not configured to let through websocket calls. Unfortunately I do not know to what port or location the calls are supposed to go. Can anyone from the CoCalc team help me with this?

[Grekkor]

Ok I tried the following for my nginx reverse proxy: In sites-enabled I created a file with the following content:

server {
    listen       443 ssl;
    server_name  sub.domain.test;

    ssl on;
    ssl_certificate /.../fullchain.pem;

    ssl_certificate_key /.../privkey.pem;

    ssl_prefer_server_ciphers on;

    location / {
        proxy_pass https://localhost:26893/;
        proxy_set_header Host $host;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }

    location /api/websocket {
        proxy_set_header Host $host;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";                                      }
}

This (as far as I know) maps HTTPS requests from sub.domain.test to port 26893 (which is the docker instance). The lower part maps websocket requests.

This seems to work for me. Connection is now established and I can use it neetly. You guys might want to consider putting this up in the documentation. I am leaving this issue open so you guys actually see it ;)

[Alhazred242]

Which location are you use for the the WS? I try to use Apache as reverse Proxy and have the same Problem with connection issues.

[FastBlit]

Same problem, here is my initial attempt at a minimal apache vhost. I believe this is currently blocked by the internal self signed cert issue. Instead of using the internal letsencrypt script we should create the cert outside of the docker container and pass the keys through by conf reference to directory.

<VirtualHost _default_:443>
RequestHeader set X-Forwarded-Proto "https"
ServerName sub.domain.com

ProxyRequests Off
ProxyPreserveHost On

ProxyPass /api/websocket/ ws://127.0.0.1:8081/api/websocket
ProxyPassReverse /api/websocket/ ws://127.0.0.1:8081/api/websocket

ProxyPass / http://127.0.0.1:8081/
ProxyPassReverse / http://127.0.0.1:8081/

Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateFile /etc/letsencrypt/live/sub.domain.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/sub.domain.com/privkey.pem
</VirtualHost>

[sarunasb]

Here is Apache reverse proxy configuration, which works for us. Using valid certificate on the host and ignoring the self-signed one in container. Container's port 443 is mapped to external 4433.

SSLProxyEngine On
ProxyRequests Off
SSLProxyCheckPeerName off
ProxyPass "/" "https://localhost:4433/"
RewriteEngine on
RewriteCond %{HTTP:Upgrade} websocket [NC]
RewriteCond %{HTTP:Connection} upgrade [NC]
RewriteRule ^/?(.*) "wss://localhost:4433/$1" [P,L]
ProxyPassReverse "/" "https://localhost:4433/"

[aaszodi]

@sarunasb : Many thanks for sharing the Apache reverse proxy configuration. I have a slightly different setup: the container's port 443 is mapped to 8443, and I would like to reverse proxy it to "https://example.com/cocalc". In other words, map the container to a subdirectory (and not to the public FQDN of the server), like this:

https://example.com/cocalc <---> https://localhost:8443

I tried to modify the configuration above but didn't get far: I get a login screen with incorrect CSS and a "sign in" button that does not work :-( Any help would be appreciated.

[williamstein]

On a purely technical level, I don't understand how it is even possible to do something like this in complete generality:

https://example.com/cocalc <---> https://localhost:8443

The problem is that code in the page https://example.com/cocalc is going to request things not aware that it is being served from /cocalc, and I don't see how the web server can possibly know that the request is coming from https://example.com/cocalc and not https://example.com/foo (say)... in general.

There are a number of special cases where this sort of technical problem could be solved, I guess, e.g., by using a special cookie, maybe. However, I think it's impossible to solve in total generalty due to how web browsers and web servers work.

What is possible would be

https://example.com/cocalc <---> https://localhost:8443/cocalc

which can be accomplished using a BASE_PATH. However, custom base path's are really easily supported yet:

https://github.com/sagemathinc/cocalc-docker#new-using-a-custom-base-path

CoCalc does actually fully support base paths. However, it's a bit more complicated how to use them for cocalc-docker, and I haven't documented it yet. The steps would be to start things up, then as root set the BASE_PATH variable and rebuild certain parts of cocalc (which would take about 5 minutes). The ticket for documenting this process is here: https://github.com/sagemathinc/cocalc-docker/issues/130

Basically, it's fully implemented but not documented. It's implemented, since we use base paths for all development on cocalc itself.

[aaszodi]

What is possible would be

https://example.com/cocalc <---> https://localhost:8443/cocalc which can be accomplished using a BASE_PATH.

Indeed. This is how I reverse-proxy a Dockerised JupyterHub. Luckily JH supports changing base paths out of the box (or out of the container in this case :-) )

The steps would be to start things up, then as root set the BASE_PATH variable

What I did was that I ran the Cocalc container with the -e BASE_PATH=/cocalc setting and found out it does not fully work for the reasons @williamstein explained in his post -- so now I know why :-)

...and rebuild certain parts of cocalc (which would take about 5 minutes). The ticket for documenting this process is here: #130

Thank you for the pointer. I must admit the recipe is not clear to me, and the idea to "rebuild certain parts of cocalc" within a container frightens me off. I think I'd wait until the CoCalc image fully supports the BASE_PATH environment variable without any tweaking.

[williamstein]

I think this is out of scope for cocalc-docker.