How to run tljh behind nginx

[mikanyman]

tljh installed nicely. I have a subdomain dedicated for tljh, but otherwise the services on the domain run under nginx. When I call the subdomain from a browser I get the nginx welcome page as response. "configure tljh to run on localhost only, and do ssl termination, proxying with nginx" seems to be the solution, but I don't know how to do that. I would really appreciate any help.

[frier-sam]

well i did the same but without ssl(don't judge me) i am using this on aws ec2 intance (ubuntu 18.04 ) here is the process i followed

i installed tljh by using command given in documentation https://tljh.jupyter.org/en/latest/install/custom-server.html

i stopped the traefik and jupterhub service once the installation completed successfully

systemct1 stop traefik.service
systemct1 stop jupyterhub.service

since i am using ec2 i had to use sudo

i opened traefik.toml in /opt/tljh/state/ and changed the entrypoint port to 1600

i installed nginx and stopped nginx once installed

sudo apt install nginx
systemct1 stop nginx.service

again using sudo

i opened the nginx config at /etc/nginx/sites-enable/default and added the following route as virtual host

server {
    listen 80;
    server_name subdomain.example.com;

    location / {
        proxy_pass http://localhost:1600;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_redirect     default;
        #proxy_set_header   Host             $http_host;
        proxy_set_header   X-Real-IP        $remote_addr;
        proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-User $http_authorization;
        proxy_max_temp_file_size 0;

        #this is the maximum upload size
        client_max_body_size       20m;
        client_body_buffer_size    128k;

        proxy_connect_timeout      90;
        proxy_send_timeout         90;
        proxy_read_timeout         90;
        proxy_buffer_size          4k;
         proxy_buffers              4 32k;
        proxy_busy_buffers_size    64k;
        proxy_temp_file_write_size 64k;

        }
}

well this takes care of websocket and uploading to subdomain

now i started all the services again

systemct1 start nginx.service
systemct1 start traefik.service
systemct1 start jupyterhub.service

now hub is available on subdomain.example.com

that's how i did it. if you have any better way i am listening

[dalwar23]

@frier-sam This nginx solution worked for me. But still the tljh is accessible via IP address also. Is there a way to bind it to localhost/27.0.0.1 and then let nginx handle the proxy? Thanks!

[rejsmont]

@frier-sam This nginx solution worked for me. But still the tljh is accessible via IP address also. Is there a way to bind it to localhost/27.0.0.1 and then let nginx handle the proxy? Thanks!

Just modify this in /opt/tljh/state/traefik.toml:

[entryPoints]
  [entryPoints.http]
  address = "127.0.0.1:1600"

[Hannnsen]

I just followed your guide (thank you for that!), because I wanted to set up several services alongside TLJH. So my URL setup looks similar to following:

jupyter.groupname.com -> access to TLJH                    (localhost:8000)
chat.groupname.com    -> access to Rocket.Chat             (localhost:3000)
groupname.com         -> static HTML content for promotion (/var/www/data/...)

For this, I wanted to use nginx. So I created the config file for jupyter similar to yours posted above. I also created some for the other services. Now, I can access Rocket.Chat and the Webpage without any issues. And, at least at first sight, TLJH runs, but it seems to be not true:

I can access the jupyter frontpage, where I'm asked to log in. But after clicking on login I get a "500 Internal Server Error". The funny thing with this is that users who logged in before putting nginx as a reverse proxy in the middle of the server are still recognized but if you log out and try to log in again it throws you the same internal server error. Additionally, the URL seems to be off (it calls: https://jupyter.groupname.com/hub/login?next=) - should next be empty? My suggestion is that there is something wrong with cookies maybe?

Does anybody know how to fix this? I would be really happy about any help!

This is how the login looks:

This is how the error looks:

[quantleap]

@frier-sam This nginx solution worked for me. But still the tljh is accessible via IP address also. Is there a way to bind it to localhost/27.0.0.1 and then let nginx handle the proxy? Thanks!

Just modify this in /opt/tljh/state/traefik.toml:

[entryPoints]
  [entryPoints.http]
  address = "127.0.0.1:1600"

this solution is not working for me: when the traefik.toml is changed and the proxy reloaded, the toml configuration file is reverted.

[Hannnsen]

@frier-sam This nginx solution worked for me. But still the tljh is accessible via IP address also. Is there a way to bind it to localhost/27.0.0.1 and then let nginx handle the proxy? Thanks!

Just modify this in /opt/tljh/state/traefik.toml:

[entryPoints]
  [entryPoints.http]
  address = "127.0.0.1:1600"

this solution is not working for me: when the traefik.toml is changed and the proxy reloaded, the toml configuration file is reverted.

Well, I already tried this. I'm currently not sure if it got messed up by an upgrade to Ubuntu 20.04 LTS, because also after installing it new it was not working properly. Now switched back to a backup on 18.04, which somehow works. I'm not sure if I'll try it again because it was giving our group headache during the time the server was not usable. However, thank you for you answer :)

[consideRatio]

I think what's in scope documentation wise for the TLJH project, is how to configure what port we expose. That is already documented here, even though I find that location to be quite hard to find. Here is how you can update the ports used.

sudo tljh-config set http.port 8080
sudo tljh-config set https.port 8443
sudo tljh-config reload proxy

I think it is out of scope for the TLJH project to document how to configure NGINX to do various things before proxying traffic towards TLJH.

I'm not sure if I can identify a concrete action point to focus on in order to close this issue that is also in scope of the TLJH project. Due to that, I'd like to close this issue. Please bring up concrete action points that we can go for if you have one, otherwise I suggest this issue is closed in a week.

[JobinJohan]

I wrote a short tutorial that explains all steps to install TLJH behind Nginx (as a reverse proxy). You can find it there: https://github.com/JobinJohan/the_littlest_jupyterhub_behind_nginx

Hope it helps :)

[eingemaischt]

I wrote a short tutorial that explains all steps to install TLJH behind Nginx (as a reverse proxy). You can find it there: https://github.com/JobinJohan/the_littlest_jupyterhub_behind_nginx

Hope it helps :)

Nice, thank you! But it does not bind the traefik to 127.0.0.1/localhost only, is there any idea here how to configure traefik directly in a way that it persists?

[eingemaischt]

Thank you @rejsmont : I added the lines of your example:

[entryPoints]
  [entryPoints.http]
  address = "127.0.0.1:8081"

to /opt/tljh/config/traefik_config.d/trefik.toml

After a tljh-config reload proxy it now binds to localhost only!

[MridulS]

The tutorial provided by @JobinJohan in https://github.com/jupyterhub/the-littlest-jupyterhub/issues/272#issuecomment-955593347 covers a way of running tljh behind a nginx proxy. As @consideRatio mentioned it's probably out of scope for TLJH to actively maintain documentation for this so I'll go ahead and close this issue. Thanks to everyone for contributing to this issue! Please feel free to add new things in this issue later too :)