Server APIs failing behind nginx as reverse proxy

[namangoel31]

Description

I'm looking to integrate jupyter notebook using UI components from Datalayer into my setup. It needs to be a single user jupyter server hosted on a private kubernetes cluster exposed via ALB. Between the Jupyter server and the ALB, sits nginx as reverse proxy. Current setup is described below:

Reproduce

Container: Ubuntu base image with jupyter_server, ipython, ipykernel, and jupyterlab.

Jupyter config:

# Configuration file for lab.
import os
c = get_config()  #noqa
c.LabServerApp.open_browser = False
c.IdentityProvider.token = os.getenv('JUPYTER_TOKEN')
#c.LabApp.collaborative = True
c.ServerApp.allow_unauthenticated_access = True
c.ServerApp.allow_credentials = True
ORIGIN='*'
#c.ServerApp.allow_origin_pat = '.*'
c.ServerApp.allow_origin="http://sahiltest.gz.local.localhost:3000"
#c.IdentityProvider.cookie_options = {
#  "SameSite": "None",
#  "Secure": True,
#}
c.ServerApp.allow_remote_access= True
c.ServerApp.disable_check_xsrf = True
c.ServerApp.ip = '0.0.0.0'
c.ServerApp.jpserver_extensions = {
    'jupyterlab': True,
}
c.ZMQChannelsWebsocketConnection.kernel_ws_protocol = None
c.ServerApp.log_level = 'INFO'
c.ServerApp.port_retries = 0
c.FileContentsManager.preferred_dir = '/home/notebooks'
c.ServerApp.root_dir = '/home/notebooks'
c.ServerApp.terminals_enabled = True
#c.ServerApp.allow_origin = ORIGIN
c.ServerApp.tornado_settings = {
  'headers': {
    #'Access-Control-Allow-Origin': ORIGIN,
    'Access-Control-Allow-Methods': '*',
    'Access-Control-Allow-Headers': 'Accept, Accept-Encoding, Accept-Language, Authorization, Cache-Control, Connection, Content-Type, Host, Origin, Pragma, Referer, sec-ch-ua, sec-ch-ua-mobile, sec-ch-ua-platform, Sec-Fetch-Dest, Sec-Fetch-Mode, Sec-Fetch-Site, Upgrade, User-Agent, X-XSRFToken, X-Datalayer, Expires, X-CustomHeader, X-Service-Name',
    'Access-Control-Allow-Credentials': 'true',
    'Content-Security-Policy': f"frame-ancestors 'self' {ORIGIN} ",
  },
  'cookie_options': {
    'SameSite': 'None',
    'Secure': True
  }
}
def upload_to_s3(model, os_path, contents_manager, **kwargs):
    if model["type"] != "notebook":
        return
    import boto3
    s3_client = boto3.client('s3')
    bucket_name = 's3bucket' #this has to come dynamically according to the site.
    s3_path_prefix = 'notebooks/'
    #base, ext = os.path.splitext(os_path)
    local_file = os_path
    base, extn = os.path.splitext(os_path)
    filename= base.strip().split('/')[-1]+extn
    log = contents_manager.log
    #local_file_path = os.path.join(local_file, filename)
    s3_file_key = f'{s3_path_prefix}{filename}'
    log.info("Saving notebook: /%s", s3_file_key)
    try:
        s3_client.upload_file(local_file, bucket_name, s3_file_key)
        print(f"Successfully uploaded {filename} to s3://{bucket_name}/{s3_file_key}")
    except Exception as e:
        print(f"Error uploading {filename}: {e}")
c.FileContentsManager.post_save_hook = upload_to_s3

nginx conf:

server {
        listen 80;
        listen [::]:80;
        server_name http://k8s-devdev-nginxtes-b82ba16213-1347592838.ap-south-1.elb.amazonaws.com;
        location / {
                proxy_pass http://jupyter-mlflow4:8888;
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "upgrade";
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
        }
}

Browser Network tools show APIs failing (screenshot and logs attached in a separate comment below).

• Operating System and version: Ubuntu 20.04 Jammy
• Browser and version: Any
• IPython : 8.28.0
• ipykernel : 6.29.5
• ipywidgets : not installed
• jupyter_client : 8.6.3
• jupyter_core : 5.7.2
• jupyter_server : 2.14.2
• jupyterlab : 4.2.5
• nbclient : 0.10.0
• nbconvert : 7.16.4
• nbformat : 5.10.4
• notebook : not installed
• qtconsole : not installed
• traitlets : 5.14.3
• tornado :6.2