[AzureAD, GitLab, maybe more] Use of web traffic proxy cause doesn't work

[mvuddaraju]

Actual behaviour

Our VMs with jupyterhub and oauth installed are behind a proxy. Jupyter comes up fine but fails to login azure-active directory user with the proxy set.

i.e when proxy is set in the shell before starting the jupyter as follows

http_proxy=http://x.x.x.x:xxxx
https_proxy=http://x.x.x.x:xxxx
no_proxy="xxx"

jupyter comes up fine with aadlogin screen, however it fails to login the user with the following error

See error logs

[D 2021-01-21 16:55:32.169 JupyterHub log:181] 200 GET /hub/logo (@::ffff:10.250.6.187) 0.74ms
[D 2021-01-21 16:55:32.236 JupyterHub log:181] 200 GET /hub/static/favicon.ico?v=fde5757cd3892b979919d3b1faa88a410f28829feb5ba22b6cf069f2c6c98675fceef90f932e49b510e74d65c681d5846b943e7f7cc1b41867422f0481085c1f (@::ffff:10.250.6.187) 0.56ms
[I 2021-01-21 16:55:33.537 JupyterHub oauth2:104] OAuth redirect: 'https://vm-dsvmv2-ihprodtest2.bdaa.prod.ams.azu.dbgcloud.io:8000/hub/oauth_callback'
[D 2021-01-21 16:55:33.538 JupyterHub base:519] Setting cookie oauthenticator-state: {'httponly': True, 'secure': True, 'expires_days': 1}
[I 2021-01-21 16:55:33.539 JupyterHub log:181] 302 GET /hub/oauth_login?next= -> https://login.microsoftonline.com/e00ddcdf-1e0f-4be5-a37a-894a4731986a/oauth2/authorize?response_type=code&redirect_uri=https%3A%2F%2Fvm-dsvmv2-ihprodtest2.bdaa.prod.ams.azu.dbgcloud.io%3A8000%2Fhub%2Foauth_callback&client_id=de9b193b-6b87-4ee8-954e-de1e6e6d99d9&state=[secret] (@::ffff:10.250.6.187) 1.94ms
[E 2021-01-21 16:55:33.974 JupyterHub web:1789] Uncaught exception GET /hub/oauth_callback?code=0.AAAA39wN4A8e5UujeolKRzGYajsZm96Ha-hOlU7eHm5tmdlHAOQ.AQABAAIAAABeStGSRwwnTq2vHplZ9KL4YTkNi3qqCQtVs2wQum5strzM01iQSJTQacPyjTFmvU3OxGWYIOHTje2sbORUYEiTuLpoL262hrjry-NjGj5Ae7r_t8fFKlJELHtB5ij2HkG2eka6CIe-GBNGRLXC7aF1j99COAG9DRck5q38tcm9T8CLGr_HfseI3fzCiOzwU7kNeUSAl-gB8X8fuq6nz5jYN49CeSJBRn1_4w3EqQg9fY39B3qCUwLSmLVSWlJiWnxyXA-I-OKRc7QXEIN0Sz3ZwRoAZ6lSZEg6P_3zLdj0p_Xei_G3VieRsSu5b0JAv9pU696drT_okqWT1w3ILqix2SiClCzrYDCrakpBVdsYNGT0_-1zj-0e4QHdeQd3Cv6TQRa2JXVbmfIBAvNtRMQ-SAu_vXlnCfm3qTRlfti8xMMfvhHa3mJv-zeqlAgJSiGFDAvVhsCIc21o8cCzxNm4s7ui2TVBKfvW-HgK43opdGI6gCqtE3_C3katJR8HZxWtQD3oId3AQKCROmIHjVNwNcMkfsolouQXuL14KvsBKXjSwKJwX24jxW-hU8p8rZD4RJwKlyjhyB7WV2xAOgxXzkW2fQck-jWteM1EOrLjhUJVFbAPzoPohCKxkDdPHfqRFqNB94z1_VkSeglukajMAV2ZHwbVDC0C9imhbkJKMPbJWnBhwA81ywjJWCCYd1ogdSgJxFl3yd4fR-9mGog6zQJnldFyn3D8W6ifSvNYmCAA&state=eyJzdGF0ZV9pZCI6ICI0ODFkMDQxMzBjNzE0NTI3YjkwZGJkMDNjOTMzMmQ2ZSIsICJuZXh0X3VybCI6ICIifQ%3d%3d&session_state=47babd93-3a49-4128-9bd2-05e3b2987279 (::ffff:10.250.6.187)
    HTTPServerRequest(protocol='https', host='vm-dsvmv2-ihprodtest2.bdaa.prod.ams.azu.dbgcloud.io:8000', method='GET', uri='/hub/oauth_callback?code=0.AAAA39wN4A8e5UujeolKRzGYajsZm96Ha-hOlU7eHm5tmdlHAOQ.AQABAAIAAABeStGSRwwnTq2vHplZ9KL4YTkNi3qqCQtVs2wQum5strzM01iQSJTQacPyjTFmvU3OxGWYIOHTje2sbORUYEiTuLpoL262hrjry-NjGj5Ae7r_t8fFKlJELHtB5ij2HkG2eka6CIe-GBNGRLXC7aF1j99COAG9DRck5q38tcm9T8CLGr_HfseI3fzCiOzwU7kNeUSAl-gB8X8fuq6nz5jYN49CeSJBRn1_4w3EqQg9fY39B3qCUwLSmLVSWlJiWnxyXA-I-OKRc7QXEIN0Sz3ZwRoAZ6lSZEg6P_3zLdj0p_Xei_G3VieRsSu5b0JAv9pU696drT_okqWT1w3ILqix2SiClCzrYDCrakpBVdsYNGT0_-1zj-0e4QHdeQd3Cv6TQRa2JXVbmfIBAvNtRMQ-SAu_vXlnCfm3qTRlfti8xMMfvhHa3mJv-zeqlAgJSiGFDAvVhsCIc21o8cCzxNm4s7ui2TVBKfvW-HgK43opdGI6gCqtE3_C3katJR8HZxWtQD3oId3AQKCROmIHjVNwNcMkfsolouQXuL14KvsBKXjSwKJwX24jxW-hU8p8rZD4RJwKlyjhyB7WV2xAOgxXzkW2fQck-jWteM1EOrLjhUJVFbAPzoPohCKxkDdPHfqRFqNB94z1_VkSeglukajMAV2ZHwbVDC0C9imhbkJKMPbJWnBhwA81ywjJWCCYd1ogdSgJxFl3yd4fR-9mGog6zQJnldFyn3D8W6ifSvNYmCAA&state=eyJzdGF0ZV9pZCI6ICI0ODFkMDQxMzBjNzE0NTI3YjkwZGJkMDNjOTMzMmQ2ZSIsICJuZXh0X3VybCI6ICIifQ%3d%3d&session_state=47babd93-3a49-4128-9bd2-05e3b2987279', version='HTTP/1.1', remote_ip='::ffff:10.250.6.187')
    Traceback (most recent call last):
      File "/datadrive/anaconda/lib/python3.8/site-packages/tornado/web.py", line 1704, in _execute
        result = await result
      File "/datadrive/anaconda/bin/oauthenticator-master/oauthenticator/oauth2.py", line 224, in get
        user = await self.login_user()
      File "/datadrive/anaconda/lib/python3.8/site-packages/jupyterhub/handlers/base.py", line 747, in login_user
        authenticated = await self.authenticate(data)
      File "/datadrive/anaconda/lib/python3.8/site-packages/jupyterhub/auth.py", line 459, in get_authenticated_user
        authenticated = await maybe_future(self.authenticate(handler, data))
      File "/datadrive/anaconda/bin/oauthenticator-master/oauthenticator/azuread.py", line 75, in authenticate
        resp = await http_client.fetch(req)
    tornado.curl_httpclient.CurlError: HTTP 599: SSL certificate problem: unable to get local issuer certificate

[D 2021-01-21 16:55:33.976 JupyterHub base:1256] No template for 500
[E 2021-01-21 16:55:33.981 JupyterHub log:173] {
      "X-Forwarded-Host": "vm-dsvmv2-ihprodtest2.bdaa.prod.ams.azu.dbgcloud.io:8000",
      "X-Forwarded-Proto": "https",
      "X-Forwarded-Port": "8000",
      "X-Forwarded-For": "::ffff:10.250.6.187",
      "Cookie": "_xsrf=[secret]; oauthenticator-state=[secret]",
      "Accept-Language": "en-US,en;q=0.9",
      "Accept-Encoding": "gzip, deflate, br",
      "Referer": "https://vm-dsvmv2-ihprodtest2.bdaa.prod.ams.azu.dbgcloud.io:8000/",
      "Sec-Fetch-Dest": "document",
      "Sec-Fetch-User": "?1",
      "Sec-Fetch-Mode": "navigate",
      "Sec-Fetch-Site": "cross-site",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
      "User-Agent": "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36",
      "Upgrade-Insecure-Requests": "1",
      "Connection": "close",
      "Host": "vm-dsvmv2-ihprodtest2.bdaa.prod.ams.azu.dbgcloud.io:8000"
    }
[E 2021-01-21 16:55:33.981 JupyterHub log:181] 500 GET /hub/oauth_callback?code=[secret]&state=[secret]&session_state=[secret] (@::ffff:10.250.6.187) 117.74ms
[D 2021-01-21 16:55:34.047 JupyterHub log:181] 200 GET /hub/static/css/style.min.css?v=a690a7f26f8eda24b6043d972eda379b7f3df65dcb4832fbec0d72d1585b6bdf2bd15c7f3ec4597b79a7c91798181ba23b9a0204da8165b21ff81cd85f89a933 (@::ffff:10.250.6.187) 1.07ms

however, when we disable proxy setup as below

http_proxy=""
https_proxy=""
no_proxy=""

Jupyter oauth login works fine.

We are seeing this issue only with recent versions (EDIT: 0.12.4) of Jupyterhub-oauthenticator and this worked fine before (EDIT: 0.11.1).

Your personal set up

• OS: REDHAT 7.7
• Python: 3.8.3

jupyter --version

jupyter --version
jupyter core     : 4.7.0
jupyter-notebook : 6.2.0
qtconsole        : not installed
ipython          : 7.12.0
ipykernel        : 5.4.2
jupyter client   : 6.1.11
jupyter lab      : 2.2.9
nbconvert        : 6.0.7
ipywidgets       : not installed
nbformat         : 5.1.2
traitlets        : 5.0.5

jupyterhub_config.py

# jupyterhub_config.py

c.Authenticator.admin_users = {'xxx'}
c.Authenticator.whitelist = {'xxx'}
from oauthenticator.azuread import AzureAdOAuthenticator
c.JupyterHub.authenticator_class = AzureAdOAuthenticator
c.AzureAdOAuthenticator.oauth_callback_url = 'https://xxx/hub/oauth_callback'
c.AzureAdOAuthenticator.client_id = 'xxx'
c.AzureAdOAuthenticator.tenant_id = 'xxx'
c.AzureAdOAuthenticator.client_secret = 'xxx'
c.AzureAdOAuthenticator.tls_verify='0'
c.JupyterHub.log_level = 'DEBUG'
c.ConfigurableHTTPProxy.api_url = 'xxx'
c.JupyterHub.ssl_cert = 'xxx'
c.JupyterHub.ssl_key = 'xxx'
c.Spawner.default_url = '/lab'
c.Spawner.notebook_dir = '~/notebooks'

[welcome[bot]]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively. You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! :wave:
Welcome to the Jupyter community! :tada:

[kianaf]

I get the same error!

[manics]

We are seeing this issue only with recent versions of Jupyterhub-oauthenticator and this worked fine before.

Do you know which release of jupyterhub-oauthenticator broke your configuration?

[kianaf]

Hi @manics, Thank you so much for your help. I am struggling with this error for a long time. I am using the z2jh distribution and I am getting the same error for all the versions from 0.9.0 to 0.11.1, which the OAuthenticator versions are from 0.11.0 to 0.12.3 The other thing that I understood is that it has something to do with ".GitLabOAuthenticator.allowed_gitlab_groups ". Because when I remove this everything works fine.

[mvuddaraju]

Hi @manics,

This is working with version 0.11.1 and failing with version 0.12.4. Unfortunately, I am not able to point at which version it started to fail as we upgraded from 0.11.1 to 0.12.4

Regards, Madhuri

[mvuddaraju]

Hello @manics ,

Any updates on the issue here?.

Regards, Madhuri

[mvuddaraju]

Hello @manics,

Any updates here?.

Regards, Madhuri

[consideRatio]

This is probably related https://github.com/jupyterhub/oauthenticator/issues/217.