airflow icon indicating copy to clipboard operation
airflow copied to clipboard
verified publisher icon apache

Upgrading to 3.1.3 yields 401 Unauthorized error

Open santurini opened this issue 1 month ago • 1 comments

Apache Airflow version

3.1.3

If "Other Airflow 2/3 version" selected, which one?

No response

What happened?

I tried to upgrade my airflow helm chart from version 3.0.2 to version 3.1.3. I started with a fresh database and namespace, but when I try to access airflow at http://airflow.staging.mydomain.cloud I get redirected to http://airflow.staging.mydomain.cloud/auth/login/?next=http://airflow.staging.mydomain.cloud/ and it shows Internal Server Error.

Inspecting the api-server logs I found this:

INFO:     10.42.5.9:48748 - "GET /ui/config HTTP/1.1" 401 Unauthorized

INFO:     10.42.5.9:48754 - "GET /static/i18n/locales/en/common.json HTTP/1.1" 304 Not Modified

INFO:     10.42.5.9:48748 - "GET /static/i18n/locales/en/dashboard.json HTTP/1.1" 304 Not Modified

INFO:     10.42.5.9:48768 - "GET /static/i18n/locales/en/dags.json HTTP/1.1" 304 Not Modified

INFO:     10.42.5.9:48786 - "GET /static/i18n/locales/en/assets.json HTTP/1.1" 304 Not Modified

INFO:     10.42.5.9:48774 - "GET /static/i18n/locales/en/components.json HTTP/1.1" 304 Not Modified

INFO:     10.42.5.9:48748 - "GET /static/i18n/locales/en/browse.json HTTP/1.1" 304 Not Modified

INFO:     10.42.5.9:48754 - "GET /static/i18n/locales/en/admin.json HTTP/1.1" 304 Not Modified

INFO:     10.42.5.9:48792 - "GET /static/i18n/locales/en/hitl.json HTTP/1.1" 304 Not Modified

INFO:     10.42.5.9:48792 - "GET /api/v2/auth/login?next=http%3A%2F%2Fairflow.staging.translated.cloud%2F HTTP/1.1" 307 Temporary Redirect

2025-12-10T15:27:14.185556Z [error    ] Exception in callback <function TaskGroup._spawn.<locals>.task_done at 0x74592c24fb00>

handle: <Handle TaskGroup._spawn.<locals>.task_done> [asyncio] loc=runners.py:118

Traceback (most recent call last):

  File "uvloop/cbhandles.pyx", line 63, in uvloop.loop.Handle._run

  File "/home/airflow/.local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 820, in task_done

    self._on_completed_fut.set_result(None)

asyncio.exceptions.InvalidStateError: invalid state

2025-12-10T15:27:14.187211Z [error    ] Exception in callback <function TaskGroup._spawn.<locals>.task_done at 0x74592c24eb60>

handle: <Handle TaskGroup._spawn.<locals>.task_done> [asyncio] loc=runners.py:118

Traceback (most recent call last):

  File "uvloop/cbhandles.pyx", line 63, in uvloop.loop.Handle._run

  File "/home/airflow/.local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 825, in task_done

    exc = _task.exception()

          ^^^^^^^^^^^^^^^^^

  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/base.py", line 144, in coro

    await self.app(scope, receive_or_disconnect, send_no_error)

  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 63, in __call__

    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)

  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app

    await app(scope, receive, sender)

  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/routing.py", line 716, in __call__

    await self.middleware_stack(scope, receive, send)

  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/routing.py", line 736, in app

    await route.handle(scope, receive, send)

  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/routing.py", line 462, in handle

    await self.app(scope, receive, send)

  File "/home/airflow/.local/lib/python3.11/site-packages/fastapi/applications.py", line 1082, in __call__

    await super().__call__(scope, receive, send)

  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/applications.py", line 113, in __call__

    await self.middleware_stack(scope, receive, send)

  File "/opt/datadog/apm/library/python/ddtrace_pkgs/site-packages-ddtrace-py3.11-manylinux2014/ddtrace/contrib/internal/asgi/middleware.py", line 456, in __call__

    return await self.app(scope, wrapped_recv, wrapped_send)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__

    await self.app(scope, receive, _send)

  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 63, in __call__

    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)

  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app

    await app(scope, receive, sender)

  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/routing.py", line 716, in __call__

    await self.middleware_stack(scope, receive, send)

  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/routing.py", line 736, in app

    await route.handle(scope, receive, send)

  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/routing.py", line 462, in handle

    await self.app(scope, receive, send)

  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/wsgi.py", line 81, in __call__

    await responder(receive, send)

  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/wsgi.py", line 101, in __call__

    message = await receive()

              ^^^^^^^^^^^^^^^

  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/base.py", line 117, in receive_or_disconnect

    async with anyio.create_task_group() as task_group:

  File "/home/airflow/.local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 755, in __aexit__

    await self._on_completed_fut

asyncio.exceptions.CancelledError: Cancelled via cancel scope 74592c5d9cd0 by <Task pending name='starlette.middleware.base.BaseHTTPMiddleware.__call__.<locals>.call_next.<locals>.coro' coro=<BaseHTTPMiddleware.__call__.<locals>.call_next.<locals>.coro() running at /home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/base.py:144> cb=[TaskGroup._spawn.<locals>.task_done() at /home/airflow/.local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py:805]>

INFO:     10.42.5.9:48792 - "GET /auth/login/?next=http://airflow.staging.translated.cloud/ HTTP/1.1" 500 Internal Server Error

ERROR:    Exception in ASGI application

  + Exception Group Traceback (most recent call last):

  |   File "/home/airflow/.local/lib/python3.11/site-packages/starlette/_utils.py", line 79, in collapse_excgroups

  |     yield

  |   File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/base.py", line 183, in __call__

  |     async with anyio.create_task_group() as task_group:

  |   File "/home/airflow/.local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 783, in __aexit__

  |     raise BaseExceptionGroup(

  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)

  +-+---------------- 1 ----------------

    | Traceback (most recent call last):

    |   File "/home/airflow/.local/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi

    |     result = await app(  # type: ignore[func-returns-value]

    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

    |   File "/home/airflow/.local/lib/python3.11/site-packages/fastapi/applications.py", line 1082, in __call__

    |     await super().__call__(scope, receive, send)

    |   File "/home/airflow/.local/lib/python3.11/site-packages/starlette/applications.py", line 113, in __call__

    |     await self.middleware_stack(scope, receive, send)

    |   File "/opt/datadog/apm/library/python/ddtrace_pkgs/site-packages-ddtrace-py3.11-manylinux2014/ddtrace/contrib/internal/asgi/middleware.py", line 456, in __call__

    |     return await self.app(scope, wrapped_recv, wrapped_send)

    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

    |   File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__

    |     raise exc

    |   File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__

    |     await self.app(scope, receive, _send)

    |   File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/gzip.py", line 29, in __call__

    |     await responder(scope, receive, send)

    |   File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/gzip.py", line 130, in __call__

    |     await super().__call__(scope, receive, send)

    |   File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/gzip.py", line 46, in __call__

    |     await self.app(scope, receive, self.send_with_compression)

    |   File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__

    |     await self.app(scope, receive, send)

    |   File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/base.py", line 182, in __call__

    |     with recv_stream, send_stream, collapse_excgroups():

    |   File "/usr/python/lib/python3.11/contextlib.py", line 158, in __exit__

    |     self.gen.throw(typ, value, traceback)

    |   File "/home/airflow/.local/lib/python3.11/site-packages/starlette/_utils.py", line 85, in collapse_excgroups

    |     raise exc

    |   File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/base.py", line 184, in __call__

    |     response = await self.dispatch_func(request, call_next)

    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

    |   File "/home/airflow/.local/lib/python3.11/site-packages/airflow/api_fastapi/auth/middlewares/refresh_token.py", line 49, in dispatch

    |     response = await call_next(request)

    |                ^^^^^^^^^^^^^^^^^^^^^^^^

    |   File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/base.py", line 160, in call_next

    |     raise RuntimeError("No response returned.")

    | RuntimeError: No response returned.

    +------------------------------------

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/home/airflow/.local/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi

    result = await app(  # type: ignore[func-returns-value]

             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/airflow/.local/lib/python3.11/site-packages/fastapi/applications.py", line 1082, in __call__

    await super().__call__(scope, receive, send)

  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/applications.py", line 113, in __call__

    await self.middleware_stack(scope, receive, send)

  File "/opt/datadog/apm/library/python/ddtrace_pkgs/site-packages-ddtrace-py3.11-manylinux2014/ddtrace/contrib/internal/asgi/middleware.py", line 456, in __call__

    return await self.app(scope, wrapped_recv, wrapped_send)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__

    raise exc

  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__

    await self.app(scope, receive, _send)

  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/gzip.py", line 29, in __call__

    await responder(scope, receive, send)

  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/gzip.py", line 130, in __call__

    await super().__call__(scope, receive, send)

  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/gzip.py", line 46, in __call__

    await self.app(scope, receive, self.send_with_compression)

  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__

    await self.app(scope, receive, send)

  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/base.py", line 182, in __call__

    with recv_stream, send_stream, collapse_excgroups():

  File "/usr/python/lib/python3.11/contextlib.py", line 158, in __exit__

    self.gen.throw(typ, value, traceback)

  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/_utils.py", line 85, in collapse_excgroups

    raise exc

  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/base.py", line 184, in __call__

    response = await self.dispatch_func(request, call_next)

               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/airflow/.local/lib/python3.11/site-packages/airflow/api_fastapi/auth/middlewares/refresh_token.py", line 49, in dispatch

    response = await call_next(request)

               ^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/base.py", line 160, in call_next

    raise RuntimeError("No response returned.")

RuntimeError: No response returned.

What you think should happen instead?

I should be able to access the login page

How to reproduce

Custom docker image:

FROM apache/airflow:3.1.3-python3.11

USER root

RUN apt-get update && \
    apt-get install -y git g++ php php-cli php-common && \
    apt-get clean && rm -rf /var/lib/apt/lists/*

USER airflow

Helm values:

images:
  airflow:
    repository: ${IMAGE_NAME}
    tag: ${VERSION}
    pullPolicy: Always

ingress:
  apiServer:
    enabled: true
    annotations:
      projectcontour.io/response-timeout: "120s"
    hosts: 
      - name: "airflow.staging.mydomain.cloud"
    ingressClassName: "contour"

env:
  - name: "AIRFLOW__API__AUTH_BACKEND"
    value: "airflow.api.auth.backend.basic_auth"
  - name: "AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION"
    value: "True"
  - name: "AIRFLOW__CORE__LOAD_EXAMPLES"
    value: "False"
  - name: AIRFLOW__CORE__EXECUTE_TASKS_NEW_PYTHON_INTERPRETER
    value: "True"
  - name: AIRFLOW__LOGGING__REMOTE_LOGGING
    value: "True"
  - name: AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID
    value: "LoggingS3Connection"
  - name: AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER
    value: "s3://logs-archive-product-team/Airflow"
  - name: AIRFLOW__LOGGING__ENCRYPT_S3_LOGS
    value: "False"
  - name: AIRFLOW__CORE__ENABLE_XCOM_PICKLING
    value: "True"
  - name: AIRFLOW__SMTP__SMTP_HOST
    value: "smtp.gmail.com"
  - name: AIRFLOW__SMTP__SMTP_PORT
    value: "587"
  - name: AIRFLOW__SMTP__SMTP_SSL
    value: "False"
  - name: AIRFLOW__SMTP__SMTP_STARTTLS
    value: "True"
  - name: AIRFLOW__SMTP__SMTP_USER
    value: "[email protected]"
  - name: "CM_API_URL"
    value: "https://cm-api.mydomain.cloud/api"

secret:
  - envName: "GIT_TOKEN"
    secretName: "airflow-git-token"
    secretKey: "GIT_TOKEN"

data:
  metadataSecretName: "airflow-pgbouncer-connection"

apiSecretKey: ${API_SECRET_KEY}
jwtSecret: ${JWT_SECRET}
fernetKey: ${FERNET_KEY}
webserverSecretKey: ${WEBSERVER_SECRET_KEY}

workers:
  replicas: 1

registry:
  secretName: regcred

postgresql:
  enabled: false

pgbouncer:
  enabled: true
  configSecretName: airflow-pgbouncer-config
  metricsExporterSidecar:
    statsSecretName: airflow-pgbouncer-stats

dags:
  gitSync:
    enabled: true
    repo: [email protected]:mydomain/airflow-dags.git
    branch: develop
    ref: develop
    depth: 1
    subPath: "dags"
    sshKeySecret: airflow-ssh-secret

Deployment script:

export PGBOUNCER_INI_BASE64=$(envsubst < k8s/pgbouncer/pgbouncer.ini | base64 -w 0)
export PGBOUNCER_USERS_BASE64=$(envsubst < k8s/pgbouncer/users.txt | base64 -w 0)
envsubst <k8s/pgbouncer/airflow-pgbouncer-connection.yaml >airflow-pgbouncer-connection.yaml
envsubst <k8s/pgbouncer/airflow-pgbouncer-stats.yaml >airflow-pgbouncer-stats.yaml
envsubst <k8s/pgbouncer/airflow-pgbouncer-config.yaml >airflow-pgbouncer-config.yaml
envsubst <k8s/airflow-aws-secrets.yaml >airflow-aws-secrets.yaml
envsubst <k8s/airflow-connections-secrets.yaml >airflow-connections-secrets.yaml
envsubst <k8s/airflow-git-secrets.yaml >airflow-git-secrets.yaml
envsubst <k8s/airflow-git-token.yaml >airflow-git-token.yaml

echo Installing secrets
ssh -p 2222 docker-deploy@${SWARM_HOST} 'cat - | kubectl apply -f -' <airflow-aws-secrets.yaml
ssh -p 2222 docker-deploy@${SWARM_HOST} 'cat - | kubectl apply -f -' <airflow-connections-secrets.yaml
ssh -p 2222 docker-deploy@${SWARM_HOST} 'cat - | kubectl apply -f -' <airflow-git-secrets.yaml
ssh -p 2222 docker-deploy@${SWARM_HOST} 'cat - | kubectl apply -f -' <airflow-git-token.yaml
ssh -p 2222 docker-deploy@${SWARM_HOST} 'cat - | kubectl apply -f -' <airflow-pgbouncer-connection.yaml
ssh -p 2222 docker-deploy@${SWARM_HOST} 'cat - | kubectl apply -f -' <airflow-pgbouncer-stats.yaml
ssh -p 2222 docker-deploy@${SWARM_HOST} 'cat - | kubectl apply -f -' <airflow-pgbouncer-config.yaml

echo Importing settings
envsubst <k8s/${ENV}-airflow-values.yml >airflow-values.yml

echo Airflow helm repo adding
ssh -p 2222 docker-deploy@${SWARM_HOST} 'helm repo add apache-airflow https://airflow.apache.org'
ssh -p 2222 docker-deploy@${SWARM_HOST} 'helm repo update'

echo Airflow Helm installing

ssh -o ServerAliveInterval=60 -o ServerAliveCountMax=10 -p 2222 docker-deploy@${SWARM_HOST} "cat - | helm upgrade --install airflow apache-airflow/airflow --namespace airflow-$ENV -f -" <airflow-values.yml

Operating System

Ubuntu 24.04.2 LTS

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

images:
  airflow:
    repository: ${IMAGE_NAME}
    tag: ${VERSION}
    pullPolicy: Always

ingress:
  apiServer:
    enabled: true
    annotations:
      projectcontour.io/response-timeout: "120s"
    hosts: 
      - name: "airflow.staging.translated.cloud"
    ingressClassName: "contour"

env:
  - name: "AIRFLOW__API__AUTH_BACKEND"
    value: "airflow.api.auth.backend.basic_auth"
  - name: "AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION"
    value: "True"
  - name: "AIRFLOW__CORE__LOAD_EXAMPLES"
    value: "False"
  - name: AIRFLOW__CORE__EXECUTE_TASKS_NEW_PYTHON_INTERPRETER
    value: "True"
  - name: AIRFLOW__LOGGING__REMOTE_LOGGING
    value: "True"
  - name: AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID
    value: "LoggingS3Connection"
  - name: AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER
    value: "s3://logs-archive-product-team/Airflow"
  - name: AIRFLOW__LOGGING__ENCRYPT_S3_LOGS
    value: "False"
  - name: AIRFLOW__CORE__ENABLE_XCOM_PICKLING
    value: "True"
  - name: AIRFLOW__SMTP__SMTP_HOST
    value: "smtp.gmail.com"
  - name: AIRFLOW__SMTP__SMTP_PORT
    value: "587"
  - name: AIRFLOW__SMTP__SMTP_SSL
    value: "False"
  - name: AIRFLOW__SMTP__SMTP_STARTTLS
    value: "True"
  - name: AIRFLOW__SMTP__SMTP_USER
    value: "[email protected]"
  - name: "CM_API_URL"
    value: "https://cm-api.translated.cloud/api"

secret:
  - envName: "GIT_TOKEN"
    secretName: "airflow-git-token"
    secretKey: "GIT_TOKEN"

data:
  metadataSecretName: "airflow-pgbouncer-connection"

apiSecretKey: ${API_SECRET_KEY}
jwtSecret: ${JWT_SECRET}
fernetKey: ${FERNET_KEY}
webserverSecretKey: ${WEBSERVER_SECRET_KEY}

workers:
  replicas: 1

registry:
  secretName: regcred

postgresql:
  enabled: false

pgbouncer:
  enabled: true
  configSecretName: airflow-pgbouncer-config
  metricsExporterSidecar:
    statsSecretName: airflow-pgbouncer-stats

dags:
  gitSync:
    enabled: true
    repo: [email protected]:translated/airflow-dags.git
    branch: develop
    ref: develop
    depth: 1
    subPath: "dags"
    sshKeySecret: airflow-ssh-secret

Anything else?

We were able to solve the issue by manually fixing the ingress definition who was pointing to the service airflow-api-server:0, by changing it to airflow-api-server:8080 everything worked fine and the page stopped giving internal serror

Are you willing to submit PR?

  • [ ] Yes I am willing to submit a PR!

Code of Conduct

santurini avatar Dec 10 '25 15:12 santurini

have you cleared the browser cookies? it solved the Internal Server Error for me

zachliu avatar Dec 10 '25 21:12 zachliu

@zachliu I did but it was not working, as I mentioned I was able to solve by manually fixing the ingress that was pointing to the wrong service's port

santurini avatar Dec 11 '25 09:12 santurini