webserver - acces task log | TypeError: '<' not supported between instances of 'Interval' and 'DateTime'

[raphaelauv]

Apache Airflow version

2.9.0

What happened?

I can't access the logs from the airflow-UI of one scpeific task in my DAG

{app.py:1744} ERROR - Exception on /api/v1/dags/YYYYYYYY/dagRuns/manual__2024-04-29T07:17:59+00:00/taskInstances/XXXXXXXX/logs/1 [GET]
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 2529, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/connexion/decorators/decorator.py", line 68, in wrapper
    response = function(request)
               ^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/connexion/decorators/uri_parsing.py", line 149, in wrapper
    response = function(request)
               ^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/connexion/decorators/validation.py", line 399, in wrapper
    return function(request)
           ^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/connexion/decorators/response.py", line 113, in wrapper
    return _wrapper(request, response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/connexion/decorators/response.py", line 90, in _wrapper
    self.operation.api.get_connexion_response(response, self.mimetype)
  File "/home/airflow/.local/lib/python3.12/site-packages/connexion/apis/abstract.py", line 366, in get_connexion_response
    return cls._framework_to_connexion_response(response=response, mimetype=mimetype)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/connexion/apis/flask_api.py", line 165, in _framework_to_connexion_response
    body=response.get_data() if not response.direct_passthrough else None,
         ^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/werkzeug/wrappers/response.py", line 314, in get_data
    self._ensure_sequence()
  File "/home/airflow/.local/lib/python3.12/site-packages/werkzeug/wrappers/response.py", line 376, in _ensure_sequence
    self.make_sequence()
  File "/home/airflow/.local/lib/python3.12/site-packages/werkzeug/wrappers/response.py", line 391, in make_sequence
    self.response = list(self.iter_encoded())
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/werkzeug/wrappers/response.py", line 50, in _iter_encoded
    for item in iterable:
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/log/log_reader.py", line 87, in read_log_stream
    logs, metadata = self.read_log_chunks(ti, current_try_number, metadata)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/log/log_reader.py", line 64, in read_log_chunks
    logs, metadatas = self.log_handler.read(ti, try_number, metadata=metadata)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/log/file_task_handler.py", line 451, in read
    log, out_metadata = self._read(task_instance, try_number_element, metadata)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/log/file_task_handler.py", line 374, in _read
    logs = "\n".join(
           ^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/log/file_task_handler.py", line 129, in _interleave_logs
    for _, _, v in sorted(
                   ^^^^^^^
TypeError: '<' not supported between instances of 'Interval' and 'DateTime'

How to reproduce

I don't know how to reproduce precisely this bug

Deployment

Official Apache Airflow Helm Chart

Are you willing to submit PR?

• [ ] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[eladkal]

Not sure what we can do here without reproduce example

[raphaelauv]

still present in 2.9.1

the full url that give the error is

https://XXXX.ZZZ/get_logs_with_metadata?dag_id=QQQQ&task_id=MMMMM&map_index=-1&execution_date=2024-04-29T07%3A17%3A59%2B00%3A00&try_number=1&metadata=null

the log file is stored on S3

I will publish it as soon I get it manually

[raphaelauv]

I could reproduce the bug by isolating the log lines that make the webserver crash

run a deferable kpo task

from pendulum import today
from airflow import DAG
from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator

dag = DAG(
    dag_id="kubernetes_dag",
    schedule_interval="0 0 * * *",
    start_date=today("UTC").add(days=-1)
)

with dag:
    cmd = "echo toto && sleep 4 && echo finish"

    KubernetesPodOperator(
        task_id="task-one",
        namespace="default",
        kubernetes_conn_id="kubernetes_default",
        name="airflow-test-pod",
        image="alpine:3.16.2",
        cmds=["sh", "-c", cmd],
        deferrable=True,
        poll_interval=2,
        get_logs=True,
    )

then replace the content of the attempt\=1.log with only this string

[2024-04-29T07:37:28.609+0000] {pod.py:795} INFO - Container logs: 
   8192/6040200 [..............................] - ETA: 0s
1474560/6040200 [======>.......................] - ETA: 0s
3194880/6040200 [==============>...............] - ETA: 0s
5505024/6040200 [==========================>...] - ETA: 0s
6040200/6040200 [==============================] - 0s 0us/step

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.

[raphaelauv]

no stale

[shahar1]

I could reproduce the bug by isolating the log lines that make the webserver crash

run a deferable kpo task

then replace the content of the attempt\=1.log with only this string

[2024-04-29T07:37:28.609+0000] {pod.py:795} INFO - Container logs: 
   8192/6040200 [..............................] - ETA: 0s����������������������������������������������������������
1474560/6040200 [======>.......................] - ETA: 0s����������������������������������������������������������
3194880/6040200 [==============>...............] - ETA: 0s����������������������������������������������������������
5505024/6040200 [==========================>...] - ETA: 0s����������������������������������������������������������
6040200/6040200 [==============================] - 0s 0us/step

Could you please edit the original issue and provide a minimal (yet full) example so other people could reproduce the issue?

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.

[raphaelauv]

no stale