default_automation_condition sensor causes "(psycopg2.errors.QueryCanceled) canceling statement due to statement timeout"

[liongkj]

Dagster version

1.8.1

What's the issue?

Operation name: {SingleSensorQuery}

Message: (psycopg2.errors.QueryCanceled) canceling statement due to statement timeout

[SQL: SELECT runs.id, runs.run_body, runs.status, runs.create_timestamp, runs.update_timestamp, runs.start_time, runs.end_time 
FROM runs JOIN run_tags AS run_tags_filter0 ON runs.run_id = run_tags_filter0.run_id AND run_tags_filter0.key = %(key_1)s AND run_tags_filter0.value = %(value_1)s JOIN run_tags AS run_tags_filter1 ON runs.run_id = run_tags_filter1.run_id AND run_tags_filter1.key = %(key_2)s AND run_tags_filter1.value = %(value_2)s ORDER BY runs.id DESC 
 LIMIT %(param_1)s]
[parameters: {'key_1': 'dagster/sensor_name', 'value_1': 'default_automation_condition_sensor', 'key_2': '.dagster/repository', 'value_2': '__repository__@thingsboard_code', 'param_1': 1}]
(Background on this error at: https://sqlalche.me/e/20/e3q8)

Path: ["sensorOrError","sensorState","runs"]

Locations: [{"line":30,"column":9}]

console error message

{
    "message": "(psycopg2.errors.QueryCanceled) canceling statement due to statement timeout\n\n[SQL: SELECT runs.id, runs.run_body, runs.status, runs.create_timestamp, runs.update_timestamp, runs.start_time, runs.end_time \nFROM runs JOIN run_tags AS run_tags_filter0 ON runs.run_id = run_tags_filter0.run_id AND run_tags_filter0.key = %(key_1)s AND run_tags_filter0.value = %(value_1)s JOIN run_tags AS run_tags_filter1 ON runs.run_id = run_tags_filter1.run_id AND run_tags_filter1.key = %(key_2)s AND run_tags_filter1.value = %(value_2)s ORDER BY runs.id DESC \n LIMIT %(param_1)s]\n[parameters: {'key_1': 'dagster/sensor_name', 'value_1': 'default_automation_condition_sensor', 'key_2': '.dagster/repository', 'value_2': '__repository__@thingsboard_code', 'param_1': 1}]\n(Background on this error at: https://sqlalche.me/e/20/e3q8)",
    "locations": [
        {
            "line": 30,
            "column": 9
        }
    ],
    "path": [
        "sensorOrError",
        "sensorState",
        "runs"
    ],
    "extensions": {
        "errorInfo": {
            "__class__": "SerializableErrorInfo",
            "message": "sqlalchemy.exc.OperationalError: (psycopg2.errors.QueryCanceled) canceling statement due to statement timeout\n\n[SQL: SELECT runs.id, runs.run_body, runs.status, runs.create_timestamp, runs.update_timestamp, runs.start_time, runs.end_time \nFROM runs JOIN run_tags AS run_tags_filter0 ON runs.run_id = run_tags_filter0.run_id AND run_tags_filter0.key = %(key_1)s AND run_tags_filter0.value = %(value_1)s JOIN run_tags AS run_tags_filter1 ON runs.run_id = run_tags_filter1.run_id AND run_tags_filter1.key = %(key_2)s AND run_tags_filter1.value = %(value_2)s ORDER BY runs.id DESC \n LIMIT %(param_1)s]\n[parameters: {'key_1': 'dagster/sensor_name', 'value_1': 'default_automation_condition_sensor', 'key_2': '.dagster/repository', 'value_2': '__repository__@thingsboard_code', 'param_1': 1}]\n(Background on this error at: https://sqlalche.me/e/20/e3q8)\n",
            "stack": [
                "  File \"/usr/local/lib/python3.11/site-packages/graphql/execution/execute.py\", line 521, in execute_field\n    result = resolve_fn(source, info, **args)\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
                "  File \"/usr/local/lib/python3.11/site-packages/dagster_graphql/schema/instigation.py\", line 661, in resolve_runs\n    for record in graphene_info.context.instance.get_run_records(\n                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
                "  File \"/usr/local/lib/python3.11/site-packages/dagster/_utils/__init__.py\", line 685, in inner\n    return func(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^\n",
                "  File \"/usr/local/lib/python3.11/site-packages/dagster/_core/instance/__init__.py\", line 1838, in get_run_records\n    return self._run_storage.get_run_records(\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
                "  File \"/usr/local/lib/python3.11/site-packages/dagster/_core/storage/runs/sql_run_storage.py\", line 414, in get_run_records\n    rows = self.fetchall(query)\n           ^^^^^^^^^^^^^^^^^^^^\n",
                "  File \"/usr/local/lib/python3.11/site-packages/dagster/_core/storage/runs/sql_run_storage.py\", line 119, in fetchall\n    return db_fetch_mappings(conn, query)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
                "  File \"/usr/local/lib/python3.11/site-packages/dagster/_core/storage/sqlalchemy_compat.py\", line 35, in db_fetch_mappings\n    return conn.execute(query).mappings().all()\n           ^^^^^^^^^^^^^^^^^^^\n",
                "  File \"/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py\", line 1418, in execute\n    return meth(\n           ^^^^^\n",
                "  File \"/usr/local/lib/python3.11/site-packages/sqlalchemy/sql/elements.py\", line 515, in _execute_on_connection\n    return connection._execute_clauseelement(\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
                "  File \"/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py\", line 1640, in _execute_clauseelement\n    ret = self._execute_context(\n          ^^^^^^^^^^^^^^^^^^^^^^\n",
                "  File \"/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py\", line 1846, in _execute_context\n    return self._exec_single_context(\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
                "  File \"/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py\", line 1986, in _exec_single_context\n    self._handle_dbapi_exception(\n",
                "  File \"/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py\", line 2355, in _handle_dbapi_exception\n    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e\n",
                "  File \"/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py\", line 1967, in _exec_single_context\n    self.dialect.do_execute(\n",
                "  File \"/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py\", line 941, in do_execute\n    cursor.execute(statement, parameters)\n"
            ],
            "cls_name": "OperationalError",
            "cause": {
                "__class__": "SerializableErrorInfo",
                "message": "psycopg2.errors.QueryCanceled: canceling statement due to statement timeout\n\n",
                "stack": [
                    "  File \"/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py\", line 1967, in _exec_single_context\n    self.dialect.do_execute(\n",
                    "  File \"/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py\", line 941, in do_execute\n    cursor.execute(statement, parameters)\n"
                ],
                "cls_name": "QueryCanceled",
                "cause": null,
                "context": null
            },
            "context": null
        }
    }
}

What did you expect to happen?

I am not able to switch on the automation sensor

I should be able to switch on the automation and see the details

How to reproduce?

not really sure, it was only actually happenining in my production server but not local development

Deployment type

Docker Compose

Deployment details

No response

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.