Ryan Hatter

@tomrutter Would you like to be assigned? Given the current design (using a task's `context` object means we have to wait for the task to start running), I think this...

@tanvn Can you check the [audit logs](https://airflow.apache.org/docs/apache-airflow/stable/security/audit_logs.html) for a task with the following log? ``` [2024-04-17, 16:56:29 JST] {local_task_job.py:223} WARNING - State of this instance has been externally set to...

@tanvn Thanks for the extra details! I've assigned you to the Issue.

@tanvn Can you share the full traceback?

What is the [worker pods creation batch size](https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/configurations-ref.html#worker-pods-creation-batch-size)? This limits the number of pods created during a given scheduler loop

Can you see if this is occurring on the latest version of the [spark provider package](https://airflow.apache.org/docs/apache-airflow-providers-apache-spark/stable/index.html) (4.8.0)?

I'm not able to reproduce -- I think the context object in the post execute function is incorrect. `context["dataset_events"]["test-cluster/test-schema/test-table"]` should be (I think) `context["triggered_dataset_events"]["test-cluster/test-schema/test-table"]`. Even with that updated, it's coming...

@dirrao can you take a look? Also, it would probably be a good idea to add to the documentation what metrics tagging even is

I _think_ this is a duplicate of #36998. BTW restarting the scheduler temporarily solves this.

@aru-trackunit are you able to reproduce? Are you sure this isn't #36998 ? If it is, [this patch](https://github.com/apache/airflow/issues/36998#issuecomment-2073367112) might resolve it.