Daniel Standish

Results 109 comments of Daniel Standish

Fix race condition in KubernetesExecutor with concurrently running schedulers

@changqian9 it looks like orig author may have gotten busy or lost motivation. anyone could pick it up and see it through if motivated.

Datasets - trigger DAG run when any dataset is updated

@sunank200 and I are working on this right now actually

Datasets - trigger DAG run when any dataset is updated

Yes there's a number of PRs that have been merged for this but this is the one that show you the syntax https://github.com/apache/airflow/pull/37101

Many running task instances are cleared by the new scheduler when an old scheduler is terminated and its health check server is periodically requested

Hi @tanvn It seems you've made some progress with your repro scenario. What I would recommend doing is continue digging in. Add more logging to try to see exactly what...

Fix: Reuse ProcessPoolExecutor in CeleryExecutor

I am a little skeptical that the overhead for process pool creation is meaningful. Because i think the dominant performance consideration is the blocking IO sending the task to redis...

Fix: Reuse ProcessPoolExecutor in CeleryExecutor

> I'm testing it through the logs. It takes a while. It usually takes about 2 days to see that the scheduling process slows down significantly. Understood. Did you see...

Fix parsing connection uri which contains a composite protocol sparated by :

This fix may not be necessary: see comment https://github.com/apache/airflow/issues/33442#issuecomment-1817228790

Fix parsing connection uri which contains a composite protocol sparated by :

See, for example, ODBC hook, which allows you to specify the sqlalchemy scheme with `sqlalchemy_scheme`, either as hook init param, or as a connection extra.

Add watchdog for immediately processing changes in the DAGs folder

Nice

Add watchdog for immediately processing changes in the DAGs folder

So what's the "solution" for dynamic? Parse those on schedule?